Date of Award
December 2024
Document Type
Dissertation
Degree Name
Doctor of Education (EdD)
Department
Educational Leadership
First Advisor
Woei Hung
Abstract
Pilot schools operating in accordance with 14 CFR 141 provide a substantial supply of pilots to the aviation industry. Recent high industry demand for pilots has created turnover among the flight instructor workforce, often considered an entry-level position used by aspiring aviators to build flight experience. This in turn means that check instructors, evaluators who are often selected from the more senior flight instructor population at a pilot school, are generally less experienced than in the past. Anecdotal evidence suggests that there is low consistency among these check instructors when conducting performance-based assessments. This study sought to determine the existing level of consistency among check instructors at a pilot school, explore the use of the Delphi method to create descriptive rubrics to be used to assess pilots on performance-based assessments, and evaluate the effectiveness of these rubrics in improving rater consistency among these check instructors. Results were mixed, indicating improvement in rater consistency in some tasks, but degraded consistency in others. The Delphi method proved to be an efficient means to gather consensus and establish written descriptions for a performance-based rubric. The methods utilized in this study provide a simple means for flight training providers and regulators to assess existing levels of consistency among check instructors and develop assessment instruments intended to enhance assessment validity and reliability.
Recommended Citation
Archer, Lewis Robert James, "Impact Of Delphi-Developed Rubrics On Rater Consistency In 14 CFR 141 Flight Training Assessment Environments" (2024). Theses and Dissertations. 6518.
https://commons.und.edu/theses/6518