Development of Scoring Procedures for the Performance Based Measurement (PBM) Test: Psychometric and Criterion Validity Investigation

摘要

This report describes findings regarding the scoring of the PBM Test and the relationships of various classical test theory (CTT) and item response theory (IRT) based subtest scores and composites with performance criteria for Navy and Marine Corps student pilots and flight officers. Overall, the IRT analyses indicated that the three parameter logistic model (3PLM) and Samejima's graded response model (SGRM) provided good fit to dichotomously and polytomously scored item-level data, respectively, for six of the seven PBM subtests. These analyses set the stage for future research involving differential item and test functioning. It was also found that PBM component scores based on examinee responses, reaction time, or tracking information often yielded criterion related validities in the range of .20-.35. IRT-based scores did not outperform CTT-based scores, therefore we recommend using subtest level CTT-based multiple regression composites for operational decision making. These results suggest that the addition of the PBM to ASTB for Naval Aviation selection will significantly reduce attrition from the Naval Aviation training pipeline, thereby saving Naval Aviation millions in training costs. The following sections of this report describe the PBM subtests, the demographics of the examinee samples, and the results of the psychometric and statistical analyses showing that the PBM subtests are valid predictors of a wide variety of training criteria.