Development of high -stakes credentialing examination test specifications using statistical models to compare with the test specifications from a Holistic Judgmental Model

摘要

This research examined the comparability of three different statistical models and a Holistic Judgmental Model to develop test specifications for a high-stakes credentialing examination with four unique sections. Results from the credentialing body's recent practice analysis were used as the basis to develop test specifications for a new credentialing examination. The credentialing body's practice analysis survey was developed using ratings scales of importance and frequency. Using statistical models, data analyses were performed to evaluate various models of combining ratings scales of importance and frequency to generate test specifications. The statistical models used include the Additive Model, Multiplicative Model, and Hierarchical Ranking Scheme (Raymond & Neustel, 2006). Holistic Judgmental Model test specifications were developed with input from subject-matter experts.;The results of this study indicate that test specifications developed using the Holistic Judgmental Model yield test specifications different from those generated using statistical models for three of four examination sections. For the fourth examination section, minor differences were noted between test specifications generated using statistical models and those developed using the Holistic Judgmental Model. Compelling similarities were found among test specifications generated by the three statistical models. In addition, no individual statistical method generated test specifications that were more comparable than another statistical model to the Holistic Judgmental Model.;Based on results of the research, alternate test specifications could be selected for three of four examination sections if one or more statistical models had been chosen as the basis for development of test specifications. In addition, this research demonstrates that there should be a proportional distribution between the number of practice analysis survey statements in content areas, and the importance of these content areas identified by practitioners in the field in which the practice analysis is being conducted.