The generalizability of performance assessment in mathematics: An evaluation of an analytic scoring procedure using univariate and multivariate methods.

摘要

The purpose of this study was to examine the dependability of teacher-constructed performance assessments in algebra. Mathematics performance-assessment items and an analytic scoring paradigm were developed and revised by a team of eight algebra teachers at a single high school in a major metropolitan area in the Midwest. Various combinations of the items were given to 379 intro algebra students. Each item response was scored by at least two trained teacher raters. Rater reliability and inter-item performance consistency were assessed using univariate and multivariate generalizability models. Results indicated that the performance-assessment items should be used with extreme caution for district-wide assessments and other large-scale evaluations used for accountability purposes. Consistent with previous findings, inter-rater reliability was shown to be of less concern than inter-task performance consistency. Multivariate generalizability models were used to show that correlations among the teacher-constructed analytic-rating scales were generally very high, suggesting that the analytic scores did not provide any unique information. For this data, it might be most appropriate to average across the five analytic ratings (or use a holistic score) and apply a univariate model. Additional research is needed to explain why inter-task performance consistency is so poor for performance-assessment items.