Assessment of Complex Abilities in Engineering Students using Structural Equation Models and Multidimensional Item Response Theory: Preliminary Results

摘要

Nowadays, several abilities in engineering students are required, one of them is the ICT (Information and Communication Technologies) literacy proficiency. The ICT literacy proficiency is defined as the ability to use digital technology, communication tools, and networks appropriately to solve information problems in order to function in an information society. This includes the ability to use technology as a tool to research, organize, evaluate, and communicate information. A computer system to diagnostic ICT literacy proficiency in engineering students is developing. This system includes interactive simulations which emulates the features of real software products (word processing, electronic sheet, internet search engine, presentation system and Chat). This system also includes monitors which follows the test taker's progress through the simulation to determine if the students answered a question correctly. In this paper is presented the model for validating the cognitive structure and scoring the ICT literacy proficiency test in a computer-based system. The model concerns the confluence of measurement models, statistical inference, and cognitive diagnosis in the design and analysis of an assessment model of ICT literacy proficiency. The system assessment is a comprehensive test of ICT proficiency specifically designed for engineering students. The system uses real-time, scenario-based tasks (not multiple-choice questions) to assess five ICT skills: obtaining information, managing information, evaluating, creating and communicating. When the purpose of an assessment includes what kind of knowledge, skill or ability we want to make inferences about, we need new assessment paradigms. Taking some ideas from Evidence Centered Assessment Design and the Rule Space approach, we proposed a Model to evaluate a cognitive structure and identifying several evidences that represent the knowledge, skills, and abilities we wish to assess.