Gamified performance assessment of collaborative problem solving skills

摘要

In this paper we introduce a game-based approach for Collaborative Problem Solving (CPS) Skills assessment and provide preliminary evidence from a validation pilot study. To date, educational assessments have focused more heavily on the concrete, and accessible aspects of CPS with a diminished representation of the social aspects of CPS. We addressed this issue through the integration of our CPS construct into the game-based assessment "Circuit Runner" in which participants interact with a virtual agent to solve a series of challenges in a first-person maze environment (von Davier, 2017). Circuit Runner provides an environment that allows for controlled interdependence between a user and a virtual agent that facilitates the demonstration of the broad range of cognitive and social skills required for effective CPS. Tasks are designed to incorporate telemetry-based (e.g., log file, clickstream, interaction-based) and item response data to provide a more comprehensive measure of CPS skills. Our study included 500 participants on Amazon Mechanical Turk, who completed Circuit Runner, pre- and post-game surveys, and a CPS situational judgment test (CPS-SJT). These elements, in conjunction with the game-play, allowed for an expanded exploration of CPS skills with different modalities and types of instruments. The findings support and extend efforts to provide a stronger theoretical and empirical foundation for insights regarding CPS as a skillset, as well as the design of scalable game-based CPS assessments.