Using Cognitive Interviews and Student Response Processes to Validate an Interpretive Argument for the ETS iSkills? Assessment

摘要

EvaluatingthetrustworthinessofInternet-basedorotherdigitalinformationhasbecomeanessential21stcenturyskill.TheiSkills?assessment,fromEducationalTestingService(ETS),purportstomeasuresuchdigitalevaluationskills,alongwithotherdigitalliteracyskills.Inthiswork,weuseanargument-basedapproachtoassessmentvalidationtoinvestigatetheextenttowhichiSkillstestscorescansupportinferencesabouttheabilityofcollegestudentstoevaluateinformationinadigitalenvironment.Eighty-eightundergraduatesrespondedtoiSkillsassessmenttasksandtomoreopen-ended“naturalistic”tasks.Thesenaturalistictasksweredesignedtobelikehomeworkassignmentsthatincorporatethecriticalevaluationofdigitalinformation.Weobservedweak-to-moderatecorrelationsbetweenscores,suggestingoverlapintheskillsassessedbytheiSkillsandthenaturalistictasks.Analysesofconcurrentcognitiveinterviews(n=11of88)suggesteddistinctionsbetweenstudents’responseprocessestotheassessmentandnaturalistictasks.AlthoughiSkillsassessmenttasksappeartoelicitskillsconsistentwithevaluationofdigitalinformationintherealworld,students’responsestothenaturalistictasksdemonstratedbroaderevaluationskillsandlessattentiontothetestingcontext.ThisstudyprovidesempiricalvalidityevidenceregardingETS’siSkillsassessment,aswellasvaluableinsightsintohowundergraduatesevaluateinformationinadigitalenvironment.