Development and analysis of a socio-scientific reasoning assessment : application of computer automated scoring and rasch analysis

Abstract

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT AUTHOR'S REQUEST.] Scientific literacy (SL) as an idea is considered by many to be one of the main goals of science education. This study focuses on Roberts' (2007) second vision of SL, the ability of students to competently navigate "situations with a scientific component..." (p. 730). Assessments centered on Vision-II-SL could be refined to better capture student's competency. To this end, this study focuses on the development of an open- ended Vision-II aligned assessment (QuASSR-oe2). To make this assessment usable in large populations this study utilizes automated scoring, which is the synergy of natural-language-processing and supervised-machine-learning. This study establishes the validity and reliability of the QuASSR-oe2 using psychometric techniques from classical-test theory and Rasch analysis. The study also establishes that accurate automated-scoring models (QWK [greater than] .7) can be generated, thus allowing for efficient scoring of open-ended student responses. Ultimately, this allows for the QuASSR-oe2 to be easily utilized in large populations.