Bridging in-game behaviors and learning outcomes : design, implementation, and validation of stealth assessment pipelines in Mission HydroSci
No Thumbnail Available
Authors
Meeting name
Sponsors
Date
Journal Title
Format
Thesis
Subject
Abstract
[EMBARGOED UNTIL 08/01/2026] Digital Game-Based Learning (DGBL) holds the potential to foster deep engagement, complex problem-solving, and high motivation among learners. However, fully leveraging these advantages requires robust, scalable, and unobtrusive assessment strategies to capture nuanced cognitive and affective developments in real time. This dissertation addresses these needs through three interlinked studies conducted in the context of Mission HydroSci (MHS), a 3D digital game designed to teach middleschool water science and scientific argumentation skills. Study 1 demonstrates how to design and implement a customized, learning-focused logging system by combining the Activity Theory-based Model of Serious Games (ATMSG) and the Experience API (xAPI). This co-designed approach ensures granular data capture aligned with key educational objectives, effectively distinguishing novice from expert in-game behaviors. Study 2 builds on these logs to create a fully interpretable stealth assessment pipeline for a single learning objective. Guided by conceptual frameworks for systematic feature engineering, the study employs various machine learning classifiers and a surrogate modeling approach to accurately predict targeted competencies, preserving interpretability for educators and stakeholders. Study 3 scales the methodology to multiple learning outcomes across diverse MHS units. By incorporating multi-layered unsupervised learning for feature extraction, ensemble-learning-based predictive modeling, and post-hoc interpretability techniques such as permutation importance scores and Accumulated Local Effects (ALE) plots, this extended pipeline adapts to complex data distributions while maintaining transparency. Findings reveal that purposeful data collection, systematic feature engineering, and robust modeling approaches can reliably capture and forecast learners' knowledge gains, thereby enabling timely feedback and targeted interventions. Collectively, these studies substantiate a coherent progression from conceptual foundations to practical solutions in stealth assessment. The dissertation underscores how embedded logging systems and sophisticated analytics can assess and enhance learning in complex DGBL environments, offering actionable insights for designers, educators, and researchers alike.
Table of Contents
PubMed ID
Degree
Ph. D