Estimating reliability under a generalizability theory model for writing scores in C-base

Abstract

Reliability appraisal is important in test development and for test use. This paper appraises the reliability of the writing scores in College Basic Academic Subjects Examination (College BASE) by both the traditional and modern means of reliability estimation. Three types of analyses were conducted: correlational analysis, generalized analysis of variance and classical reliability analysis. Classical Test Theory (CTT) was compared and contrasted to Generalizability Theory (G-Theory). The G study design used in this study was):(tip[x], where p refers to the persons effect and is the object of measurement, i refers to the test items and is a random effect, and t represents the writing skills and is a fixed effect. The Generalizability coefficient from the G study was close to the Cronbach's alpha from the classical test theory. About 60% of the observed-score variance in the multiple-choice writing items in College BASE was due to the persons effect.