Stability of rater agreement measures: the effect of shifting cell values within fixed margins

Abstract

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT AUTHOR'S REQUEST.] This thesis examines the effect of shifting cell observations within marginal values on rater agreement measures to provide guides for researchers in selecting rater agreement measures when presented with highly unequally distributed marginal values. The first study of the measures indicate von Eye's [kappa]sc is not as influenced by strictly increasing agreement as other measures of agreement and may be preferable to Cohen's (1960) [kappa]c in these cases. Cohen's (1968) weighted kappa measure, [kappa]w appears to be resistant to strictly increasing agreement in the same way as [kappa]sc. The second study investigates rater agreement measure patterns as different percent of agreement counts are moved across columns while maintaining fixed margins. Results indicate all kappa measures decrease with the same pattern and rate under the same conditions, indicating that regardless of assumptions or attempts to correct for chance agreement each measure takes into account, the absolute proportion of agreement may be the most parsimonious measure of agreement.