From independent ratings to communal ratings: A study of CWA raters’ decision-making behaviors.

The present exploratory study maps the decision-making behaviors of raters in a well-established communal writing assessment (CWA) context, tracing their behaviors all the way from independent rating sessions, where initial images and judgments are formed, to communal rating sessions, where final scores are assigned on the basis of collaboration between two raters. Results from think-aloud protocols, recorded discussions, and retrospective reports from 20 experienced raters rating 15 EFL essays showed that when moving from independent ratings to communal ratings, raters gradually refined their assessments and balanced their attention more evenly among the official assessment criteria to reach what they believed to be more accurate scores. These interpretations support a hermeneutic rather than a psychometric approach to establishing the validity of the CWA practices. [ABSTRACT FROM AUTHOR]