Concept map / Rater agreement
Share
Rater agreement
Compare exact agreement, Cohen’s kappa, weighted kappa and two intraclass correlations for two raters scoring the same essays.
1Two raters score the same 200 essays
When two people score the same essays, part of each score depends on who happened to read it. Classical test theory counts that as measurement error. The first number many people reach for is the correlation between the raters, but it only asks whether they put the essays in the same order. Two raters can order the essays identically and never give the same score.
Here raters A and B score 200 simulated essays from 1 to 4. At the start they agree exactly on 75% of them. Make rater B one point harsher and exact agreement falls to 29%, while the correlation only drops from 0.85 to 0.77.
The two raters are equally harsh, with means of 2.40 and 2.42. They give the same score to 75% of the 200 essays and are within one point on 100%. Chance alone would give 29% exact agreement, so Cohen’s kappa is 0.65. The consistency ICC(3,1) is 0.85 and the absolute-agreement ICC(2,1) is 0.85.
More controls
True essay quality has standard deviation 1, which gives σ its scale.
2A weaker class, more agreement, lower kappa?
Two pairs of raters use the same rubric and make errors of the same size. Pair 1 scores a typical class, where all four scores are common. Pair 2 scores a weaker class, where most essays deserve a 1. Pair 1 agrees exactly on 75% of essays and pair 2 on 89%. Which pair has the higher Cohen’s kappa?
3What kappa subtracts, and what ICC(2,1) adds
Kappa compares the observed agreement po with pe, the agreement two raters would reach if each scored at random at their own base rates. Weighted kappa gives partial credit for near misses.
The two intraclass correlations come from an analysis of variance of the essays-by-raters table. ICC(3,1) asks only whether the raters rank essays alike. ICC(2,1) asks whether one rater’s score could stand in for the other’s, so a gap between the rater means goes into its denominator. Point at a colored term to find it in the chart.
Kappa: (0.750 − 0.288) / (1 − 0.288) = 0.65. Here pe adds up rater A’s share times rater B’s share over the four scores.
Show the mean squares behind the ICCs
Shrout and Fleiss’s formulas use mean squares for essays (MSR), raters (MSC) and residual error (MSE). For these 200 essays: MSR = 1.595, MSC = 0.040, MSE = 0.125. ICC(3,1) = (1.595 − 0.125) / (1.595 + 0.125) = 0.85. The rater term is 2/200 × (0.040 − 0.125) = −0.001, so ICC(2,1) = 0.85.
The raters’ means are close, so the rater term is about zero and the two ICCs agree. Weighted kappa is 0.85, almost exactly ICC(2,1).
4Check yourself
The tasks use the three sliders in section 1.
- Make rater B harsh enough that exact agreement falls below 50% while ICC(3,1) stays at 0.70 or higher.Now: Offset 0.0, exact agreement 75%, ICC(3,1) 0.85
- With δ at 0, move μ until exact agreement is at least 85% and Cohen’s kappa is below 0.60.Now: Offset 0.0, exact agreement 75%, kappa 0.65
- With δ at 0, find a setting where weighted kappa is at least 0.25 above unweighted kappa.Now: Offset 0.0, weighted 0.85 minus unweighted 0.65 = 0.20
5Kappa and the ICCs in irr
The code simulates 200 essays the same way and scores them twice: once with rater B as harsh as rater A, once with B a point harsher. irr::kappa2 gives unweighted and quadratic weighted kappa, and irr::icc gives the consistency and absolute-agreement ICCs for a single rater. R draws its own sample, so its numbers differ a little from the lab’s. With the harsh rater, exact agreement falls from 0.70 to 0.32 and ICC(2,1) from 0.83 to 0.54, while ICC(3,1) only moves from 0.82 to 0.76.
Kappa, weighted kappa and two ICCsShow the code
library(irr)
set.seed(1)
# 200 essays; true quality on the 1 to 4 score scale
n <- 200
quality <- 2.5 + 1.0 * rnorm(n)
# Round a latent score to the nearest point from 1 to 4
score <- function(s) pmin(4, pmax(1, round(s)))
a <- score(quality + 0.3 * rnorm(n))
err_b <- 0.3 * rnorm(n)
b <- score(quality + err_b)
# The same rater B, made one point harsher before rounding
b_harsh <- score(quality + err_b - 1)
agree <- function(a, b) {
# Factor levels 1:4 keep unused scores in the kappa weights
f <- data.frame(factor(a, 1:4), factor(b, 1:4))
c(exact = mean(a == b), adjacent = mean(abs(a - b) <= 1),
kappa = kappa2(f)$value,
qwk = kappa2(f, weight = "squared")$value,
r = cor(a, b),
# Two-way ICCs for a single rater (the default unit)
icc_c = icc(cbind(a, b), "twoway", "consistency")$value,
icc_a = icc(cbind(a, b), "twoway", "agreement")$value)
}
round(rbind(same = agree(a, b), harsh = agree(a, b_harsh)), 2)Uses irr. Install once with install.packages(c("irr")).
Agreement against consistency
- A rater who is a point harsher on every essay can still correlate highly with another rater. Pearson r and ICC(3,1) mostly ignore a constant offset; exact agreement and ICC(2,1) count it as disagreement.
- Cohen’s kappa is (po − pe) / (1 − pe), where pe is the agreement the raters’ base rates would give by chance. It is 0 for chance agreement and 1 for perfect agreement.
- When most essays get the same score, chance agreement is high, so the same raters get a lower kappa while agreeing more often. Report the score distribution next to kappa.
- Quadratic weighted kappa counts a two-point miss as four times worse than a one-point miss, and it comes out almost equal to ICC(2,1). Say which ICC you report: for decisions made on one rater’s score, ICC(2,1) is usually the one that matters.
- Generalizability theory splits score variance among essays, raters and tasks in one design, and the AI rater audit runs these agreement checks on an automated scorer.
Recommended reading
- A coefficient of agreement for nominal scalesJacob CohenPaper · Educational and Psychological Measurement, 20(1), 1960
- Intraclass correlations: Uses in assessing rater reliabilityPatrick E. Shrout and Joseph L. FleissPaper · Psychological Bulletin, 86(2), 1979