Concept map / Rater agreement

Share

Rater agreement

Compare exact agreement, Cohen’s kappa, weighted kappa and two intraclass correlations for two raters scoring the same essays.

Prerequisites Reliability, Correlation

1Two raters score the same 200 essays

When two people score the same essays, part of each score depends on who happened to read it. Classical test theory counts that as measurement error. The first number many people reach for is the correlation between the raters, but it only asks whether they put the essays in the same order. Two raters can order the essays identically and never give the same score.

Here raters A and B score 200 simulated essays from 1 to 4. At the start they agree exactly on 75% of them. Make rater B one point harsher and exact agreement falls to 29%, while the correlation only drops from 0.85 to 0.77.

Rater B scoreRater A score12341234308004551000134790061819%35%35%12%17%38%32%14%Shares of each score. 200 essays.
Each cell counts essays. The outlined diagonal is exact agreement.
Statistic00.51Exact agreement0.75Exact or adjacent1.00Cohen’s kappa0.65Weighted kappa0.85ICC(2,1) agreement0.85ICC(3,1) consistency0.85Pearson r0.85
Penalizes an offset Ignores an offset

The two raters are equally harsh, with means of 2.40 and 2.42. They give the same score to 75% of the 200 essays and are within one point on 100%. Chance alone would give 29% exact agreement, so Cohen’s kappa is 0.65. The consistency ICC(3,1) is 0.85 and the absolute-agreement ICC(2,1) is 0.85.

More controls

True essay quality has standard deviation 1, which gives σ its scale.

2A weaker class, more agreement, lower kappa?

Two pairs of raters use the same rubric and make errors of the same size. Pair 1 scores a typical class, where all four scores are common. Pair 2 scores a weaker class, where most essays deserve a 1. Pair 1 agrees exactly on 75% of essays and pair 2 on 89%. Which pair has the higher Cohen’s kappa?

3What kappa subtracts, and what ICC(2,1) adds

Kappa compares the observed agreement po with pe, the agreement two raters would reach if each scored at random at their own base rates. Weighted kappa gives partial credit for near misses.

The two intraclass correlations come from an analysis of variance of the essays-by-raters table. ICC(3,1) asks only whether the raters rank essays alike. ICC(2,1) asks whether one rater’s score could stand in for the other’s, so a gap between the rater means goes into its denominator. Point at a colored term to find it in the chart.

κ  =  po−pe1−pe\kappa \;=\; \frac{\htmlClass{term-po}{p_o} - \htmlClass{term-pe}{p_e}}{1 - \htmlClass{term-pe}{p_e}}
ICC(3,1)=MSR−MSEMSR+MSE\text{ICC}(3,1) = \frac{\text{MS}_R - \text{MS}_E}{\text{MS}_R + \text{MS}_E}
ICC(2,1)=MSR−MSEMSR+MSE+2n(MSC−MSE)\text{ICC}(2,1) = \frac{\text{MS}_R - \text{MS}_E}{\text{MS}_R + \text{MS}_E + \htmlClass{term-sev}{\tfrac{2}{n}(\text{MS}_C - \text{MS}_E)}}

Kappa: (0.750 − 0.288) / (1 − 0.288) = 0.65. Here pe adds up rater A’s share times rater B’s share over the four scores.

Show the mean squares behind the ICCs

Shrout and Fleiss’s formulas use mean squares for essays (MSR), raters (MSC) and residual error (MSE). For these 200 essays: MSR = 1.595, MSC = 0.040, MSE = 0.125. ICC(3,1) = (1.595 − 0.125) / (1.595 + 0.125) = 0.85. The rater term is 2/200 × (0.040 − 0.125) = −0.001, so ICC(2,1) = 0.85.

The raters’ means are close, so the rater term is about zero and the two ICCs agree. Weighted kappa is 0.85, almost exactly ICC(2,1).

4Check yourself

The tasks use the three sliders in section 1.

  • Make rater B harsh enough that exact agreement falls below 50% while ICC(3,1) stays at 0.70 or higher.Now: Offset 0.0, exact agreement 75%, ICC(3,1) 0.85
  • With δ at 0, move μ until exact agreement is at least 85% and Cohen’s kappa is below 0.60.Now: Offset 0.0, exact agreement 75%, kappa 0.65
  • With δ at 0, find a setting where weighted kappa is at least 0.25 above unweighted kappa.Now: Offset 0.0, weighted 0.85 minus unweighted 0.65 = 0.20

5Kappa and the ICCs in irr

The code simulates 200 essays the same way and scores them twice: once with rater B as harsh as rater A, once with B a point harsher. irr::kappa2 gives unweighted and quadratic weighted kappa, and irr::icc gives the consistency and absolute-agreement ICCs for a single rater. R draws its own sample, so its numbers differ a little from the lab’s. With the harsh rater, exact agreement falls from 0.70 to 0.32 and ICC(2,1) from 0.83 to 0.54, while ICC(3,1) only moves from 0.82 to 0.76.

Kappa, weighted kappa and two ICCsShow the code
library(irr)
set.seed(1)
# 200 essays; true quality on the 1 to 4 score scale
n <- 200
quality <- 2.5 + 1.0 * rnorm(n)
# Round a latent score to the nearest point from 1 to 4
score <- function(s) pmin(4, pmax(1, round(s)))
a <- score(quality + 0.3 * rnorm(n))
err_b <- 0.3 * rnorm(n)
b <- score(quality + err_b)
# The same rater B, made one point harsher before rounding
b_harsh <- score(quality + err_b - 1)
agree <- function(a, b) {
  # Factor levels 1:4 keep unused scores in the kappa weights
  f <- data.frame(factor(a, 1:4), factor(b, 1:4))
  c(exact = mean(a == b), adjacent = mean(abs(a - b) <= 1),
    kappa = kappa2(f)$value,
    qwk = kappa2(f, weight = "squared")$value,
    r = cor(a, b),
    # Two-way ICCs for a single rater (the default unit)
    icc_c = icc(cbind(a, b), "twoway", "consistency")$value,
    icc_a = icc(cbind(a, b), "twoway", "agreement")$value)
}
round(rbind(same = agree(a, b), harsh = agree(a, b_harsh)), 2)

Uses irr. Install once with install.packages(c("irr")).

Agreement against consistency