Concept map / AI rater audit

Share

Auditing an AI rater

Compare simulated AI essay scores with a human rater using exact and adjacent agreement, quadratic weighted kappa and the standardized mean difference.

Prerequisites Score reporting

1200 essays, two humans and an AI

An automated essay scorer is checked against human raters before its scores reach a score report. Two humans rarely agree perfectly either. So the question is whether the AI agrees with a human about as well as a second human does, and whether it leans one way: harsher, more lenient, or swayed by something outside the rubric such as essay length.

Here 200 simulated essays are scored from 1 to 6 by two humans and an AI. The table counts essays by human score (rows) and AI score (columns), and the green diagonal holds the essays that got the same score from both.

Human (rows) by AI (columns)6211111103173017231928111198442112233445566AI scoreMean AI minus human scoreShorter (105)−0.34Longer (95)+0.27All essays−0.05−1.5−1.0−0.50.00.51.01.5Score points
Exact agreementOne point apartTwo or more apart
Exact agreement42.5%two humans: 53.5%
Within one point94.5%two humans: 96.0%
Weighted kappa0.73two humans: 0.78
Standardized mean difference−0.04two humans: +0.01
  • Weighted kappa at least 0.70
  • No more than 0.10 below two humans (0.78)
  • Standardized mean difference within ±0.15

The AI gives the same score as the human on 42.5% of 200 essays and is within one point on 94.5%. The weighted kappa is 0.73 (two humans: 0.78) and the standardized mean difference is −0.04. Longer essays average +0.27 points against the human score and shorter essays −0.34.

More controls
Run 1. Only the AI’s random inconsistency is redrawn.

2Predict the kappa of an AI that is always one point low

On 50 essays scored 2 to 6 by a human, an AI gives every essay exactly one point less. Exact agreement is 0%. What is the quadratic weighted kappa?

3Kappa weighs the misses, the SMD measures the shift

Quadratic weighted kappa compares the observed disagreement with the disagreement expected by chance from each rater’s own score distribution. With squared weights a two-point miss counts four times as much as a one-point miss. The standardized mean difference measures a systematic shift, which kappa only partly penalizes.

κw=1−∑ijwijOij∑ijwijEij,wij=(i−j)2(K−1)2SMD=xˉAI−xˉH(sAI2+sH2)/2\kappa_w = 1 - \frac{\htmlClass{term-obs}{\sum_{ij} w_{ij} O_{ij}}}{\htmlClass{term-exp}{\sum_{ij} w_{ij} E_{ij}}},\quad w_{ij} = \frac{(i-j)^2}{(K-1)^2} \qquad \mathrm{SMD} = \frac{\htmlClass{term-gapm}{\bar x_{AI} - \bar x_{H}}}{\sqrt{(s^2_{AI} + s^2_{H})/2}}
κw=1−∑ijwijOij∑ijwijEij,wij=(i−j)2(K−1)2SMD=xˉAI−xˉH(sAI2+sH2)/2\begin{gathered} \kappa_w = 1 - \frac{\htmlClass{term-obs}{\sum_{ij} w_{ij} O_{ij}}}{\htmlClass{term-exp}{\sum_{ij} w_{ij} E_{ij}}},\quad w_{ij} = \frac{(i-j)^2}{(K-1)^2} \\[6pt] \mathrm{SMD} = \frac{\htmlClass{term-gapm}{\bar x_{AI} - \bar x_{H}}}{\sqrt{(s^2_{AI} + s^2_{H})/2}} \end{gathered}

Now: κ = 1 − 0.0296 / 0.1105 = 0.732, using the weighted disagreement per essay. SMD = (3.305 − 3.355) / √((1.233² + 1.120²) / 2) = −0.042.

4Check yourself

The tasks use the sliders in section 1 and tick themselves when you meet them.

  • Make the AI agree exactly with the human on at least 60% of essays.Now: exact agreement 42.5%
  • Find settings where the weighted kappa is at least 0.70 but the standardized mean difference is beyond ±0.15.Now: kappa 0.73, SMD −0.04
  • Make longer essays gain at least half a point over the human score on average while the overall SMD stays within ±0.15.Now: longer essays +0.27, overall SMD −0.04

5Agreement statistics with irr

The irr package computes percentage agreement with agree() and Cohen’s kappa with kappa2(); weight = "squared" gives quadratic weighting. The snippet simulates 200 essays with the lab’s starting AI settings. It prints exact agreement of 49%, agreement within one point of 96%, a weighted kappa of 0.78 and an SMD of −0.01, then the length effect: longer essays +0.14 and shorter essays −0.18. The lab’s own sample gives 42.5%, 94.5%, 0.73 and −0.04, and irr returns those same values when given the lab’s scores.

Agreement between an AI rater and a humanShow the code
library(irr)
set.seed(4)

# 200 essays on a 1-6 rubric, as on the page
n <- 200
quality <- rnorm(n)
words <- round(300 + 80 * rnorm(n) + 20 * quality)
score <- function(x) pmin(6, pmax(1, round(x)))
human <- score(3.5 + 1.1 * quality + rnorm(n, sd = 0.4))
# AI: no severity, 0.4 points per 100 words, noise 0.6
ai <- score(3.5 + 1.1 * quality +
            0.4 * (words - 300) / 100 + rnorm(n, sd = 0.6))
ratings <- cbind(human, ai)

agree(ratings)$value                 # exact agreement, %
agree(ratings, tolerance = 1)$value  # within one point, %
kappa2(ratings, weight = "squared")$value
pooled <- sqrt((var(ai) + var(human)) / 2)
# Standardized mean difference, AI minus human
(mean(ai) - mean(human)) / pooled

# Mean AI minus human score, shorter and longer essays
tapply(ai - human, ifelse(words < 300, "short", "long"), mean)

Uses irr. Install once with install.packages(c("irr")).

What an audit of an AI rater needs