Concept map / AI rater audit
Share
Auditing an AI rater
Compare simulated AI essay scores with a human rater using exact and adjacent agreement, quadratic weighted kappa and the standardized mean difference.
1200 essays, two humans and an AI
An automated essay scorer is checked against human raters before its scores reach a score report. Two humans rarely agree perfectly either. So the question is whether the AI agrees with a human about as well as a second human does, and whether it leans one way: harsher, more lenient, or swayed by something outside the rubric such as essay length.
Here 200 simulated essays are scored from 1 to 6 by two humans and an AI. The table counts essays by human score (rows) and AI score (columns), and the green diagonal holds the essays that got the same score from both.
- Weighted kappa at least 0.70
- No more than 0.10 below two humans (0.78)
- Standardized mean difference within ±0.15
The AI gives the same score as the human on 42.5% of 200 essays and is within one point on 94.5%. The weighted kappa is 0.73 (two humans: 0.78) and the standardized mean difference is −0.04. Longer essays average +0.27 points against the human score and shorter essays −0.34.
More controls
2Predict the kappa of an AI that is always one point low
On 50 essays scored 2 to 6 by a human, an AI gives every essay exactly one point less. Exact agreement is 0%. What is the quadratic weighted kappa?
3Kappa weighs the misses, the SMD measures the shift
Quadratic weighted kappa compares the observed disagreement with the disagreement expected by chance from each rater’s own score distribution. With squared weights a two-point miss counts four times as much as a one-point miss. The standardized mean difference measures a systematic shift, which kappa only partly penalizes.
Now: κ = 1 − 0.0296 / 0.1105 = 0.732, using the weighted disagreement per essay. SMD = (3.305 − 3.355) / √((1.233² + 1.120²) / 2) = −0.042.
4Check yourself
The tasks use the sliders in section 1 and tick themselves when you meet them.
- Make the AI agree exactly with the human on at least 60% of essays.Now: exact agreement 42.5%
- Find settings where the weighted kappa is at least 0.70 but the standardized mean difference is beyond ±0.15.Now: kappa 0.73, SMD −0.04
- Make longer essays gain at least half a point over the human score on average while the overall SMD stays within ±0.15.Now: longer essays +0.27, overall SMD −0.04
5Agreement statistics with irr
The irr package computes percentage agreement with agree() and Cohen’s kappa with kappa2(); weight = "squared" gives quadratic weighting. The snippet simulates 200 essays with the lab’s starting AI settings. It prints exact agreement of 49%, agreement within one point of 96%, a weighted kappa of 0.78 and an SMD of −0.01, then the length effect: longer essays +0.14 and shorter essays −0.18. The lab’s own sample gives 42.5%, 94.5%, 0.73 and −0.04, and irr returns those same values when given the lab’s scores.
Agreement between an AI rater and a humanShow the code
library(irr)
set.seed(4)
# 200 essays on a 1-6 rubric, as on the page
n <- 200
quality <- rnorm(n)
words <- round(300 + 80 * rnorm(n) + 20 * quality)
score <- function(x) pmin(6, pmax(1, round(x)))
human <- score(3.5 + 1.1 * quality + rnorm(n, sd = 0.4))
# AI: no severity, 0.4 points per 100 words, noise 0.6
ai <- score(3.5 + 1.1 * quality +
0.4 * (words - 300) / 100 + rnorm(n, sd = 0.6))
ratings <- cbind(human, ai)
agree(ratings)$value # exact agreement, %
agree(ratings, tolerance = 1)$value # within one point, %
kappa2(ratings, weight = "squared")$value
pooled <- sqrt((var(ai) + var(human)) / 2)
# Standardized mean difference, AI minus human
(mean(ai) - mean(human)) / pooled
# Mean AI minus human score, shorter and longer essays
tapply(ai - human, ifelse(words < 300, "short", "long"), mean)Uses irr. Install once with install.packages(c("irr")).
What an audit of an AI rater needs
- Exact agreement, agreement within one point, weighted kappa and the standardized mean difference (SMD) each catch a different failure, so a report needs all four.
- Human raters disagree too. A common guideline asks that AI-human kappa be at least 0.70 and no more than 0.10 below the kappa between two humans (Williamson, Xi and Breyer, 2012).
- An AI that scores every essay one point low agrees exactly on none of them, yet its weighted kappa is 0.72. The SMD of −0.88 shows the shift; it should usually stay within ±0.15.
- An AI that rewards length can match the human on average while it scores long essays up and short ones down. Check agreement within groups of essays and of students.
- The same questions, what is being measured and for whom it works, apply to AI-generated items and to AI benchmarks.
Recommended reading
- Test Administration and ScoringMark D. Shermis, Allan S. Cohen, and Jordan M. WheelerEducational Measurement, 5th ed., chapter 8