Concept map / Evaluating AI benchmarks

Share

Evaluating AI benchmarks

See how task weights, exposure to public tasks and the number of tasks change a benchmark ranking and the uncertainty around it.

1Two models, a public set and a held-out set

A benchmark is a set of tasks plus a rule for combining the results into one score. That makes it a test, and the questions from validity and score reporting apply. Which tasks count, and how much? Could a model have seen them before? Is a lead of a few points bigger than the noise from the particular tasks chosen?

Two fictional models answer reasoning, writing and coding tasks. Model A is stronger at reasoning, Model B at writing and coding. The chart shows Model A’s score minus Model B’s, on a public task set and on a held-out set that no model has seen.

−30−20−100+10+20+30Model A minus Model B, with 95% intervalDifference in weighted score (percentage points)Public set−7.3Held-out set−5.6
Weighted difference95% interval, paired by task

On the held-out set, Model B ranks first by 5.6 points, with a 95% interval from −13.9 to +2.7, which includes zero, so this many tasks cannot tell the models apart. On the public set, Model B ranks first.

Percent correct in each category

Model A has seen none of the public tasks.

CategoryWeightPublicHeld-out
ABAB
Reasoning35.0%82.566.381.361.3
Writing32.5%56.380.055.078.8
Coding32.5%55.071.356.371.3
Weighted100%65.072.364.670.2

2Predict the margin of error on 240 tasks

Both models answer the same 80 held-out tasks in each of three categories, 240 tasks in all, and the categories count equally. Model B’s weighted score is 6.3 percentage points higher. How wide is the 95% interval for that difference?

3Weights set the score, task counts set its precision

A benchmark score is a weighted average of category accuracies, so the weights decide what it rewards. Both models answer the same tasks, so the difference is analyzed task by task. Its uncertainty depends on how much the per-task differences vary (s²) and on the number of tasks.

S=∑kwk p^kSE(SA−SB)=∑kwk2 sd,k2nkS = \sum_k \htmlClass{term-w}{w_k}\,\hat p_k \qquad \mathrm{SE}(S_A - S_B) = \sqrt{\sum_k \htmlClass{term-w}{w_k^2}\,\frac{\htmlClass{term-sd}{s^2_{d,k}}}{\htmlClass{term-n}{n_k}}}
S=∑kwk p^kSE(SA−SB)=∑kwk2 sd,k2nk\begin{gathered} S = \sum_k \htmlClass{term-w}{w_k}\,\hat p_k \\[4pt] \mathrm{SE}(S_A - S_B) = \sqrt{\sum_k \htmlClass{term-w}{w_k^2}\,\frac{\htmlClass{term-sd}{s^2_{d,k}}}{\htmlClass{term-n}{n_k}}} \end{gathered}

Now, held-out: SA − SB = 64.6 − 70.2 = −5.6 points, SE = 4.24 points, 95% interval = difference ± 1.96 × SE = ± 8.3 points.

A common mistake is to treat a narrow interval as a guard against contamination. Exposure does not appear in the formula. Four times as many tasks halve the standard error, and a contaminated benchmark can be precise and still wrong.

4Check yourself

The tasks use the sliders in section 1 and tick themselves when you meet them.

  • Change only the task weights until Model A ranks first on the held-out set.Now: reasoning weight 35%, held-out leader Model B (64.6 vs 70.2)
  • Find settings where Model A ranks first on the public set but second on the held-out set.Now: public leader Model B, held-out leader Model B
  • Add tasks until the 95% interval for the held-out difference excludes zero.Now: 80 tasks per category, held-out interval −13.9 to +2.7 points

5Weighted scores and a paired interval in base R

Base R is enough. The snippet simulates the held-out set with the lab’s starting weights: 80 tasks per category, each with an easiness that both models share. It prints the category accuracies, then the weighted difference of −5.7 points with a 95% interval from −13.3 to +1.9. The lab’s own sample gives −5.6, from −13.9 to +2.7.

Weighted benchmark scores with a paired intervalShow the code
set.seed(3)

# Held-out set: 80 tasks per category, as on the page
n <- 80
acc_a <- c(reasoning = 0.85, writing = 0.61, coding = 0.63)
acc_b <- c(reasoning = 0.69, writing = 0.82, coding = 0.78)
w <- c(0.35, 0.325, 0.325)

res <- sapply(1:3, function(k) {
  # Each task has an easiness that both models share
  ease <- rnorm(n)
  a <- rbinom(n, 1, plogis(qlogis(acc_a[k]) + ease))
  b <- rbinom(n, 1, plogis(qlogis(acc_b[k]) + ease))
  c(a = mean(a), b = mean(b), v = var(a - b) / n)
})
colnames(res) <- names(acc_a)
round(res[c("a", "b"), ], 2)

# Weighted difference (A minus B) and a paired 95% interval
d <- sum(w * (res["a", ] - res["b", ]))
se <- sqrt(sum(w^2 * res["v", ]))
round(100 * c(diff = d, low = d - 1.96 * se,
              high = d + 1.96 * se), 1)

Uses base R only.

What a benchmark score depends on