Concept map / Field-testing AI-generated items
Share
Field-testing AI-generated items
Review draft multiple-choice items with p values, corrected item-total correlations, distractor analysis and a content blueprint.
1Four draft items, one field test
An AI can draft reading items in seconds. A draft is not an item until students have answered it. A field test puts draft items into a real form, and the students’ scores on the operational items are the yardstick. The first checks are the share answering correctly (the p value), whether stronger students tend to get it right, and who picks each wrong option. Whether the item fits the blueprint is a content judgment that none of them settles.
The form has 20 operational items and four drafts, three of them from an AI. The chart sorts students into fifths by operational score. In a working item the key rises from left to right and the distractors fall.
Why does the narrator reopen the letter after reading it once?
- Option C is chosen more often by stronger students (r = 0.22).
38% of 240 students answer the draft inference item correctly. Its corrected item-total correlation is 0.26: students who answer it correctly average 11.5 of 20 on the operational items, against 9.2 for the rest. It has 1 flag for review. The reading form includes 4 of 4 draft items.
Every option’s share and correlation
| Option | Share choosing | r with rest score |
|---|---|---|
| A | 19.6% | −0.27 |
| B (key) | 38.3% | 0.26 |
| C | 30.4% | 0.22 |
| D | 11.7% | −0.36 |
More controls
2Predict what a popular distractor means
In the field test of the AI-drafted inference item, the key B is chosen by 38% of 240 students. Distractor C is chosen by 30%, and its correlation with the operational rest score is 0.22, positive like the key’s. What should the reviewers conclude?
3The item-total correlation leaves the item out
The corrected item-total correlation uses the total of the other items, here the 20 operational ones, so the item is not correlated with itself. For a right-or-wrong item it is a point-biserial correlation: the gap between the mean rest score of students who answer correctly and those who do not, scaled by the rest-score spread and the p value.
Now: (11.49 − 9.24) / 4.29 × √(0.383 × 0.617) = 0.255. Here sR is the rest-score standard deviation with n in the denominator. Applied to choosing a distractor, the same formula gives that option’s correlation, and a good distractor’s is negative.
4Check yourself
The tasks use the item buttons and checkboxes in section 1 and tick themselves when you meet them.
- Find an AI-drafted version with a corrected item-total correlation of at least 0.30 and a p value between 0.30 and 0.80.Now: Inference, draft: r = 0.26, p = 0.38
- Open the item whose distractors draw almost nobody and switch to a version in which every distractor draws at least 5% of students.Now: Inference: smallest distractor 11.7%
- Assemble the reading form: include every comprehension and vocabulary item, exclude the speed task, and leave no flags on any included item.Now: 4 items included, 6 flags on included items, speed task included
5Item statistics with psych
psych::alpha() reports each item’s p value (mean) and corrected item-total correlation (r.drop, the correlation with the total of the other items). The snippet simulates the draft inference item on 240 students, as the lab does. It prints a p value of 0.35 and an r.drop of 0.24, then each option’s share and its correlation with the rest score. Option C draws 28% and correlates +0.30, the same warning sign the lab shows.
Item statistics and distractor analysis for a draft itemShow the code
library(psych)
set.seed(91)
# 240 students, 20 operational items (2PL)
n <- 240
theta <- rnorm(n)
a <- c(0.8, 1.2, 1.0, 1.5, 0.9, 1.3, 1.1, 1.6, 0.8, 1.2,
1.4, 1.0, 0.9, 1.3, 1.1, 1.5, 1.0, 1.2, 0.9, 1.4)
b <- seq(-1.6, 1.6, length.out = 20)
oper <- sapply(1:20, function(j)
rbinom(n, 1, plogis(a[j] * (theta - b[j]))))
colnames(oper) <- paste0("op", 1:20)
# Draft inference item (nominal model), key B
slope <- c(-0.7, 0.9, 0.8, -1.0)
icpt <- c(-0.2, 0.7, 0.4, -0.6)
p <- exp(outer(theta, slope) + rep(icpt, each = n))
choice <- apply(p / rowSums(p), 1, function(pr)
sample(c("A", "B", "C", "D"), 1, prob = pr))
form <- cbind(oper, draft = as.numeric(choice == "B"))
# p value (mean) and corrected item-total correlation
st <- alpha(form, warnings = FALSE)$item.stats
round(st["draft", c("mean", "r.drop")], 2)
# Each option: share choosing it, correlation with rest score
rest <- rowSums(oper)
round(sapply(c("A", "B", "C", "D"), function(o)
c(share = mean(choice == o), r = cor(choice == o, rest))), 2)Uses psych. Install once with install.packages(c("psych")).
What a field test can and cannot tell you
- Ability is never observed, so the yardstick is the score on the other items. Leaving the item out of that total stops it from correlating with itself.
- A p value is only a start. Items near 0 or 1 tell you little about most students, and a very easy item often has distractors nobody believes.
- Read every option, not only the key. A distractor that stronger students prefer usually means a second defensible answer or a wrong key, and one chosen by fewer than about 5% of students turns a four-option item into a three-option one.
- The speed task can show acceptable numbers and still not belong on a reading-comprehension blueprint. Statistics cannot settle content.
- With 240 students a correlation has a standard error of about 0.06. Items that pass are calibrated and placed on the scale, and the same evidence logic applies to AI benchmarks.
Recommended reading
- Designing and Developing Educational AssessmentsKristen Huff, Paul Nichols, and M. Christina SchneiderEducational Measurement, 5th ed., chapter 7
- Evaluating the Instrumental Quality of LLM-Generated Assessment ItemsIgor Crk and Eren GultepePaper · 2026, an empirical comparison of item difficulty and discrimination