Concept map / Field-testing AI-generated items

Share

Field-testing AI-generated items

Review draft multiple-choice items with p values, corrected item-total correlations, distractor analysis and a content blueprint.

1Four draft items, one field test

An AI can draft reading items in seconds. A draft is not an item until students have answered it. A field test puts draft items into a real form, and the students’ scores on the operational items are the yardstick. The first checks are the share answering correctly (the p value), whether stronger students tend to get it right, and who picks each wrong option. Whether the item fits the blueprint is a content judgment that none of them settles.

The form has 20 operational items and four drafts, three of them from an AI. The chart sorts students into fifths by operational score. In a working item the key rises from left to right and the distractors fall.

AI draft · Comprehension · key B

Why does the narrator reopen the letter after reading it once?

0.000.250.500.751.00Lowest2nd3rd4thHighestShare choosing each optionFifth of students by operational rest scoreAB (key)CD
p value (share correct)0.38
Corrected item-total r0.26
Flags1
  • Option C is chosen more often by stronger students (r = 0.22).

38% of 240 students answer the draft inference item correctly. Its corrected item-total correlation is 0.26: students who answer it correctly average 11.5 of 20 on the operational items, against 9.2 for the rest. It has 1 flag for review. The reading form includes 4 of 4 draft items.

Every option’s share and correlation
OptionShare choosingr with rest score
A19.6%−0.27
B (key)38.3%0.26
C30.4%0.22
D11.7%−0.36
More controls
Sample 1

2Predict what a popular distractor means

In the field test of the AI-drafted inference item, the key B is chosen by 38% of 240 students. Distractor C is chosen by 30%, and its correlation with the operational rest score is 0.22, positive like the key’s. What should the reviewers conclude?

3The item-total correlation leaves the item out

The corrected item-total correlation uses the total of the other items, here the 20 operational ones, so the item is not correlated with itself. For a right-or-wrong item it is a point-biserial correlation: the gap between the mean rest score of students who answer correctly and those who do not, scaled by the rest-score spread and the p value.

ri(X−i)=Rˉ1−Rˉ0sRp (1−p)r_{i(X-i)} = \frac{\htmlClass{term-m1}{\bar R_{1}} - \htmlClass{term-m0}{\bar R_{0}}}{s_R}\sqrt{\htmlClass{term-pv}{p}\,(1-\htmlClass{term-pv}{p})}

Now: (11.49 − 9.24) / 4.29 × √(0.383 × 0.617) = 0.255. Here sR is the rest-score standard deviation with n in the denominator. Applied to choosing a distractor, the same formula gives that option’s correlation, and a good distractor’s is negative.

4Check yourself

The tasks use the item buttons and checkboxes in section 1 and tick themselves when you meet them.

  • Find an AI-drafted version with a corrected item-total correlation of at least 0.30 and a p value between 0.30 and 0.80.Now: Inference, draft: r = 0.26, p = 0.38
  • Open the item whose distractors draw almost nobody and switch to a version in which every distractor draws at least 5% of students.Now: Inference: smallest distractor 11.7%
  • Assemble the reading form: include every comprehension and vocabulary item, exclude the speed task, and leave no flags on any included item.Now: 4 items included, 6 flags on included items, speed task included

5Item statistics with psych

psych::alpha() reports each item’s p value (mean) and corrected item-total correlation (r.drop, the correlation with the total of the other items). The snippet simulates the draft inference item on 240 students, as the lab does. It prints a p value of 0.35 and an r.drop of 0.24, then each option’s share and its correlation with the rest score. Option C draws 28% and correlates +0.30, the same warning sign the lab shows.

Item statistics and distractor analysis for a draft itemShow the code
library(psych)
set.seed(91)

# 240 students, 20 operational items (2PL)
n <- 240
theta <- rnorm(n)
a <- c(0.8, 1.2, 1.0, 1.5, 0.9, 1.3, 1.1, 1.6, 0.8, 1.2,
       1.4, 1.0, 0.9, 1.3, 1.1, 1.5, 1.0, 1.2, 0.9, 1.4)
b <- seq(-1.6, 1.6, length.out = 20)
oper <- sapply(1:20, function(j)
  rbinom(n, 1, plogis(a[j] * (theta - b[j]))))
colnames(oper) <- paste0("op", 1:20)

# Draft inference item (nominal model), key B
slope <- c(-0.7, 0.9, 0.8, -1.0)
icpt <- c(-0.2, 0.7, 0.4, -0.6)
p <- exp(outer(theta, slope) + rep(icpt, each = n))
choice <- apply(p / rowSums(p), 1, function(pr)
  sample(c("A", "B", "C", "D"), 1, prob = pr))
form <- cbind(oper, draft = as.numeric(choice == "B"))

# p value (mean) and corrected item-total correlation
st <- alpha(form, warnings = FALSE)$item.stats
round(st["draft", c("mean", "r.drop")], 2)
# Each option: share choosing it, correlation with rest score
rest <- rowSums(oper)
round(sapply(c("A", "B", "C", "D"), function(o)
  c(share = mean(choice == o), r = cor(choice == o, rest))), 2)

Uses psych. Install once with install.packages(c("psych")).

What a field test can and cannot tell you