Concept map / Classical item analysis
Share
Classical item analysis
Read item difficulty, corrected and uncorrected item-total correlations, and distractor choices from a simulated multiple-choice test.
1A ten-item test with two faulty items
After a multiple-choice test is scored, each item gets a short report. Its difficulty p is the proportion who answer it correctly. Its discrimination is the correlation between the item score (1 or 0) and the total score. A distractor analysis counts which wrong options the low, middle and high scorers choose. These are the classical test theory checks for keeping, revising or dropping an item.
Here 300 simulated examinees take a four-option test. Item 3 is scored against the wrong key, and one option on item 5 draws almost nobody. Pick an item to edit it.
Item 3
Possible miskey: the corrected correlation is negative, and more of the upper third choose D than the keyed A.
On this 10-item test, 14% of the 300 examinees choose the keyed option A on item 3, so p = 0.14. The item correlates −0.06 with the total score and −0.20 with the total of the other 9 items. Counting the item in its own total adds 0.14. The key draws 10% of the upper third and 18% of the lower third.
More controls
Show the distractor table
| Option | Lower | Middle | Upper | All 300 |
|---|---|---|---|---|
| A (key) | 0.18 | 0.13 | 0.10 | 0.14 |
| B | 0.16 | 0.12 | 0.04 | 0.11 |
| C | 0.28 | 0.13 | 0.06 | 0.16 |
| D (chosen by the upper third) | 0.38 | 0.62 | 0.80 | 0.60 |
2An item of pure guessing on a 5-item test
Change item 1 of the 5-item test so that knowing its answer has nothing to do with ability: ρ = 0, and about half the examinees answer correctly. Its corrected item-total correlation is 0.10, close to zero as it should be. What will its uncorrected correlation with the total score be?
3The item correlates with itself
Write the total as X = item + rest, where the rest score X − i adds up the other items. The covariance of the item with X is then its covariance with the rest score plus its covariance with itself, which is its own variance si2. The corrected correlation leaves that second part out. Point at a colored term to find it in the item-total chart.
Item 3. Uncorrected: (−0.167 + 0.118) / (0.344 × 2.446) = −0.058. Corrected: −0.167 / (0.344 × 2.489) = −0.195.
The covariance with the rest score is negative, so the item and the other items disagree. Its variance is 2% of the total-score variance on this 10-item test. A longer test shrinks that share.
4Check yourself
The tasks use the controls in section 1.
- Item 3 is flagged as a possible miskey. Find the option the upper third prefers in its distractor chart and make it the key, so the corrected correlation turns positive.Now: Item 3 key A, corrected correlation −0.20
- Almost nobody chooses one distractor on item 5. Raise its pull until at least 10% of the lower third choose it.Now: Option D of item 5 is chosen by 0% of the lower third and 0% overall
- On the 5-item test, make an item whose uncorrected correlation is at least 0.30 and whose corrected correlation is below 0.10.Now: The test has 10 items
5Item statistics in psych
The code simulates a test like the lab’s first 10 items, scores it against a key with one error, and asks psych::alpha for the item statistics. In its output mean is the difficulty, raw.r the uncorrected and r.drop the corrected item-total correlation. The two tables are distractor analyses by thirds of the total score. R draws its own sample, so the numbers differ a little from the lab’s: item 3 has an r.drop of −0.18, most of its upper third choose D, and nobody chooses D on item 5.
Item statistics and distractor tablesShow the code
library(psych)
set.seed(1)
# 300 examinees and the first 10 items of the lab
n <- 300
p <- c(.75, .7, .6, .65, .7, .5, .65, .4, .55, .7)
rho <- c(.6, .65, .6, .7, .6, .5, .75, .55, .65, .6)
truth <- c("B", "A", "D", "C", "B", "A", "D", "C", "B", "A")
# Item 3 is miskeyed: the answer is D but the key says A
key <- replace(truth, 3, "A")
theta <- rnorm(n)
resp <- sapply(1:10, function(i) {
e <- sqrt(1 - rho[i]^2) * rnorm(n)
knows <- rho[i] * theta + e > qnorm(1 - p[i])
# Item 5 has a third distractor that almost nobody picks
w <- if (i == 5) c(1, .8, .02) else c(1, .8, .6)
guess <- sample(setdiff(LETTERS[1:4], truth[i]), n, TRUE, w)
ifelse(knows, truth[i], guess)
})
scored <- sapply(1:10, function(i) 1 * (resp[, i] == key[i]))
# mean is the difficulty p; r.drop is the corrected correlation
st <- alpha(scored, warnings = FALSE)$item.stats
round(st[, c("mean", "raw.r", "r.drop")], 2)
# Distractor tables by thirds of the total score
third <- ceiling(3 * rank(rowSums(scored), ties = "first") / n)
tab <- function(i) table(third, factor(resp[, i], LETTERS[1:4]))
round(prop.table(tab(3), 1), 2)
round(prop.table(tab(5), 1), 2)Uses psych. Install once with install.packages(c("psych")).
What the item report tells you
- The difficulty index p is the proportion who choose the key, so a higher p means an easier item. Give the same item to a stronger group and p goes up.
- The uncorrected item-total correlation counts the item in its own total. On the 5-item test an item that measures nothing correlates 0.44 with the total, and its corrected correlation is 0.10.
- On a longer test the item is a smaller share of the total, and the two correlations draw together. On 40 items they differ by 0.03 to 0.05.
- A negative corrected correlation, with the upper third preferring a distractor, usually means the key is wrong. Check the key before rewriting the item.
- A working distractor draws more of the lower third than the upper third, and one chosen by under 5% of examinees adds nothing. IRT calibration replaces p and the item-total correlation with parameters on the ability scale, and field-testing AI-generated items runs these same checks on new item pools.
Recommended reading
- A review of multiple-choice item-writing guidelines for classroom assessmentThomas M. Haladyna, Steven M. Downing, and Michael C. RodriguezPaper · Applied Measurement in Education, 15(3), 2002