Concept map / Classical item analysis

Share

Classical item analysis

Read item difficulty, corrected and uncorrected item-total correlations, and distractor choices from a simulated multiple-choice test.

Prerequisites Reliability, Correlation

1A ten-item test with two faulty items

After a multiple-choice test is scored, each item gets a short report. Its difficulty p is the proportion who answer it correctly. Its discrimination is the correlation between the item score (1 or 0) and the total score. A distractor analysis counts which wrong options the low, middle and high scorers choose. These are the classical test theory checks for keeping, revising or dropping an item.

Here 300 simulated examinees take a four-option test. Item 3 is scored against the wrong key, and one option on item 5 draws almost nobody. Pick an item to edit it.

Test length
−0.4−0.20.00.20.40.60.81.012345678910Item-total correlationItemalpha 0.72
Uncorrected: the total includes the itemCorrected: total of the other items0.20 review line

Item 3

Scoring key
0.000.250.500.751.00LowerMiddleUpperProportion choosing each optionThird of the total scoreD 80%A (key) 10%C 6%B 4%

Possible miskey: the corrected correlation is negative, and more of the upper third choose D than the keyed A.

On this 10-item test, 14% of the 300 examinees choose the keyed option A on item 3, so p = 0.14. The item correlates −0.06 with the total score and −0.20 with the total of the other 9 items. Counting the item in its own total adds 0.14. The key draws 10% of the upper third and 18% of the lower third.

More controls
Show the distractor table
Proportion of each total-score group choosing each option of item 3
OptionLowerMiddleUpperAll 300
A (key)0.180.130.100.14
B0.160.120.040.11
C0.280.130.060.16
D (chosen by the upper third)0.380.620.800.60

2An item of pure guessing on a 5-item test

Change item 1 of the 5-item test so that knowing its answer has nothing to do with ability: ρ = 0, and about half the examinees answer correctly. Its corrected item-total correlation is 0.10, close to zero as it should be. What will its uncorrected correlation with the total score be?

3The item correlates with itself

Write the total as X = item + rest, where the rest score X − i adds up the other items. The covariance of the item with X is then its covariance with the rest score plus its covariance with itself, which is its own variance si2. The corrected correlation leaves that second part out. Point at a colored term to find it in the item-total chart.

riX  =  cov⁡(i, X−i)  +  si2si sXr_{iX} \;=\; \frac{\htmlClass{term-rest}{\operatorname{cov}(i,\,X-i)} \;+\; \htmlClass{term-own}{s_i^{2}}}{\htmlClass{term-sd}{s_i\, s_X}}
ri(X−i)  =  cov⁡(i, X−i)si sX−ir_{i(X-i)} \;=\; \frac{\htmlClass{term-rest}{\operatorname{cov}(i,\,X-i)}}{\htmlClass{term-sd}{s_i\, s_{X-i}}}

Item 3. Uncorrected: (−0.167 + 0.118) / (0.344 × 2.446) = −0.058. Corrected: −0.167 / (0.344 × 2.489) = −0.195.

The covariance with the rest score is negative, so the item and the other items disagree. Its variance is 2% of the total-score variance on this 10-item test. A longer test shrinks that share.

4Check yourself

The tasks use the controls in section 1.

  • Item 3 is flagged as a possible miskey. Find the option the upper third prefers in its distractor chart and make it the key, so the corrected correlation turns positive.Now: Item 3 key A, corrected correlation −0.20
  • Almost nobody chooses one distractor on item 5. Raise its pull until at least 10% of the lower third choose it.Now: Option D of item 5 is chosen by 0% of the lower third and 0% overall
  • On the 5-item test, make an item whose uncorrected correlation is at least 0.30 and whose corrected correlation is below 0.10.Now: The test has 10 items

5Item statistics in psych

The code simulates a test like the lab’s first 10 items, scores it against a key with one error, and asks psych::alpha for the item statistics. In its output mean is the difficulty, raw.r the uncorrected and r.drop the corrected item-total correlation. The two tables are distractor analyses by thirds of the total score. R draws its own sample, so the numbers differ a little from the lab’s: item 3 has an r.drop of −0.18, most of its upper third choose D, and nobody chooses D on item 5.

Item statistics and distractor tablesShow the code
library(psych)
set.seed(1)
# 300 examinees and the first 10 items of the lab
n <- 300
p <- c(.75, .7, .6, .65, .7, .5, .65, .4, .55, .7)
rho <- c(.6, .65, .6, .7, .6, .5, .75, .55, .65, .6)
truth <- c("B", "A", "D", "C", "B", "A", "D", "C", "B", "A")
# Item 3 is miskeyed: the answer is D but the key says A
key <- replace(truth, 3, "A")
theta <- rnorm(n)
resp <- sapply(1:10, function(i) {
  e <- sqrt(1 - rho[i]^2) * rnorm(n)
  knows <- rho[i] * theta + e > qnorm(1 - p[i])
  # Item 5 has a third distractor that almost nobody picks
  w <- if (i == 5) c(1, .8, .02) else c(1, .8, .6)
  guess <- sample(setdiff(LETTERS[1:4], truth[i]), n, TRUE, w)
  ifelse(knows, truth[i], guess)
})
scored <- sapply(1:10, function(i) 1 * (resp[, i] == key[i]))
# mean is the difficulty p; r.drop is the corrected correlation
st <- alpha(scored, warnings = FALSE)$item.stats
round(st[, c("mean", "raw.r", "r.drop")], 2)
# Distractor tables by thirds of the total score
third <- ceiling(3 * rank(rowSums(scored), ties = "first") / n)
tab <- function(i) table(third, factor(resp[, i], LETTERS[1:4]))
round(prop.table(tab(3), 1), 2)
round(prop.table(tab(5), 1), 2)

Uses psych. Install once with install.packages(c("psych")).

What the item report tells you