Concept map / Test characteristic curve

Share

The test characteristic curve

Add response probabilities across items to find the expected number correct at an ability level.

Prerequisites IRT: item curves

1From item chances to a total score

Each item characteristic curve gives one student’s chance on one item, but a score report shows a total. A student with ability θ = 0.8 takes a six-item test. What total should we expect, and what changes on a harder version of the test?

The test characteristic curve (TCC) gives the expected number correct at each ability: the average total a student at that ability would earn over many sittings of the same form. Move θ, switch forms, and give the test to compare observed scores with the expected one.

0123456−4−3−2−101234Expected number correctAbility θ
Expected score, mixed formObserved scores at this ability
  • Item 1a 1.2, b −2.50.98
  • Item 2a 0.9, b −1.60.90
  • Item 3a 2.0, b −0.80.96
  • Item 4a 0.8, b 0.30.60
  • Item 5a 1.4, b 1.00.43
  • Item 6a 1.3, b 2.10.16
  • Sum4.02

At ability 0.8, the mixed form has an expected score of 4.02 out of 6. Give the test to see observed scores, which are always whole numbers. Moving θ or switching forms clears them.

2Predict where a gain in ability pays off

Two students take the mixed form. Ana’s ability rises from θ = 0 to θ = 1. Ben’s rises from θ = 2.5 to θ = 3.5. Both gain one unit of ability. Whose expected number correct rises more?

3Six item chances add up to the curve

An item scored 1 or 0 has an expected score equal to its chance of a correct answer, so the expected total is the sum of the six chances. Point at a symbol to light up its part of the lab.

E⁡[X∣θ]  =  ∑i=16Pi(θ),Pi(θ)=11+e−ai(θ−bi)\operatorname{E}[X \mid \htmlClass{term-theta}{\theta}] \;=\; \sum_{i=1}^{6} \htmlClass{term-P}{P_i(\theta)}, \qquad \htmlClass{term-P}{P_i(\theta)} = \frac{1}{1 + e^{-a_i(\htmlClass{term-theta}{\theta} - b_i)}}
E⁡[X∣θ]  =  ∑i=16Pi(θ)Pi(θ)=11+e−ai(θ−bi)\begin{gathered} \operatorname{E}[X \mid \htmlClass{term-theta}{\theta}] \;=\; \sum_{i=1}^{6} \htmlClass{term-P}{P_i(\theta)} \\[6pt] \htmlClass{term-P}{P_i(\theta)} = \frac{1}{1 + e^{-a_i(\htmlClass{term-theta}{\theta} - b_i)}} \end{gathered}
E⁡[X∣θ=0.8]=0.98+0.90+0.96+0.60+0.43+0.16≈4.02\operatorname{E}[X \mid \htmlClass{term-theta}{\theta = 0.8}] = \htmlClass{term-P}{0.98} + \htmlClass{term-P}{0.90} + \htmlClass{term-P}{0.96} + \htmlClass{term-P}{0.60} + \htmlClass{term-P}{0.43} + \htmlClass{term-P}{0.16} \approx 4.02
E⁡[X∣θ=0.8]=0.98+0.90+0.96+0.60+0.43+0.16≈4.02\begin{aligned} \operatorname{E}[X \mid \htmlClass{term-theta}{\theta = 0.8}] &= \htmlClass{term-P}{0.98} + \htmlClass{term-P}{0.90} + \htmlClass{term-P}{0.96} \\ &\quad + \htmlClass{term-P}{0.60} + \htmlClass{term-P}{0.43} + \htmlClass{term-P}{0.16} \\ &\approx 4.02 \end{aligned}

Each Pi depends only on θ − bi. Add the same amount to every b, as the harder form does, and the whole curve slides right by that amount without changing shape.

4Check yourself

Each task ticks itself when the lab meets its condition.

  • On the mixed form, find an ability where the expected score is half the test, 3 out of 6 (within 0.05).Now: Mixed form, θ = 0.8, expected score 4.02
  • A student at θ = 0.8 expects 4.02 on the mixed form. On the harder form, find the ability that gives the same expected score (within 0.05).Now: Mixed form, θ = 0.8, expected score 4.02
  • Give the test at least 20 times at one ability and form, and get the average observed score within 0.3 of the expected score. Moving θ or the form starts the count again.Now: no administrations at this setting yet

5The same curve by hand and with mirt

The code adds the six item chances at θ = 0.8 by hand, giving 4.02 as in the lab, and shows that the harder form gives the same expected score at θ = 2.0. It then fits the six items to simulated answers with mirt and reads the fitted TCC with expected.test(). The fitted curve gives 3.04, 4.07 and 5.05 at θ = −0.3, 0.8 and 2.0, close to the true values of 2.99, 4.02 and 5.02.

Compute a test characteristic curve by hand and with mirtShow the code
library(mirt)

# The six items of the mixed form (2PL, no 1.7 constant)
a <- c(1.2, 0.9, 2.0, 0.8, 1.4, 1.3)
b <- c(-2.5, -1.6, -0.8, 0.3, 1.0, 2.1)
p <- function(theta) 1 / (1 + exp(-a * (theta - b)))

# Expected number correct at theta = 0.8: add the chances
round(p(0.8), 2)
round(sum(p(0.8)), 2)
# The harder form adds 1.2 to every b, so theta = 2.0 there
# gives the same expected score
round(sum(1 / (1 + exp(-a * (2.0 - (b + 1.2))))), 2)

# Fit the six items to 3000 simulated students with mirt,
# then read the fitted curve at three abilities
set.seed(2)
resp <- simdata(matrix(a), matrix(-a * b), N = 3000,
                itemtype = "2PL")
fit <- mirt(resp, 1, itemtype = "2PL", verbose = FALSE)
round(expected.test(fit, Theta = matrix(c(-0.3, 0.8, 2))), 2)

Uses mirt. Install once with install.packages(c("mirt")).

What the curve says about scores