Concept map / Ability estimation

Share

Estimating ability from item responses

Use a response pattern to compare likelihood-based and Bayesian ability estimates.

1Six answers, one most likely ability

A person answers six items whose item curves are already known. They get items 1, 2, 3 and 5 right and miss items 4 and 6. What ability best explains that pattern?

Pick a candidate ability θ. Each right answer contributes that item’s chance of success at θ, and each wrong answer contributes its chance of failure. The model assumes the answers are independent once ability is fixed, so the product of the six numbers is the likelihood of the whole pattern. Slide θ and watch the product rise and fall.

00.51-4-3-2-101234Ability θChance of each answer this person gave
Chance of a right answerChance of a wrong answer
0max-4-3-2-101234Ability θLikelihood: all six multiplied togetherpeak at θ = 0.68

A person with ability −1.00 would produce exactly this pattern of answers about 1 time in 117. At the best value, θ = 0.68, it is 1 time in 15.

2Estimate ability after a perfect score

This person got the five easiest items right and missed only the hardest one. The likelihood has a clear peak. Now suppose they get the hardest item right too, for a perfect score. What happens to the peak?

0max-4-2024LikelihoodAbility θ5 of 6 right

3The peak is where observed meets expected

At any candidate θ the item curves give an expected score. Weight each item by its discrimination a and compare that with the person's own weighted score, where u is 1 for a right answer and 0 for a wrong one. While the observed score is higher, the likelihood is still rising to the right. Where the two match, the slope is zero and the likelihood peaks.

slope at θ  =  ∑iai (ui−Pi(θ))\text{slope at } \htmlClass{term-theta}{\theta} \;=\; \sum_i a_i\,\bigl(\htmlClass{term-u}{u_i} - \htmlClass{term-P}{P_i(\theta)}\bigr)

Point at u, P or θ to light up its part of the charts above.

Observed score (weighted)4.40
Expected at θ = −1.002.01
Slope2.39

This person scored higher than someone at θ = −1.00 would be expected to, so the peak lies to the right.

Steps: 0

4A prior gives a perfect score an estimate

A perfect score leaves the likelihood without a peak. A prior describes where abilities sit in the population; here it is standard normal. Multiply it by the likelihood and rescale, and the posterior peaks even for six right answers.

-4-3-2-101234Ability θEach curve scaled to its own peak
Likelihood (the test)Thick bar: MLE ± one approximate standard error
Maximum likelihood (MLE)0.68 ± 0.83
Posterior peak (MAP)prior off
Posterior mean (EAP)prior off

From the test alone, the best estimate is 0.68. Switch the prior on to see it pulled toward the population average.

5Check yourself

Each task ticks itself when the lab meets its condition.

  • Build a response pattern whose most likely ability is below −1.Now: Peak at θ = 0.68
  • Give a perfect score, so the likelihood has no peak, then switch the prior on to get a usable estimate.Now: 4 of 6 right, prior off
  • Make an odd pattern: miss the two easiest items but get the three hardest right. Compare its ± with an ordinary pattern that has the same number right.Now: Estimate 0.68 ± 0.83

6Score the same pattern with mirt

mirt needs a data set to build a model, so the code simulates one and then fixes the item parameters at the page’s values. fscores scores the lab’s starting pattern three ways. Maximum likelihood gives 0.678 with a standard error of 0.826. With the standard normal prior, the posterior mode (MAP) is 0.407 and the posterior mean (EAP) is 0.415, which the lab rounds to 0.41 and 0.42.

Score a response pattern with mirtShow the code
library(mirt)
# The page's six 2PL items, from easiest to hardest
a <- c(1.0, 1.4, 0.8, 1.6, 1.2, 1.0)
b <- c(-2, -1, -0.3, 0.4, 1.1, 2)
# mirt needs data to build a model; the parameters are then
# fixed at the page's values, so nothing is estimated
set.seed(1)
dat <- simdata(a, -a * b, N = 500, itemtype = "2PL")
sv <- mirt(dat, 1, pars = "values")
sv$value[sv$name == "a1"] <- a
sv$value[sv$name == "d"] <- -a * b
sv$est <- FALSE
mod <- mirt(dat, 1, pars = sv, verbose = FALSE)

# The starting pattern: items 4 and 6 wrong
u <- c(1, 1, 1, 0, 1, 0)
# ML: likelihood peak. MAP and EAP: N(0, 1) prior
for (m in c("ML", "MAP", "EAP"))
  print(fscores(mod, method = m, response.pattern = u))

Uses mirt. Install once with install.packages(c("mirt")).

What the likelihood and the prior each do