Concept map / Ability estimation
Share
Estimating ability from item responses
Use a response pattern to compare likelihood-based and Bayesian ability estimates.
1Six answers, one most likely ability
A person answers six items whose item curves are already known. They get items 1, 2, 3 and 5 right and miss items 4 and 6. What ability best explains that pattern?
Pick a candidate ability θ. Each right answer contributes that item’s chance of success at θ, and each wrong answer contributes its chance of failure. The model assumes the answers are independent once ability is fixed, so the product of the six numbers is the likelihood of the whole pattern. Slide θ and watch the product rise and fall.
A person with ability −1.00 would produce exactly this pattern of answers about 1 time in 117. At the best value, θ = 0.68, it is 1 time in 15.
2Estimate ability after a perfect score
This person got the five easiest items right and missed only the hardest one. The likelihood has a clear peak. Now suppose they get the hardest item right too, for a perfect score. What happens to the peak?
3The peak is where observed meets expected
At any candidate θ the item curves give an expected score. Weight each item by its discrimination a and compare that with the person's own weighted score, where u is 1 for a right answer and 0 for a wrong one. While the observed score is higher, the likelihood is still rising to the right. Where the two match, the slope is zero and the likelihood peaks.
Point at u, P or θ to light up its part of the charts above.
This person scored higher than someone at θ = −1.00 would be expected to, so the peak lies to the right.
4A prior gives a perfect score an estimate
A perfect score leaves the likelihood without a peak. A prior describes where abilities sit in the population; here it is standard normal. Multiply it by the likelihood and rescale, and the posterior peaks even for six right answers.
From the test alone, the best estimate is 0.68. Switch the prior on to see it pulled toward the population average.
5Check yourself
Each task ticks itself when the lab meets its condition.
- Build a response pattern whose most likely ability is below −1.Now: Peak at θ = 0.68
- Give a perfect score, so the likelihood has no peak, then switch the prior on to get a usable estimate.Now: 4 of 6 right, prior off
- Make an odd pattern: miss the two easiest items but get the three hardest right. Compare its ± with an ordinary pattern that has the same number right.Now: Estimate 0.68 ± 0.83
6Score the same pattern with mirt
mirt needs a data set to build a model, so the code simulates one and then fixes the item parameters at the page’s values. fscores scores the lab’s starting pattern three ways. Maximum likelihood gives 0.678 with a standard error of 0.826. With the standard normal prior, the posterior mode (MAP) is 0.407 and the posterior mean (EAP) is 0.415, which the lab rounds to 0.41 and 0.42.
Score a response pattern with mirtShow the code
library(mirt)
# The page's six 2PL items, from easiest to hardest
a <- c(1.0, 1.4, 0.8, 1.6, 1.2, 1.0)
b <- c(-2, -1, -0.3, 0.4, 1.1, 2)
# mirt needs data to build a model; the parameters are then
# fixed at the page's values, so nothing is estimated
set.seed(1)
dat <- simdata(a, -a * b, N = 500, itemtype = "2PL")
sv <- mirt(dat, 1, pars = "values")
sv$value[sv$name == "a1"] <- a
sv$value[sv$name == "d"] <- -a * b
sv$est <- FALSE
mod <- mirt(dat, 1, pars = sv, verbose = FALSE)
# The starting pattern: items 4 and 6 wrong
u <- c(1, 1, 1, 0, 1, 0)
# ML: likelihood peak. MAP and EAP: N(0, 1) prior
for (m in c("ML", "MAP", "EAP"))
print(fscores(mod, method = m, response.pattern = u))Uses mirt. Install once with install.packages(c("mirt")).
What the likelihood and the prior each do
- The maximum likelihood estimate is the θ where the person’s score, weighted by each item’s discrimination, equals the score the model expects. For the starting pattern that is θ = 0.68.
- When discriminations differ, two patterns with the same number right can get different estimates, because a right answer on a steep item counts for more.
- A perfect score, or a score of zero, has no finite maximum likelihood estimate. The standard normal prior gives both a finite one, and it pulls the starting pattern from 0.68 to a posterior mean of 0.42.
- Near a finite estimate, one over the square root of the test information approximates its standard error. For the starting pattern it is 0.83, so six items pin ability down only loosely.
- Every estimate assumes the model describes the person. Person-fit statistics flag patterns the model finds unlikely, such as missing easy items and getting hard ones right.
Recommended reading
- The Basics of Item Response Theory Using RFrank B. Baker and Seock-Ho KimChapter 6, estimating an examinee’s ability
- The Theory and Practice of Item Response TheoryR. J. de Ayala2nd ed., chapter 4, marginal maximum likelihood and person estimation