Concept map / IRT calibration and model fit
Share
Item response theory calibration and model fit
Estimate Rasch item difficulties from simulated responses and inspect an item-pair diagnostic.
1From right and wrong answers to item difficulties
Earlier pages took each item characteristic curve as known. In practice all you have is a table of right and wrong answers. How do you recover each item’s difficulty from it, and how do you tell whether the model fits?
Calibration estimates item parameters from response data. Here the Rasch difficulties are fitted by marginal maximum likelihood, which averages each student’s likelihood over a standard normal spread of ability instead of estimating every ability first. The fit sees only the answers; tick the box to compare it with the true difficulties. Below the fit, a second chart asks whether Items 1 and 2 are both right more often than the model predicts, a sign of local dependence.
Show percent correct by item
| Item | Starting b | Observed correct | Model correct |
|---|---|---|---|
| 1 | −1.29 | 78.3% | 74.5% |
| 2 | −0.78 | 68.7% | 65.7% |
| 3 | −0.34 | 58.3% | 56.9% |
| 4 | 0.20 | 45.0% | 45.9% |
| 5 | 0.74 | 32.3% | 35.1% |
| 6 | 1.45 | 19.0% | 22.9% |
Are Items 1 and 2 right together too often?
300 students answered six items. These are starting estimates from each item’s percent correct. Fit to estimate the difficulties. Items 1 and 2 were both correct for 56.7% of students; the model predicts 52.2%, a gap of 4.5 points.
2What shared answers do to the fit
Suppose every student answers Items 1 and 2 from one shared random draw. Each item’s chance of a correct answer at a given ability stays the same, but the two answers now move together beyond what ability explains. You fit the Rasch model as usual. What do you see?
3What the fit maximizes, and what it predicts
Marginal maximum likelihood averages each student’s response pattern over a standard normal ability distribution φ, then picks the difficulties b that make the observed patterns most likely. Each fitting step is one round of the EM algorithm. Point at a symbol to light up its part of the lab.
The fitted model also predicts how often two items are both correct. Under local independence two answers are linked only through ability, so the prediction multiplies the two item curves and averages over abilities.
A gap of a point or two can come from sampling alone, so do not read dependence into one small gap. A gap that grows with d and survives new responses means the two items share something beyond ability, such as a reading passage.
4Check yourself
Each task ticks itself when the lab meets its condition.
- With 600 students, fit until the estimates stop changing, reveal the true difficulties, and see every fitted difficulty land within 0.30 of its true value.Now: 300 students, 0 steps, not settled, reveal the true difficulties to see the largest miss
- Raise the shared tendency for Items 1 and 2 and fit until the estimates settle, so that the pair is correct together at least 5 points more often than the model predicts.Now: shared tendency 0%, not settled, gap 4.5 points
- With 150 students and no shared tendency, fit to convergence and draw new responses until some fitted difficulty misses its true value by more than 0.30.Now: 300 students, shared tendency 0%, not settled, reveal the true difficulties to see the largest miss
5Calibrate and check every pair in mirt
The code simulates 600 students answering the six items, with Items 1 and 2 sharing one draw for half of them, as with d at 50%. mirt also fits by marginal maximum likelihood and prints difficulties of −1.62, −1.00, −0.37, 0.38, 0.78 and 1.57; the true values are −1.55, −0.95, −0.35, 0.20, 0.80 and 1.45. residuals(fit, type = "LD") then gives a local dependence chi-square for every pair of items. Items 1 and 2 give 53.75 and a residual correlation of 0.30, and every other pair stays below 4.2. With 15 pairs, a few values just above 3.84, the 5% cutoff for one degree of freedom, are expected by chance.
Calibrate Rasch items and check local dependence with mirtShow the code
library(mirt)
# 600 students answer six Rasch items with the page's
# difficulties. Items 1 and 2 share one random draw for
# half of the students, as with the slider at 50%.
set.seed(4)
b <- c(-1.55, -0.95, -0.35, 0.2, 0.8, 1.45)
n <- 600
theta <- rnorm(n)
u <- matrix(runif(n * 6), n)
linked <- runif(n) < 0.5
u[linked, 2] <- u[linked, 1]
resp <- (u < plogis(outer(theta, b, "-"))) * 1
colnames(resp) <- paste0("Item_", 1:6)
# Marginal maximum likelihood calibration
fit <- mirt(resp, 1, itemtype = "Rasch", verbose = FALSE)
pars <- coef(fit, IRTpars = TRUE, simplify = TRUE)
round(pars$items[, "b"], 2)
# Local dependence: chi-square below the diagonal, signed
# standardized residual correlation above it
ld <- residuals(fit, type = "LD", verbose = FALSE)
round(ld, 2)Uses mirt. Install once with install.packages(c("mirt")).
What calibration can and cannot see
- Marginal maximum likelihood finds the difficulties that make the observed answers most likely, averaging over an assumed ability distribution. Each EM step improves the fit, and the estimates have converged when the largest change in a step rounds to zero.
- With 600 students the fitted difficulties usually land within 0.3 of the truth. With 150, misses larger than 0.3 are common.
- At convergence each item’s model percent correct matches its observed percent correct. A shared draw leaves those percentages alone, so it barely moves the difficulties.
- Local dependence shows up in pairs: two items that share something beyond ability are both right more often than the model predicts. The LD chi-square in
mirtchecks every pair. - Before trusting a calibration, compare the Rasch and 2PL models and check item and person fit.
Recommended reading
- The Theory and Practice of Item Response TheoryR. J. de Ayala2nd ed., chapter 4, marginal maximum likelihood parameter estimation
- Marginal Maximum Likelihood Estimation of Item ParametersR. Darrell Bock and Murray AitkinPaper · 1981, the EM algorithm for item calibration