Concept map / IRT calibration and model fit

Share

Item response theory calibration and model fit

Estimate Rasch item difficulties from simulated responses and inspect an item-pair diagnostic.

1From right and wrong answers to item difficulties

Earlier pages took each item characteristic curve as known. In practice all you have is a table of right and wrong answers. How do you recover each item’s difficulty from it, and how do you tell whether the model fits?

Calibration estimates item parameters from response data. Here the Rasch difficulties are fitted by marginal maximum likelihood, which averages each student’s likelihood over a standard normal spread of ability instead of estimating every ability first. The fit sees only the answers; tick the box to compare it with the true difficulties. Below the fit, a second chart asks whether Items 1 and 2 are both right more often than the model predicts, a sign of local dependence.

−3−2−10123Item 1Item 2Item 3Item 4Item 5Item 6ItemDifficulty b (logits)
Starting estimate
Show percent correct by item
ItemStarting bObserved correctModel correct
1−1.2978.3%74.5%
2−0.7868.7%65.7%
3−0.3458.3%56.9%
40.2045.0%45.9%
50.7432.3%35.1%
61.4519.0%22.9%

Are Items 1 and 2 right together too often?

Both Items 1 and 2 correctObserved56.7%Model52.2%

300 students answered six items. These are starting estimates from each item’s percent correct. Fit to estimate the difficulties. Items 1 and 2 were both correct for 56.7% of students; the model predicts 52.2%, a gap of 4.5 points.

2What shared answers do to the fit

Suppose every student answers Items 1 and 2 from one shared random draw. Each item’s chance of a correct answer at a given ability stays the same, but the two answers now move together beyond what ability explains. You fit the Rasch model as usual. What do you see?

3What the fit maximizes, and what it predicts

Marginal maximum likelihood averages each student’s response pattern over a standard normal ability distribution φ, then picks the difficulties b that make the observed patterns most likely. Each fitting step is one round of the EM algorithm. Point at a symbol to light up its part of the lab.

L(b)=∏p∫∏i=16Pi(θ)upi [1−Pi(θ)]1−upi ϕ(θ) dθ,Pi(θ)=11+e−(θ−bi)L(\htmlClass{term-b}{\mathbf b}) = \prod_{p}\int \prod_{i=1}^{6} P_i(\theta)^{u_{pi}}\,\bigl[1 - P_i(\theta)\bigr]^{1-u_{pi}}\,\phi(\theta)\,d\theta, \qquad P_i(\theta) = \frac{1}{1 + e^{-(\theta - \htmlClass{term-b}{b_i})}}
L(b)=∏p∫∏i=16Pi(θ)upi [1−Pi(θ)]1−upi×ϕ(θ) dθPi(θ)=11+e−(θ−bi)\begin{aligned} L(\htmlClass{term-b}{\mathbf b}) &= \prod_{p}\int \prod_{i=1}^{6} P_i(\theta)^{u_{pi}}\,\bigl[1 - P_i(\theta)\bigr]^{1-u_{pi}} \\ &\qquad \times \phi(\theta)\,d\theta \\[6pt] P_i(\theta) &= \frac{1}{1 + e^{-(\theta - \htmlClass{term-b}{b_i})}} \end{aligned}

The fitted model also predicts how often two items are both correct. Under local independence two answers are linked only through ability, so the prediction multiplies the two item curves and averages over abilities.

Pr⁡(u1=1, u2=1)=∫P1(θ) P2(θ) ϕ(θ) dθ\htmlClass{term-pair}{\Pr(u_1 = 1,\, u_2 = 1)} = \int P_1(\theta)\,P_2(\theta)\,\phi(\theta)\,d\theta
56.7%⏟observed−52.2%⏟model=4.5 points\underbrace{56.7\%}_{\text{observed}} - \underbrace{\htmlClass{term-pair}{52.2\%}}_{\text{model}} = 4.5\text{ points}

A gap of a point or two can come from sampling alone, so do not read dependence into one small gap. A gap that grows with d and survives new responses means the two items share something beyond ability, such as a reading passage.

4Check yourself

Each task ticks itself when the lab meets its condition.

  • With 600 students, fit until the estimates stop changing, reveal the true difficulties, and see every fitted difficulty land within 0.30 of its true value.Now: 300 students, 0 steps, not settled, reveal the true difficulties to see the largest miss
  • Raise the shared tendency for Items 1 and 2 and fit until the estimates settle, so that the pair is correct together at least 5 points more often than the model predicts.Now: shared tendency 0%, not settled, gap 4.5 points
  • With 150 students and no shared tendency, fit to convergence and draw new responses until some fitted difficulty misses its true value by more than 0.30.Now: 300 students, shared tendency 0%, not settled, reveal the true difficulties to see the largest miss

5Calibrate and check every pair in mirt

The code simulates 600 students answering the six items, with Items 1 and 2 sharing one draw for half of them, as with d at 50%. mirt also fits by marginal maximum likelihood and prints difficulties of −1.62, −1.00, −0.37, 0.38, 0.78 and 1.57; the true values are −1.55, −0.95, −0.35, 0.20, 0.80 and 1.45. residuals(fit, type = "LD") then gives a local dependence chi-square for every pair of items. Items 1 and 2 give 53.75 and a residual correlation of 0.30, and every other pair stays below 4.2. With 15 pairs, a few values just above 3.84, the 5% cutoff for one degree of freedom, are expected by chance.

Calibrate Rasch items and check local dependence with mirtShow the code
library(mirt)

# 600 students answer six Rasch items with the page's
# difficulties. Items 1 and 2 share one random draw for
# half of the students, as with the slider at 50%.
set.seed(4)
b <- c(-1.55, -0.95, -0.35, 0.2, 0.8, 1.45)
n <- 600
theta <- rnorm(n)
u <- matrix(runif(n * 6), n)
linked <- runif(n) < 0.5
u[linked, 2] <- u[linked, 1]
resp <- (u < plogis(outer(theta, b, "-"))) * 1
colnames(resp) <- paste0("Item_", 1:6)

# Marginal maximum likelihood calibration
fit <- mirt(resp, 1, itemtype = "Rasch", verbose = FALSE)
pars <- coef(fit, IRTpars = TRUE, simplify = TRUE)
round(pars$items[, "b"], 2)

# Local dependence: chi-square below the diagonal, signed
# standardized residual correlation above it
ld <- residuals(fit, type = "LD", verbose = FALSE)
round(ld, 2)

Uses mirt. Install once with install.packages(c("mirt")).

What calibration can and cannot see