Concept map / Item and person fit

Share

Item and person fit

See which fit statistics catch an item that breaks the Rasch model, and use lz to judge one person's response pattern.

1Ten items, and one breaks the Rasch model

A test of the whole model, like the one on the calibration page, says whether the model fits but not where it fails. Item fit compares each item’s observed responses with the curve the model fitted. Person fit asks whether one person’s right and wrong answers are plausible at that person’s estimated ability.

Here 1000 simulated people answer ten items and a Rasch model is fitted to all of them. Nine items follow it. Item 8 does not: you give it its own slope or a guessing floor. Which statistics notice, and which way do they move?

Item 8: proportion correct00.250.50.751123456789Sum score on all ten items (ability group)
Observed (larger dot: more people)Expected under the Rasch fit
Orange p-values are below .05.
ItemTrue bS-X² (df)pInfitOutfit
Item 1−1.87.05 (6)0.3160.940.83
Item 2−1.48.31 (6)0.2170.910.83
Item 3−1.01.61 (6)0.9520.890.85
Item 4−0.64.92 (7)0.6700.870.83
Item 5−0.28.26 (7)0.3100.880.86
Item 60.26.09 (7)0.5300.850.83
Item 70.66.28 (6)0.3930.880.85
Item 8slope 0.51.024.28 (6)< 0.0010.930.90
Item 91.45.01 (6)0.5420.930.85
Item 101.811.50 (6)0.0740.940.84

Item 8 has a true slope of 0.5; the other nine have 1. The Rasch fit gives item 8 S-X²(6) = 24.28, p < 0.001, so it is flagged at .05. Its infit is 0.93 and its outfit 0.90. The other items run from 0.85 to 0.94 on infit and 0.83 to 0.86 on outfit. Flagged at .05: item 8.

More controls

This is sample 1, with 1000 people. Some flags are pure chance. In 300 simulated samples where the Rasch model was true, each item was flagged at .05 in 8.5% of them, and at least one of the ten items in 59%.

2Guess the outfit of a steep item

Give item 8 a true slope of 2, twice the slope of the other nine, and fit a Rasch model to 1000 people. Rasch software reports an outfit mean square for each item: the average squared standardized residual. How will item 8's outfit compare with the items that fit?

3Where the item flags come from

S-X² (Orlando and Thissen) sorts people into groups by sum score and compares each group's proportion correct with what the model expects. Infit and outfit work person by person instead, from the residual at each person's estimated ability. Point at a symbol to see what it stands for.

S-X2=∑kNk (Ok−Ek)2Ek(1−Ek)outfit=1N∑i(ui−Pi)2Pi(1−Pi)infit=∑i(ui−Pi)2∑iPi(1−Pi)\begin{aligned} S\text{-}X^2 &= \sum_k N_k\,\frac{(\htmlClass{term-o}{O_k} - \htmlClass{term-e}{E_k})^2}{\htmlClass{term-e}{E_k}(1 - \htmlClass{term-e}{E_k})} \\[2pt] \text{outfit} &= \frac{1}{N}\sum_i \frac{(u_i - P_i)^2}{\htmlClass{term-w}{P_i(1 - P_i)}} \qquad \text{infit} = \frac{\sum_i (u_i - P_i)^2}{\sum_i \htmlClass{term-w}{P_i(1 - P_i)}} \end{aligned}
S-X2=∑kNk (Ok−Ek)2Ek(1−Ek)outfit=1N∑i(ui−Pi)2Pi(1−Pi)infit=∑i(ui−Pi)2∑iPi(1−Pi)\begin{aligned} S\text{-}X^2 &= \sum_k N_k\,\frac{(\htmlClass{term-o}{O_k} - \htmlClass{term-e}{E_k})^2}{\htmlClass{term-e}{E_k}(1 - \htmlClass{term-e}{E_k})} \\[2pt] \text{outfit} &= \frac{1}{N}\sum_i \frac{(u_i - P_i)^2}{\htmlClass{term-w}{P_i(1 - P_i)}} \\[2pt] \text{infit} &= \frac{\sum_i (u_i - P_i)^2}{\sum_i \htmlClass{term-w}{P_i(1 - P_i)}} \end{aligned}

Point at O, E or P(1 − P). The matching part of the item chart above lights up.

A common misreading is to pass any item whose mean square sits inside 0.8 to 1.2. With only ten items, each person's ability comes from the same ten responses, so residuals are smaller than the model expects and items that fit land below 1. Here their outfit runs from 0.83 to 0.86. Compare items with each other.

Show the S-X² working for item 8

The largest term comes from sum score 5: 173 people, observed 0.45 against expected 0.36. It contributes 6.48 of the total 24.28. As in mirt, a group with an expected count below 1, or with a single person, is merged with a neighbor (here 1 and 2). The degrees of freedom are the 8 groups minus two, one for the item's difficulty and one for the estimated spread of ability: 8 − 2 = 6.

4When one person's answers do not fit

lz does the same job for a person. It standardizes the log-likelihood of one response pattern at that person's estimated ability, so a large negative value means the pattern is less likely than the model expects. Press an item to switch its response. Items run from easiest (1) to hardest (10).

Chance of a correct answer at the estimated θ = 0.3200.5112345678910Item, easiest to hardest
Model probability at CorrectWrongDotted line: the residual

6 correct, so = 0.32 and lz = 0.21. That is inside the usual ±2, so nothing is flagged.

Show how lz is computed
l0=∑j[ujlog⁡Pj+(1−uj)log⁡(1−Pj)]E(l0)=∑j[Pjlog⁡Pj+(1−Pj)log⁡(1−Pj)]Var(l0)=∑jPj(1−Pj)[log⁡Pj1−Pj]2lz=l0−E(l0)Var(l0)\begin{aligned} l_0 &= \textstyle\sum_j \bigl[u_j \log P_j + (1 - u_j)\log(1 - P_j)\bigr] \\[2pt] \mathrm{E}(l_0) &= \textstyle\sum_j \bigl[P_j \log P_j + (1 - P_j)\log(1 - P_j)\bigr] \\[2pt] \mathrm{Var}(l_0) &= \textstyle\sum_j P_j(1 - P_j)\Bigl[\log\frac{P_j}{1 - P_j}\Bigr]^2 \\[2pt] l_z &= \dfrac{l_0 - \mathrm{E}(l_0)}{\sqrt{\mathrm{Var}(l_0)}} \end{aligned}
l0=∑j[ujlog⁡Pj+(1−uj)log⁡(1−Pj)]E(l0)=∑j[Pjlog⁡Pj+(1−Pj)log⁡(1−Pj)]Var(l0)=∑jPj(1−Pj)[log⁡Pj1−Pj]2lz=l0−E(l0)Var(l0)\begin{aligned} l_0 &= \textstyle\sum_j \bigl[u_j \log P_j \\ &\qquad + (1 - u_j)\log(1 - P_j)\bigr] \\[2pt] \mathrm{E}(l_0) &= \textstyle\sum_j \bigl[P_j \log P_j \\ &\qquad + (1 - P_j)\log(1 - P_j)\bigr] \\[2pt] \mathrm{Var}(l_0) &= \textstyle\sum_j P_j(1 - P_j)\Bigl[\log\frac{P_j}{1 - P_j}\Bigr]^2 \\[2pt] l_z &= \dfrac{l_0 - \mathrm{E}(l_0)}{\sqrt{\mathrm{Var}(l_0)}} \end{aligned}

For your pattern, l0 = −5.45, E(l0) = −5.73 and Var(l0) = 1.84, so lz = (−5.45 − (−5.73)) / √1.84 = 0.21.

Show lz for all 1000 simulated peoplelz in this sample, with the standard normal curve−6−4−202lzyour pattern 0.21
Simulated peopleStandard normalGray dashed line: lz = −2

Most of these people fit the model, yet lz has mean 0.32 and standard deviation 0.91, and only 1.6% fall below −2, where a standard normal puts 2.3%. The reason is that is estimated from the same ten responses that lz then judges.

5Check yourself

Each task ticks itself when the lab meets its condition.

  • Switch to Guessing floor and raise it until S-X² flags item 8 at the .01 level.Now: item 8 has a different slope, not a guessing floor
  • Make item 8 steep enough that its outfit falls below 0.80 while every other item stays at 0.80 or above.Now: item 8 outfit 0.90, other items 0.83 to 0.86
  • In step 4, build a pattern with exactly 5 correct answers and an lz below −2.Now: 6 correct, lz = 0.21

6The same statistics in mirt

The code rebuilds the lab’s starting setup with R’s own random numbers: ten items, item 8 with slope 0.5, and 1000 people. It fits a Rasch model with mirt, then gets S-X², infit and outfit from itemfit() and lz from personfit(), where it is called Zh. Item 8 has S-X²(7) = 50.867, p < 0.001. Its infit (0.971) and outfit (0.964) are the highest of the ten and still under 1. The lz values have mean 0.328 and standard deviation 0.902, and 1.5% fall below −2. On the lab’s own simulated data, mirt’s S-X², degrees of freedom, p-values, infit, outfit and lz match the lab’s to within 0.0001.

Item and person fit for a Rasch model with mirtShow the code
library(mirt)
set.seed(18)
# Ten items; item 8 has slope 0.5 instead of 1
b <- seq(-1.8, 1.8, by = 0.4)
a <- rep(1, 10)
a[8] <- 0.5
dat <- simdata(a = matrix(a), d = matrix(-a * b), N = 1000,
               itemtype = "2PL")
fit <- mirt(dat, 1, itemtype = "Rasch", verbose = FALSE)

# Item fit: S-X2 by sum score; infit and outfit at EAP
out <- itemfit(fit, c("S_X2", "infit"))
cols <- c("S_X2", "df.S_X2", "p.S_X2", "infit", "outfit")
cbind(out["item"], round(out[cols], 3))

# Person fit: lz (mirt calls it Zh) is not quite N(0, 1)
lz <- personfit(fit)$Zh
round(c(mean = mean(lz), sd = sd(lz), below2 = mean(lz < -2)), 3)

Uses mirt. Install once with install.packages(c("mirt")).

What to take away