Concept map / Item and person fit
Share
Item and person fit
See which fit statistics catch an item that breaks the Rasch model, and use lz to judge one person's response pattern.
1Ten items, and one breaks the Rasch model
A test of the whole model, like the one on the calibration page, says whether the model fits but not where it fails. Item fit compares each item’s observed responses with the curve the model fitted. Person fit asks whether one person’s right and wrong answers are plausible at that person’s estimated ability.
Here 1000 simulated people answer ten items and a Rasch model is fitted to all of them. Nine items follow it. Item 8 does not: you give it its own slope or a guessing floor. Which statistics notice, and which way do they move?
| Item | True b | S-X² (df) | p | Infit | Outfit |
|---|---|---|---|---|---|
| Item 1 | −1.8 | 7.05 (6) | 0.316 | 0.94 | 0.83 |
| Item 2 | −1.4 | 8.31 (6) | 0.217 | 0.91 | 0.83 |
| Item 3 | −1.0 | 1.61 (6) | 0.952 | 0.89 | 0.85 |
| Item 4 | −0.6 | 4.92 (7) | 0.670 | 0.87 | 0.83 |
| Item 5 | −0.2 | 8.26 (7) | 0.310 | 0.88 | 0.86 |
| Item 6 | 0.2 | 6.09 (7) | 0.530 | 0.85 | 0.83 |
| Item 7 | 0.6 | 6.28 (6) | 0.393 | 0.88 | 0.85 |
| Item 8slope 0.5 | 1.0 | 24.28 (6) | < 0.001 | 0.93 | 0.90 |
| Item 9 | 1.4 | 5.01 (6) | 0.542 | 0.93 | 0.85 |
| Item 10 | 1.8 | 11.50 (6) | 0.074 | 0.94 | 0.84 |
Item 8 has a true slope of 0.5; the other nine have 1. The Rasch fit gives item 8 S-X²(6) = 24.28, p < 0.001, so it is flagged at .05. Its infit is 0.93 and its outfit 0.90. The other items run from 0.85 to 0.94 on infit and 0.83 to 0.86 on outfit. Flagged at .05: item 8.
More controls
This is sample 1, with 1000 people. Some flags are pure chance. In 300 simulated samples where the Rasch model was true, each item was flagged at .05 in 8.5% of them, and at least one of the ten items in 59%.
2Guess the outfit of a steep item
Give item 8 a true slope of 2, twice the slope of the other nine, and fit a Rasch model to 1000 people. Rasch software reports an outfit mean square for each item: the average squared standardized residual. How will item 8's outfit compare with the items that fit?
3Where the item flags come from
S-X² (Orlando and Thissen) sorts people into groups by sum score and compares each group's proportion correct with what the model expects. Infit and outfit work person by person instead, from the residual at each person's estimated ability. Point at a symbol to see what it stands for.
Point at O, E or P(1 − P). The matching part of the item chart above lights up.
A common misreading is to pass any item whose mean square sits inside 0.8 to 1.2. With only ten items, each person's ability comes from the same ten responses, so residuals are smaller than the model expects and items that fit land below 1. Here their outfit runs from 0.83 to 0.86. Compare items with each other.
Show the S-X² working for item 8
The largest term comes from sum score 5: 173 people, observed 0.45 against expected 0.36. It contributes 6.48 of the total 24.28. As in mirt, a group with an expected count below 1, or with a single person, is merged with a neighbor (here 1 and 2). The degrees of freedom are the 8 groups minus two, one for the item's difficulty and one for the estimated spread of ability: 8 − 2 = 6.
4When one person's answers do not fit
lz does the same job for a person. It standardizes the log-likelihood of one response pattern at that person's estimated ability, so a large negative value means the pattern is less likely than the model expects. Press an item to switch its response. Items run from easiest (1) to hardest (10).
6 correct, so = 0.32 and lz = 0.21. That is inside the usual ±2, so nothing is flagged.
Show how lz is computed
For your pattern, l0 = −5.45, E(l0) = −5.73 and Var(l0) = 1.84, so lz = (−5.45 − (−5.73)) / √1.84 = 0.21.
Show lz for all 1000 simulated people
Most of these people fit the model, yet lz has mean 0.32 and standard deviation 0.91, and only 1.6% fall below −2, where a standard normal puts 2.3%. The reason is that is estimated from the same ten responses that lz then judges.
5Check yourself
Each task ticks itself when the lab meets its condition.
- Switch to Guessing floor and raise it until S-X² flags item 8 at the .01 level.Now: item 8 has a different slope, not a guessing floor
- Make item 8 steep enough that its outfit falls below 0.80 while every other item stays at 0.80 or above.Now: item 8 outfit 0.90, other items 0.83 to 0.86
- In step 4, build a pattern with exactly 5 correct answers and an lz below −2.Now: 6 correct, lz = 0.21
6The same statistics in mirt
The code rebuilds the lab’s starting setup with R’s own random numbers: ten items, item 8 with slope 0.5, and 1000 people. It fits a Rasch model with mirt, then gets S-X², infit and outfit from itemfit() and lz from personfit(), where it is called Zh. Item 8 has S-X²(7) = 50.867, p < 0.001. Its infit (0.971) and outfit (0.964) are the highest of the ten and still under 1. The lz values have mean 0.328 and standard deviation 0.902, and 1.5% fall below −2. On the lab’s own simulated data, mirt’s S-X², degrees of freedom, p-values, infit, outfit and lz match the lab’s to within 0.0001.
Item and person fit for a Rasch model with mirtShow the code
library(mirt)
set.seed(18)
# Ten items; item 8 has slope 0.5 instead of 1
b <- seq(-1.8, 1.8, by = 0.4)
a <- rep(1, 10)
a[8] <- 0.5
dat <- simdata(a = matrix(a), d = matrix(-a * b), N = 1000,
itemtype = "2PL")
fit <- mirt(dat, 1, itemtype = "Rasch", verbose = FALSE)
# Item fit: S-X2 by sum score; infit and outfit at EAP
out <- itemfit(fit, c("S_X2", "infit"))
cols <- c("S_X2", "df.S_X2", "p.S_X2", "infit", "outfit")
cbind(out["item"], round(out[cols], 3))
# Person fit: lz (mirt calls it Zh) is not quite N(0, 1)
lz <- personfit(fit)$Zh
round(c(mean = mean(lz), sd = sd(lz), below2 = mean(lz < -2)), 3)Uses mirt. Install once with install.packages(c("mirt")).
What to take away
- S-X² groups people by sum score, compares the observed and expected proportion correct in each group, and refers the total to a chi-square distribution.
- Misfit has a direction. A flatter item, or one with a guessing floor, is less predictable than the model expects and its mean squares rise. A steeper item is more predictable and its mean squares drop below the other items’, which Rasch users call overfit. S-X² flags both.
- On a ten-item test, items that fit have mean squares below 1, so the usual 0.8 to 1.2 range is a loose guide at best. Chance adds flags too: testing ten items at .05 flags at least one in most samples even when the model is true. Look for large, repeatable departures and check the observed curve before you drop an item.
- Lucky guesses on hard items and careless errors on easy ones push lz well below 0. Because θ is estimated from the same responses, lz here has a mean above 0 and a standard deviation below 1, so a cutoff of −2 flags fewer than 2.3% of people. Snijders’ corrected lz* adjusts for this.
- An item can fit in the whole sample and still behave differently for two groups of equal ability, which is differential item functioning. Person fit matters again in computerized adaptive testing, where each person sees different items.
Recommended reading
- Likelihood-based item-fit indices for dichotomous item response theory modelsMaria Orlando and David ThissenPaper · Applied Psychological Measurement, 24(1), 2000
- Methodology review: Evaluating person fitRob R. Meijer and Klaas SijtsmaPaper · Applied Psychological Measurement, 25(2), 2001