Concept map / DIF
Share
Differential item functioning
Compare item response probabilities at the same ability and distinguish this comparison from a difference in average performance.
1One item, a 20-point gap
Suppose 70% of one group and 50% of another answer a test item correctly. Is the item unfair to the second group? The answer turns on a narrower comparison: two people with the same ability, one from each group. If their chances differ, the item shows differential item functioning (DIF).
The groups are called the reference group and the focal group. The labels say which group is compared with which, not why they might differ. Each group’s chances follow an item characteristic curve.
Predict first. On one test item, 70% of the reference group and 50% of the focal group answer correctly. Is the item biased against the focal group?
2Move the curves apart, or move the groups
The top chart has one item curve per group. The bottom chart shows where each group’s abilities lie. Δb slides the focal curve sideways and Δa changes its slope, and both create DIF. Δμ moves the focal group’s abilities and leaves the curves alone. That difference in average ability is called impact, and on its own it is not DIF.
The item (DIF)
The groups (impact)
More controls
Take two people at θ = 0.00, one from each group. The reference-group member answers correctly with probability 50%, the focal-group member 35%. The item is harder for the focal group at every ability, as if its difficulty were 0.50 higher for them. That is uniform DIF. The two groups have the same average ability. Overall, 50.0% of the reference group and 38.5% of the focal group answer correctly. The whole gap comes from the item.
3Matching on total score in a simulated sample
Real data never show the curves, or anyone’s ability. You have each person’s group, their answer to this item and their total test score, so the score stands in for ability. Here 1,000 people per group take a 41-item test: this item plus 40 items with no DIF. The sliders above still set the truth.
In this sample 48.2% of the reference group and 40.0% of the focal group got the item right, a raw gap of 8.2 points. Among people with the same total score, a reference-group member’s odds of a right answer are 1.70 times a focal-group member’s (95% range 1.38 to 2.09). The range excludes 1, so matched people do differ: on balance the item is harder for the focal group.
A common misreading is to take an odds ratio near 1 as proof of no DIF. It is one number for the whole score range, so gaps in opposite directions can cancel. Set Δb = 0, Δa = 0.80 and no impact, and the dots separate at both ends while the odds ratio stays close to 1.
How the chart and the test are built
Each dot is one group’s proportion correct among people whose total score falls in that band, placed at their average score. A dot appears only when the group has at least 20 people in the band. The Mantel–Haenszel test itself matches on every single score, 36 of them in this sample. The same sample number always gives the same people.
On the ETS delta scale, −2.35 × ln(1.70) = −1.25. Negative values mean the item is harder for the focal group. By the ETS convention, absolute values under 1 count as negligible and 1.5 or more as large, each judged alongside a significance test.
Real data are less tidy. The 40 matching items here all have the same discrimination, so total score tracks ability cleanly. When items differ in discrimination and the groups differ a lot in average ability, matching on total score is less exact, and the test can flag a little DIF that is not there.
4How big the DIF is: the area between the curves
Add up the gap between the two curves over every ability and you get one number for the size of the DIF. The signed area lets gaps in opposite directions cancel. The unsigned area does not. Point at the bracketed gap in a formula and the shaded gap darkens in the top chart.
With your settings the signed area is 0.50 (Δb is 0.50) and the unsigned area is 0.50. The gap never changes sign, so the two areas are the same size. For curves with no guessing floor, like these, the signed area always equals Δb. Δμ appears in neither formula, so moving the groups cannot change either area.
The two curve formulas, with your numbers
Each group gets its own item curve. The focal curve uses the reference values plus the two differences. Point at a symbol, or at a slider, and the matching part of the chart lights up.
With aR = 1.20, bR = 0.00, Δa = 0.00 and Δb = 0.50: PR(0.00) = 0.500 and PF(0.00) = 0.354, a gap of 0.146 at this ability. The areas are found by numerical integration over the whole ability line, including the parts beyond the edges of the chart.
5Check yourself
All three use the sliders in the first panel. Each ticks itself off when you meet it.
- Make the overall proportions correct differ by at least 20 points while the item has no DIF (area 0.00). Remove the DIF is under More controls.Now: Gap in proportions 11.5 points, DIF area 0.50
- Hide DIF behind impact: make the item harder for the focal group at every ability (Δb of 0.50 or more, Δa = 0), then move Δμ until the overall proportions differ by less than 2 points.Now: Δb 0.50, Δa 0.00, gap in proportions 11.5 points
- Build DIF that a signed measure misses: a signed area of 0.00 and an unsigned area of at least 0.30.Now: Signed area 0.50, unsigned area 0.50
6The Mantel–Haenszel test in R
The code simulates the lab’s starting design and runs base R’s mantelhaen.test, matching on total score. R draws different people from the lab’s sample 1, so it reports a common odds ratio of 1.73 (95% range 1.40 to 2.13) and an ETS delta of −1.29. The lab’s sample 1 gives 1.70 and −1.25.
Mantel–Haenszel DIF test in base RShow the code
# 1000 people per group sit 40 DIF-free items and the studied
# item, which is 0.5 harder for the focal group (a = 1.2)
set.seed(1)
n <- 1000
grp <- factor(rep(c("ref", "focal"), each = n))
grp <- relevel(grp, "ref")
theta <- rnorm(2 * n)
b_anchor <- seq(-2.5, 2.5, length = 40)
p <- plogis(1.2 * outer(theta, b_anchor, "-"))
anchors <- matrix(rbinom(length(p), 1, p), 2 * n)
b <- ifelse(grp == "ref", 0, 0.5)
y <- rbinom(2 * n, 1, plogis(1.2 * (theta - b)))
score <- rowSums(anchors) + y
# Match on total score, keeping scores both groups reached
both <- intersect(score[grp == "ref"], score[grp == "focal"])
keep <- score %in% both
tab <- table(grp[keep], factor(y[keep], 1:0), score[keep])
mh <- mantelhaen.test(tab)
# Common odds ratio (reference over focal) and its 95% range
or <- unname(mh$estimate)
round(c(odds_ratio = or, lower = mh$conf.int[1],
upper = mh$conf.int[2]), 2)
# ETS delta: negative means harder for the focal group
round(-2.35 * log(or), 2)Uses base R only.
Impact and DIF, in brief
- A gap in overall proportion correct can come entirely from impact, while the two item curves sit on top of each other.
- When only the difficulty differs, one curve slides sideways and the item is harder for that group at every ability. That is uniform DIF.
- When the discriminations differ, the curves cross and the direction of the gap depends on ability. A signed area, or a single Mantel–Haenszel odds ratio, can then add up to almost nothing.
- In real data the total score stands in for ability. Mantel–Haenszel compares the groups’ odds within each score; logistic regression does the same with score, group and their interaction.
- Finding DIF tells you that matched people differ, not why. Deciding whether that is bias takes a look at the item’s content. Measurement invariance asks the same question of a whole factor model.
Recommended reading
- Differential Item FunctioningPaul W. Holland and Howard Wainer, eds.DIF definitions, detection, and interpretation