Concept map / DIF

Share

Differential item functioning

Compare item response probabilities at the same ability and distinguish this comparison from a difference in average performance.

1One item, a 20-point gap

Suppose 70% of one group and 50% of another answer a test item correctly. Is the item unfair to the second group? The answer turns on a narrower comparison: two people with the same ability, one from each group. If their chances differ, the item shows differential item functioning (DIF).

The groups are called the reference group and the focal group. The labels say which group is compared with which, not why they might differ. Each group’s chances follow an item characteristic curve.

Predict first. On one test item, 70% of the reference group and 50% of the focal group answer correctly. Is the item biased against the focal group?

2Move the curves apart, or move the groups

The top chart has one item curve per group. The bottom chart shows where each group’s abilities lie. Δb slides the focal curve sideways and Δa changes its slope, and both create DIF. Δμ moves the focal group’s abilities and leaves the curves alone. That difference in average ability is called impact, and on its own it is not DIF.

00.250.50.751-4-3-2-101234P(correct)Ability θat θ = 0.00: reference 50%, focal 35%
Item curve, reference groupItem curve, focal groupGap between the curves (DIF)
-4-3-2-101234How many people at each abilityAbility θno impact: same average abilityReferenceFocal
Gap in overall correct11.5 pointsmoves with impact and DIF
Area between the curves0.50moves with DIF only

The item (DIF)

The groups (impact)

More controls

Take two people at θ = 0.00, one from each group. The reference-group member answers correctly with probability 50%, the focal-group member 35%. The item is harder for the focal group at every ability, as if its difficulty were 0.50 higher for them. That is uniform DIF. The two groups have the same average ability. Overall, 50.0% of the reference group and 38.5% of the focal group answer correctly. The whole gap comes from the item.

3Matching on total score in a simulated sample

Real data never show the curves, or anyone’s ability. You have each person’s group, their answer to this item and their total test score, so the score stands in for ability. Here 1,000 people per group take a 41-item test: this item plus 40 items with no DIF. The sliders above still set the truth.

00.250.50.7510 to 56 to 1112 to 1718 to 2324 to 2930 to 3536 to 41Proportion correct on this itemTotal test score (band)
Reference groupFocal groupWhiskers: about a 95% margin of error
Sample 1

In this sample 48.2% of the reference group and 40.0% of the focal group got the item right, a raw gap of 8.2 points. Among people with the same total score, a reference-group member’s odds of a right answer are 1.70 times a focal-group member’s (95% range 1.38 to 2.09). The range excludes 1, so matched people do differ: on balance the item is harder for the focal group.

A common misreading is to take an odds ratio near 1 as proof of no DIF. It is one number for the whole score range, so gaps in opposite directions can cancel. Set Δb = 0, Δa = 0.80 and no impact, and the dots separate at both ends while the odds ratio stays close to 1.

How the chart and the test are built

Each dot is one group’s proportion correct among people whose total score falls in that band, placed at their average score. A dot appears only when the group has at least 20 people in the band. The Mantel–Haenszel test itself matches on every single score, 36 of them in this sample. The same sample number always gives the same people.

On the ETS delta scale, −2.35 × ln(1.70) = −1.25. Negative values mean the item is harder for the focal group. By the ETS convention, absolute values under 1 count as negligible and 1.5 or more as large, each judged alongside a significance test.

Real data are less tidy. The 40 matching items here all have the same discrimination, so total score tracks ability cleanly. When items differ in discrimination and the groups differ a lot in average ability, matching on total score is less exact, and the test can flag a little DIF that is not there.

4How big the DIF is: the area between the curves

Add up the gap between the two curves over every ability and you get one number for the size of the DIF. The signed area lets gaps in opposite directions cancel. The unsigned area does not. Point at the bracketed gap in a formula and the shaded gap darkens in the top chart.

signed area=∫−∞∞[PR(θ)−PF(θ)] dθ=Δbunsigned area=∫−∞∞∣PR(θ)−PF(θ)∣ dθ\begin{gathered} \text{signed area} = \int_{-\infty}^{\infty} \htmlClass{term-gap}{\big[P_R(\theta)-P_F(\theta)\big]}\,d\theta = \htmlClass{term-b}{\Delta b} \\[8pt] \text{unsigned area} = \int_{-\infty}^{\infty} \htmlClass{term-gap}{\big|P_R(\theta)-P_F(\theta)\big|}\,d\theta \end{gathered}

With your settings the signed area is 0.50 (Δb is 0.50) and the unsigned area is 0.50. The gap never changes sign, so the two areas are the same size. For curves with no guessing floor, like these, the signed area always equals Δb. Δμ appears in neither formula, so moving the groups cannot change either area.

The two curve formulas, with your numbers

Each group gets its own item curve. The focal curve uses the reference values plus the two differences. Point at a symbol, or at a slider, and the matching part of the chart lights up.

PR(θ)=11+e−aR(θ−bR)PF(θ)=11+e−(aR+Δa) (θ−bR−Δb)\begin{gathered} P_R(\htmlClass{term-theta}{\theta}) = \frac{1}{1+e^{-a_R(\htmlClass{term-theta}{\theta}-b_R)}} \\[8pt] P_F(\htmlClass{term-theta}{\theta}) = \frac{1}{1+e^{-(a_R+\htmlClass{term-a}{\Delta a})\,(\htmlClass{term-theta}{\theta}-b_R-\htmlClass{term-b}{\Delta b})}} \end{gathered}

With aR = 1.20, bR = 0.00, Δa = 0.00 and Δb = 0.50: PR(0.00) = 0.500 and PF(0.00) = 0.354, a gap of 0.146 at this ability. The areas are found by numerical integration over the whole ability line, including the parts beyond the edges of the chart.

5Check yourself

All three use the sliders in the first panel. Each ticks itself off when you meet it.

  • Make the overall proportions correct differ by at least 20 points while the item has no DIF (area 0.00). Remove the DIF is under More controls.Now: Gap in proportions 11.5 points, DIF area 0.50
  • Hide DIF behind impact: make the item harder for the focal group at every ability (Δb of 0.50 or more, Δa = 0), then move Δμ until the overall proportions differ by less than 2 points.Now: Δb 0.50, Δa 0.00, gap in proportions 11.5 points
  • Build DIF that a signed measure misses: a signed area of 0.00 and an unsigned area of at least 0.30.Now: Signed area 0.50, unsigned area 0.50

6The Mantel–Haenszel test in R

The code simulates the lab’s starting design and runs base R’s mantelhaen.test, matching on total score. R draws different people from the lab’s sample 1, so it reports a common odds ratio of 1.73 (95% range 1.40 to 2.13) and an ETS delta of −1.29. The lab’s sample 1 gives 1.70 and −1.25.

Mantel–Haenszel DIF test in base RShow the code
# 1000 people per group sit 40 DIF-free items and the studied
# item, which is 0.5 harder for the focal group (a = 1.2)
set.seed(1)
n <- 1000
grp <- factor(rep(c("ref", "focal"), each = n))
grp <- relevel(grp, "ref")
theta <- rnorm(2 * n)
b_anchor <- seq(-2.5, 2.5, length = 40)
p <- plogis(1.2 * outer(theta, b_anchor, "-"))
anchors <- matrix(rbinom(length(p), 1, p), 2 * n)
b <- ifelse(grp == "ref", 0, 0.5)
y <- rbinom(2 * n, 1, plogis(1.2 * (theta - b)))
score <- rowSums(anchors) + y

# Match on total score, keeping scores both groups reached
both <- intersect(score[grp == "ref"], score[grp == "focal"])
keep <- score %in% both
tab <- table(grp[keep], factor(y[keep], 1:0), score[keep])
mh <- mantelhaen.test(tab)
# Common odds ratio (reference over focal) and its 95% range
or <- unname(mh$estimate)
round(c(odds_ratio = or, lower = mh$conf.int[1],
        upper = mh$conf.int[2]), 2)
# ETS delta: negative means harder for the focal group
round(-2.35 * log(or), 2)

Uses base R only.

Impact and DIF, in brief