Concept map / Missing data
Share
Missing data and multiple imputation
Lose posttest scores under MCAR, MAR or MNAR, then compare listwise deletion, mean imputation, regression imputation and multiple imputation against the complete data.
1Two hundred students, some posttests lost
Two hundred students take a pretest in September and a posttest in May, and some of the posttests never arrive. What to do about them depends on why they are missing. If the loss has nothing to do with the students (MCAR, missing completely at random), the rest are a random subset. If it depends on something you observed, such as the pretest (MAR, missing at random), it can be corrected. If it depends on the missing score itself (MNAR), the data alone cannot correct it.
The lab knows every true posttest, so you can see what each fix does. Choose why scores go missing and how many, then compare four fixes with the complete data on two numbers: the mean posttest and the regression slope of posttest on pretest.
Listwise deletion keeps the 140 students who have both scores. Their posttest mean is 50.1 (complete data 49.8) and their slope is 0.78 (complete data 0.80).
More controls
m is the number of imputed data sets that multiple imputation pools.
2Dropping students by their pretest
Take the 200 students in the lab, all with a posttest, and delete 40% of the posttests, mostly from students with low pretest scores. That is MAR: whether a score is missing depends on the pretest, which you still have. You analyze only the 120 complete cases. What happens to the estimates?
3Pooling m analyses with Rubin’s rules
Regression imputation fills each gap with the single best guess, so the analysis treats a guess as a measured score. Multiple imputation draws m different plausible values for every gap instead, adding the scatter around the line and the uncertainty in the line itself. Each completed data set is analyzed as if nothing were missing, and the m results are combined.
For the posttest mean: Q̄ = 49.82, the average of 20 estimates. Ū = 0.437 is their average squared standard error and B = 0.073 is the variance between them, so T = 0.437 + 1.05 × 0.073 = 0.514 and the standard error is √T = 0.72. Missing data account for 15% of T. Point at Q̄, B or T to find it on the multiple imputation row of the estimates chart.
B is what single imputation leaves out. A regression-imputed data set behaves as if all 200 posttests had been observed, so its interval for the mean is too narrow; here it is 2.45 points wide, against 2.84 for multiple imputation.
4What the observed data cannot tell you
The 60 students without a posttest averaged 48.8 on the pretest, against 49.9 for the 140 with one. Under MCAR the two groups differ only by chance. A large gap would rule MCAR out.
Both mechanisms leave the same kind of data behind: a full set of pretests and posttests missing mostly among low scorers. Whether the missing students did worse than their pretest predicts is a fact about the scores you do not have, so no test on the observed data can settle it. What you can do is collect variables that predict dropout, so MAR becomes more believable, and check how far the answer moves if the missing scores were a few points lower than imputed.
5Check yourself
- Find a setting where listwise deletion misses the posttest mean by more than 4 points, while its slope stays within 0.05 of the complete-data slope.Now: listwise misses the mean by 0.3 and the slope by 0.02
- Find a setting where mean imputation cuts the pretest–posttest correlation by at least a third.Now: correlation 0.73 after mean imputation, 0.82 in the complete data
- Find a setting where multiple imputation misses the complete-data mean by more than 2 points.Now: multiple imputation gives 49.8 against 49.8, a miss of 0.0
6Run the same comparison with mice
The snippet simulates its own 200 students with the same design, deletes 40% of the posttests under MAR, and repeats the comparison with mice. R draws a different sample, so its numbers differ a little from the lab’s, but the pattern is the same. Listwise deletion and mean imputation both report a mean of 53.87 against 50.53 in the complete data. Listwise deletion keeps the slope at 0.79, mean imputation cuts the correlation from 0.77 to 0.50, and regression imputation pushes it up to 0.85. Multiple imputation returns a mean of 50.53, with a 95% interval of 49.20 to 51.85, and a slope of 0.79.
Compare four fixes for missing posttests with miceShow the code
library(mice)
# 200 students: pretest x and posttest y, correlation about 0.8
set.seed(1)
n <- 200
x <- round(rnorm(n, 50, 10), 1)
y <- round(10 + 0.8 * x + rnorm(n, 0, 6), 1)
# MAR: a low pretest makes a missing posttest more likely.
# The 40% with the highest propensity lose their posttest.
prop <- -2 * (x - 50) / 10 + rlogis(n)
d <- data.frame(x, y = ifelse(rank(-prop) <= 0.4 * n, NA, y))
# Mean, slope and correlation for a completed data set
est <- function(f) round(c(mean = mean(f$y),
slope = coef(lm(y ~ x, f))[[2]], r = cor(f$x, f$y)), 2)
fill <- function(how) complete(mice(d, method = how,
m = 1, maxit = 1, printFlag = FALSE))
cbind(complete = est(data.frame(x, y)),
listwise = est(na.omit(d)), mean = est(fill("mean")),
regression = est(fill("norm.predict")))
# Multiple imputation: 20 data sets pooled with Rubin's rules
imp <- mice(d, method = "norm", m = 20, seed = 1,
printFlag = FALSE)
pooled <- function(f) summary(pool(with(imp, f)),
conf.int = TRUE)[, c("estimate", "2.5 %", "97.5 %")]
pooled(lm(y ~ 1))
pooled(lm(y ~ x))[2, ]Uses mice. Install once with install.packages(c("mice")).
What to do about missing scores
- Listwise deletion is unbiased for a regression slope when the missingness depends only on the predictor, but not for the mean. Deleting 40% of the posttests by pretest left the slope at 0.80 and raised the mean from 49.8 to 53.7.
- Mean imputation keeps the observed mean and flattens everything else. At 40% MAR it cut the correlation from 0.82 to 0.52 and the slope to 0.34.
- Regression imputation repairs the mean and the slope under MAR but treats every guess as a measured score. Multiple imputation puts that uncertainty back by drawing m plausible values and pooling the m analyses with Rubin’s rules.
- Under MNAR every method is biased, and no test on the observed data can tell MAR from MNAR. That assumption has to be argued from how the data were collected.
- Missing item responses raise the same questions. Ability estimation uses only the items a student answered, which is sound when the skipped items are missing at random, and mixed-effects models fitted by likelihood treat missing waves the same way.
Recommended reading
- Missing data: Our view of the state of the artJoseph L. Schafer and John W. GrahamPaper · Psychological Methods, 7(2), 2002
- Inference and missing dataDonald B. RubinPaper · Biometrika, 63(3), 1976