Concept map / Missing data

Share

Missing data and multiple imputation

Lose posttest scores under MCAR, MAR or MNAR, then compare listwise deletion, mean imputation, regression imputation and multiple imputation against the complete data.

Prerequisites Regression, Likelihood

1Two hundred students, some posttests lost

Two hundred students take a pretest in September and a posttest in May, and some of the posttests never arrive. What to do about them depends on why they are missing. If the loss has nothing to do with the students (MCAR, missing completely at random), the rest are a random subset. If it depends on something you observed, such as the pretest (MAR, missing at random), it can be corrected. If it depends on the missing score itself (MNAR), the data alone cannot correct it.

The lab knows every true posttest, so you can see what each fix does. Choose why scores go missing and how many, then compare four fixes with the complete data on two numbers: the mean posttest and the regression slope of posttest on pretest.

2040608020304050607080PosttestPretest
ObservedMissing (true score, unseen)Fitted lineComplete data
ListwiseMean imp.RegressionMultiple imp.45505560Mean of posttest0.00.40.81.2Slope on pretest
Estimate with 95% intervalComplete-data value

Listwise deletion keeps the 140 students who have both scores. Their posttest mean is 50.1 (complete data 49.8) and their slope is 0.78 (complete data 0.80).

More controls

m is the number of imputed data sets that multiple imputation pools.

2Dropping students by their pretest

Take the 200 students in the lab, all with a posttest, and delete 40% of the posttests, mostly from students with low pretest scores. That is MAR: whether a score is missing depends on the pretest, which you still have. You analyze only the 120 complete cases. What happens to the estimates?

3Pooling m analyses with Rubin’s rules

Regression imputation fills each gap with the single best guess, so the analysis treats a guess as a measured score. Multiple imputation draws m different plausible values for every gap instead, adding the scatter around the line and the uncertainty in the line itself. Each completed data set is analyzed as if nothing were missing, and the m results are combined.

Qˉ  =  1m∑j=1mQ^j\htmlClass{term-qbar}{\bar{Q}} \;=\; \frac{1}{m} \sum_{j=1}^{m} \hat{Q}_j
T  =  Uˉ  +  (1+1m) B\htmlClass{term-total}{T} \;=\; \bar{U} \;+\; \Big(1 + \frac{1}{m}\Big)\, \htmlClass{term-between}{B}

For the posttest mean: Q̄ = 49.82, the average of 20 estimates. Ū = 0.437 is their average squared standard error and B = 0.073 is the variance between them, so T = 0.437 + 1.05 × 0.073 = 0.514 and the standard error is √T = 0.72. Missing data account for 15% of T. Point at Q̄, B or T to find it on the multiple imputation row of the estimates chart.

B is what single imputation leaves out. A regression-imputed data set behaves as if all 200 posttests had been observed, so its interval for the mean is too narrow; here it is 2.45 points wide, against 2.84 for multiple imputation.

4What the observed data cannot tell you

The 60 students without a posttest averaged 48.8 on the pretest, against 49.9 for the 140 with one. Under MCAR the two groups differ only by chance. A large gap would rule MCAR out.

Both mechanisms leave the same kind of data behind: a full set of pretests and posttests missing mostly among low scorers. Whether the missing students did worse than their pretest predicts is a fact about the scores you do not have, so no test on the observed data can settle it. What you can do is collect variables that predict dropout, so MAR becomes more believable, and check how far the answer moves if the missing scores were a few points lower than imputed.

5Check yourself

  • Find a setting where listwise deletion misses the posttest mean by more than 4 points, while its slope stays within 0.05 of the complete-data slope.Now: listwise misses the mean by 0.3 and the slope by 0.02
  • Find a setting where mean imputation cuts the pretest–posttest correlation by at least a third.Now: correlation 0.73 after mean imputation, 0.82 in the complete data
  • Find a setting where multiple imputation misses the complete-data mean by more than 2 points.Now: multiple imputation gives 49.8 against 49.8, a miss of 0.0

6Run the same comparison with mice

The snippet simulates its own 200 students with the same design, deletes 40% of the posttests under MAR, and repeats the comparison with mice. R draws a different sample, so its numbers differ a little from the lab’s, but the pattern is the same. Listwise deletion and mean imputation both report a mean of 53.87 against 50.53 in the complete data. Listwise deletion keeps the slope at 0.79, mean imputation cuts the correlation from 0.77 to 0.50, and regression imputation pushes it up to 0.85. Multiple imputation returns a mean of 50.53, with a 95% interval of 49.20 to 51.85, and a slope of 0.79.

Compare four fixes for missing posttests with miceShow the code
library(mice)
# 200 students: pretest x and posttest y, correlation about 0.8
set.seed(1)
n <- 200
x <- round(rnorm(n, 50, 10), 1)
y <- round(10 + 0.8 * x + rnorm(n, 0, 6), 1)
# MAR: a low pretest makes a missing posttest more likely.
# The 40% with the highest propensity lose their posttest.
prop <- -2 * (x - 50) / 10 + rlogis(n)
d <- data.frame(x, y = ifelse(rank(-prop) <= 0.4 * n, NA, y))

# Mean, slope and correlation for a completed data set
est <- function(f) round(c(mean = mean(f$y),
  slope = coef(lm(y ~ x, f))[[2]], r = cor(f$x, f$y)), 2)
fill <- function(how) complete(mice(d, method = how,
  m = 1, maxit = 1, printFlag = FALSE))
cbind(complete = est(data.frame(x, y)),
  listwise = est(na.omit(d)), mean = est(fill("mean")),
  regression = est(fill("norm.predict")))

# Multiple imputation: 20 data sets pooled with Rubin's rules
imp <- mice(d, method = "norm", m = 20, seed = 1,
  printFlag = FALSE)
pooled <- function(f) summary(pool(with(imp, f)),
  conf.int = TRUE)[, c("estimate", "2.5 %", "97.5 %")]
pooled(lm(y ~ 1))
pooled(lm(y ~ x))[2, ]

Uses mice. Install once with install.packages(c("mice")).

What to do about missing scores