Concept map / Multicollinearity
Share
Multicollinearity in regression
Change the correlation between two predictors and compare the precision of their coefficients with the precision of predictions.
1Practice time or problems solved?
Forty-eight students report two study habits, practice time and problems solved, and then take a test. In the simulation each habit adds 4 points per standard deviation (SD) when the other is held fixed, and a multiple regression tries to recover both effects. When the habits are strongly correlated, students who practice more also solve more problems, and the data say little about either habit on its own. That is multicollinearity.
Raise the correlation and watch the two coefficient intervals. Then compare the prediction for a typical student, high on both habits, with an unusual one, high on one and low on the other.
At a correlation of 0.40 the VIF is 1.19, so each coefficient’s interval is 1.09 times as wide as it would be with uncorrelated habits. Practice time comes out at 3.4, with a 95% interval from 1.0 to 5.7. The typical student’s mean score is predicted at 71.9 ± 4.6, against ±6.6 for the unusual one.
2Does the typical student's interval widen too?
In the lab’s first class, raising the correlation from 0.40 to 0.95 widens the practice-time interval from ±2.3 to ±6.9, about 2.9 times. The typical student is 1.6 SD above average on both habits, a common combination in these data. What happens to the 95% interval for that student’s predicted mean score?
3How the correlation inflates a standard error
A coefficient is estimated from the part of its predictor that the other predictors cannot explain. The part they do explain is R²ⱼ, from regressing predictor j on the others, and with two predictors it is r². The variance inflation factor (VIF) is 1 / (1 − R²ⱼ), and the standard error grows with its square root. Point at a term to light it up.
Here r = 0.40, so VIF = 1 / (1 − 0.40²) = 1.19. With residual standard deviation s = 7.32, predictors standardized to SD 1, and n = 48, SE = 7.32 / √47 × √1.19 = 1.16, and the 95% interval is ± 2.014 × 1.16 = ±2.35.
4Check yourself
The tasks use the controls in section 1 and tick themselves when you meet them.
- Raise the correlation until the variance inflation factor passes 10, a common flag for trouble.Now: r = 0.40, VIF = 1.19
- Find a class and correlation where the 95% interval for practice time includes zero, though its true coefficient is 4.Now: Practice time 3.4, interval 1.0 to 5.7
- Make the unusual student’s interval at least three times as wide as the typical student’s.Now: ±6.6 against ±4.6, ratio 1.43
5The VIF by hand in R
The snippet builds standardized predictors with a correlation of exactly 0.40 and then 0.95, fits lm, and computes the VIF as 1 / (1 − R²) from regressing one predictor on the other. The VIF goes from 1.19 to 10.26 and the standard error of the practice-time coefficient from 1.43 to 4.20. Over the same change the interval for the typical student’s mean score narrows a little, from ±5.68 to ±5.01. R draws a different class from the lab’s, so its coefficients differ, but the VIFs are the same.
VIF by hand and prediction intervals with lmShow the code
set.seed(64)
# 48 students; each habit adds 4 points per SD, error SD 8
n <- 48
# Standardized predictors built to correlate exactly r, as on the page
time <- as.numeric(scale(rnorm(n)))
z <- resid(lm(rnorm(n) ~ time))
z <- z / sd(z)
e <- rnorm(n, sd = 8)
fit_at <- function(r) {
problems <- r * time + sqrt(1 - r^2) * z
score <- 60 + 4 * time + 4 * problems + e
fit <- lm(score ~ time + problems)
# VIF by hand: regress one predictor on the other
r2 <- summary(lm(time ~ problems))$r.squared
typical <- data.frame(time = 1.6, problems = 1.6)
ci <- predict(fit, typical, interval = "confidence")
c(r = r, vif = 1 / (1 - r2),
se_time = coef(summary(fit))["time", "Std. Error"],
typical_halfwidth = (ci[, "upr"] - ci[, "lwr"]) / 2)
}
round(rbind(fit_at(0.4), fit_at(0.95)), 2)Uses base R only.
What correlated predictors cost
- When two predictors move together, the data cannot tell which one deserves the effect, so each coefficient is estimated loosely.
- The variance inflation factor is 1 / (1 − R²ⱼ), where R²ⱼ comes from regressing predictor j on the others, and the standard error grows with its square root. A correlation of 0.40 gives a VIF of 1.19; 0.95 gives one above 10, and 0.99 about 50.
- With strong collinearity a coefficient’s interval can include zero even though the predictor has a real effect of 4 points.
- Predictions for combinations like those in the data stay precise. A student high on one habit and low on the other sits where the data are thin, and that interval widens sharply.
- Correlated predictors are why factor analysis groups related items into one factor, and why correlated dimensions in multidimensional IRT are hard to separate.
Recommended reading
- Regression and Other StoriesAndrew Gelman, Jennifer Hill, and Aki VehtariChapter 10, adding predictors and interpreting regression coefficients
- STAT 501: Regression MethodsPenn State Department of StatisticsCourse notes · Lesson 12, multicollinearity and variance inflation factors