Concept map / Regression diagnostics

Share

Regression diagnostics

Read residual patterns, leverage, and Cook's distance, and compare the interval for a mean response with the interval for one new observation.

1What the line leaves behind

A fitted regression line assumes a straight-line mean, the same spread around it everywhere, and no single student steering the fit. A residual is a student’s observed score minus the fitted one. Plotted against hours of practice, the residuals should form a flat band around zero; a bend or a fan means the line is wrong in a way the top chart hides.

The line also answers two questions with different precision: what students who practice 5 hours score on average, and what one new student who practices 5 hours will score. Switch the data pattern, move the prediction point, and add an extra student to see how far one point can pull the line.

10305070−60602468101214Test scoreResidual (observed − fitted)Hours of practice
Fitted line95% band for the mean95% band for one studentLargest Cook’s distance

The residuals scatter evenly around zero. The fitted line is score = 20.20 + 1.95 × hours with residual standard deviation 2.04. At 5.00 hours the mean score lies between 29.1 and 30.8, but one new student could score anywhere from 25.7 to 34.3. The largest Cook’s distance is 0.18, for the student at 1.7 hours.

2Four times the data at 5 hours

With the straight-line data, the 95% interval at 5 hours is ±0.86 for the mean score and ±4.31 for one new student. Suppose each of the 24 hours values had been observed four times, giving 96 students with the same residual spread. What happens to the two intervals at 5 hours?

Interval95% interval around the fitted mean (score points)−4−2024Mean, n = 24±0.86Mean, n = 96One student, n = 24±4.31One student, n = 96

3Why one interval stops shrinking

Leverage h measures how far a predictor value sits from the mean of the predictors, relative to their spread. The interval for the mean grows with √h. The prediction interval adds one student’s own variation around that mean, the 1 under the square root, so it never gets narrower than about ±t × s. Cook’s distance D combines a student’s residual e with its leverage to measure how far the line would move without that student.

h0=1n+(x0−xˉ)2Sxx,y^0±t sh0 (mean),y^0±t s1+h0 (one student)\htmlClass{term-theta}{h_0} = \frac{1}{n} + \frac{(x_0-\bar x)^2}{S_{xx}},\qquad \hat y_0 \pm t\,s\sqrt{\htmlClass{term-theta}{h_0}}\ \text{(mean)},\qquad \hat y_0 \pm t\,s\sqrt{\htmlClass{term-b}{1} + \htmlClass{term-theta}{h_0}}\ \text{(one student)}
h0=1n+(x0−xˉ)2Sxx,y^0±t sh0 (mean),y^0±t s1+h0 (one student)\begin{gathered} \htmlClass{term-theta}{h_0} = \frac{1}{n} + \frac{(x_0-\bar x)^2}{S_{xx}}, \\[6pt] \hat y_0 \pm t\,s\sqrt{\htmlClass{term-theta}{h_0}}\ \text{(mean)}, \\[4pt] \hat y_0 \pm t\,s\sqrt{\htmlClass{term-b}{1} + \htmlClass{term-theta}{h_0}}\ \text{(one student)} \end{gathered}

Here n = 24, x̄ = 5.00, Sxx = 139.1, s = 2.04 and t = 2.074, so h₀ = 0.042. The mean interval is 30.0 ± 0.86 and the prediction interval is 30.0 ± 4.31.

Di=ei22 s2⋅hi(1−hi)2D_i = \frac{\htmlClass{term-a}{e_i}^{2}}{2\,s^{2}}\cdot\frac{h_i}{(1-h_i)^{2}}

Add the extra student to see this worked out. A common rough flag is D above 1, or above 4/n.

4Check yourself

The tasks use the controls in section 1 and tick themselves when you meet them.

  • Move the prediction location until the confidence interval for the mean is at least half as wide as the prediction interval.Now: At 5.00 hours the ratio of widths is 0.20
  • Add the extra student and place it so that its Cook’s distance is greater than 1.Now: No extra student yet
  • Place the extra student where its leverage is above 0.30 but its Cook’s distance stays below 0.10.Now: No extra student yet

5Residuals, leverage and influence with lm()

The snippet rebuilds the page’s straight-line data from the same 24 offsets, so the coefficients (20.20 and 1.95) and both intervals at 5 hours match the lab: 29.10 to 30.83 for the mean, 25.66 to 34.28 for one student. It then adds a student at 13 hours who scores 21 and prints that student’s residual (−16.37), leverage (0.33) and Cook’s distance (4.70).

Residuals, leverage, influence, and intervals with lmShow the code
# The page's straight-line data: 24 students, 1 to 9 hours
e <- c(-0.4, 0.7, -1.5, -0.6, -0.6, 1.5, -1, 1.2, 1.9,
       -0.9, 0.3, 0, -0.4, 0.6, 0.3, 1.7, -1.8, 0.2,
       -0.7, 0.6, 0.2, -0.9, 0.4, -1.2)
d <- data.frame(hours = 1 + 8 * (0:23) / 23)
d$score <- 20 + 2 * d$hours + 2 * e
fit <- lm(score ~ hours, data = d)
round(coef(fit), 2)

# Mean and one-student intervals at 5 hours
new <- data.frame(hours = 5)
predict(fit, new, interval = "confidence")
predict(fit, new, interval = "prediction")

# Add one student at 13 hours who scores 21
d2 <- rbind(d, data.frame(hours = 13, score = 21))
fit2 <- lm(score ~ hours, data = d2)
round(c(residual = resid(fit2)[25],
        leverage = hatvalues(fit2)[25],
        cook = cooks.distance(fit2)[25]), 2)
round(coef(fit2), 2)

Uses base R only.

Reading the leftovers