Concept map / Regression

Share

Simple linear regression and residuals

Move a fitted line, inspect the residuals, and compare its sum of squared residuals with the least-squares fit.

1Twenty-four students and a line you can move

Twenty-four students took two tests, and a straight line predicts the second score from the first. A student’s residual is the observed second score minus the line’s prediction. Each orange square has a residual as its side, so its area is that residual squared.

The page starts with a poor line, second = 50 + 0.1 × first. Tilt and lift it until the squares add up to as little as you can manage, then snap to the least-squares line and compare.

2040608010020406080100Second test scoreFirst test scoresum of squared residuals: 3726
StudentYour lineSquared residualPoint of means

Drag the green handles to move the line. Drag a student, or Tab to it and use the arrow keys.

Your line predicts second test = 50.0 + 0.10 × first test. Each extra point on the first test goes with 0.10 more points on the second. The sum of squared residuals is 3726, 221% above the smallest possible (1160). This line does worse than ignoring the first test and predicting the average for everyone.

More controls

2Two standard deviations up on the first test

Two tests correlate 0.5. A student scores 2 standard deviations (SDs) above average on the first test. What is the best prediction for the second?

3Five numbers fix the best line

Least squares needs no search. Its answer comes from the two means, the two standard deviations and the correlation r. The slope is r times the ratio of the spreads, and the intercept puts the line through the point of means. Point at a colored term to see it on the chart.

b  =  r×sysxa  =  yˉ  −  b xˉ\htmlClass{term-regb}{b} \;=\; r \times \htmlClass{term-regsd}{\frac{s_y}{s_x}} \qquad\quad a \;=\; \htmlClass{term-regmean}{\bar{y}} \;-\; \htmlClass{term-regb}{b}\,\htmlClass{term-regmean}{\bar{x}}

With the students as they are now: b = 0.65 × 9.35 / 13.27 = 0.46, and a = 65.75 − 0.46 × 65.17 = 35.88. sx and sy are the standard deviations of the two tests, x̄ and ȳ their means.

R2  =  1−smallest sum of squared residualssum of squares around the mean  =  r2R^2 \;=\; 1 - \frac{\htmlClass{term-regsse}{\text{smallest sum of squared residuals}}}{\htmlClass{term-regsst}{\text{sum of squares around the mean}}} \;=\; r^2
R2=1−smallest sum of squared residualssum of squares around the mean=r2\begin{aligned} R^2 &= 1 - \frac{\htmlClass{term-regsse}{\text{smallest sum of squared residuals}}}{\htmlClass{term-regsst}{\text{sum of squares around the mean}}} \\[4pt] &= r^2 \end{aligned}

With the students as they are now: 1 − 1160 / 2011 = 0.42, which is 0.65 squared. The best line accounts for 42% of the spread in second-test scores.

Why the best line is flatter than the SD line

In standard units the slope is r. A student 1 standard deviation above average on the first test (13.3 points) is predicted 0.65 standard deviations above average on the second, 6.1 points rather than 9.3. The SD line (under More controls in section 1) has slope sy / sx = 0.70, where predictions would fall if r were 1. Unless the students lie exactly on a line, the best line is the flatter of the two.

4Check yourself

The tasks use the chart and controls in section 1 and tick themselves when you meet them.

  • Start from the worst sensible guess. Set the slope to 0 and move the intercept to the average second-test score (65.8), so the line accounts for 0% of the spread.Now: Slope 0.10, intercept 50.0, line accounts for −85% of the spread
  • Fit by hand. Without the snap button, bring the sum of squared residuals within 2% of the smallest possible.Now: Sum of squares 3726, which is 221.1% above the smallest possible (1160)
  • Move students until the best-fit slope falls below 0.10. One student placed far enough from the rest can do it alone.Now: Best-fit slope 0.46, r = 0.65

5Fit the same line with lm()

lm() fits the least-squares line to the page’s 24 students: intercept 35.88 and slope 0.46, with a sum of squared residuals of 1160. The starting line, 50 + 0.1 × first, gives 3726. R² is 0.42, the squared correlation.

Fit the least-squares line with lm()Show the code
# The page's 24 students: first and second test scores
first <- c(75, 74, 68, 62, 43, 94, 83, 89, 58, 49, 71, 66,
           52, 57, 53, 65, 46, 55, 63, 76, 58, 80, 59, 68)
second <- c(74, 70, 51, 65, 56, 87, 88, 72, 64, 65, 56, 70,
            70, 58, 56, 72, 52, 66, 60, 66, 63, 69, 69, 59)
fit <- lm(second ~ first)
round(coef(fit), 2)

# The slope is r times the ratio of the two SDs
r <- cor(first, second)
round(r * sd(second) / sd(first), 2)

# Smallest sum of squared residuals, then the page's
# starting line, second = 50 + 0.1 x first
round(sum(residuals(fit)^2))
round(sum((second - (50 + 0.1 * first))^2))

# R squared equals the squared correlation
round(c(R2 = summary(fit)$r.squared, r_sq = r^2), 2)

Uses base R only.

What least squares gives you