Concept map / Regression
Share
Simple linear regression and residuals
Move a fitted line, inspect the residuals, and compare its sum of squared residuals with the least-squares fit.
1Twenty-four students and a line you can move
Twenty-four students took two tests, and a straight line predicts the second score from the first. A student’s residual is the observed second score minus the line’s prediction. Each orange square has a residual as its side, so its area is that residual squared.
The page starts with a poor line, second = 50 + 0.1 × first. Tilt and lift it until the squares add up to as little as you can manage, then snap to the least-squares line and compare.
Drag the green handles to move the line. Drag a student, or Tab to it and use the arrow keys.
Your line predicts second test = 50.0 + 0.10 × first test. Each extra point on the first test goes with 0.10 more points on the second. The sum of squared residuals is 3726, 221% above the smallest possible (1160). This line does worse than ignoring the first test and predicting the average for everyone.
More controls
2Two standard deviations up on the first test
Two tests correlate 0.5. A student scores 2 standard deviations (SDs) above average on the first test. What is the best prediction for the second?
3Five numbers fix the best line
Least squares needs no search. Its answer comes from the two means, the two standard deviations and the correlation r. The slope is r times the ratio of the spreads, and the intercept puts the line through the point of means. Point at a colored term to see it on the chart.
With the students as they are now: b = 0.65 × 9.35 / 13.27 = 0.46, and a = 65.75 − 0.46 × 65.17 = 35.88. sx and sy are the standard deviations of the two tests, x̄ and ȳ their means.
With the students as they are now: 1 − 1160 / 2011 = 0.42, which is 0.65 squared. The best line accounts for 42% of the spread in second-test scores.
Why the best line is flatter than the SD line
In standard units the slope is r. A student 1 standard deviation above average on the first test (13.3 points) is predicted 0.65 standard deviations above average on the second, 6.1 points rather than 9.3. The SD line (under More controls in section 1) has slope sy / sx = 0.70, where predictions would fall if r were 1. Unless the students lie exactly on a line, the best line is the flatter of the two.
4Check yourself
The tasks use the chart and controls in section 1 and tick themselves when you meet them.
- Start from the worst sensible guess. Set the slope to 0 and move the intercept to the average second-test score (65.8), so the line accounts for 0% of the spread.Now: Slope 0.10, intercept 50.0, line accounts for −85% of the spread
- Fit by hand. Without the snap button, bring the sum of squared residuals within 2% of the smallest possible.Now: Sum of squares 3726, which is 221.1% above the smallest possible (1160)
- Move students until the best-fit slope falls below 0.10. One student placed far enough from the rest can do it alone.Now: Best-fit slope 0.46, r = 0.65
5Fit the same line with lm()
lm() fits the least-squares line to the page’s 24 students: intercept 35.88 and slope 0.46, with a sum of squared residuals of 1160. The starting line, 50 + 0.1 × first, gives 3726. R² is 0.42, the squared correlation.
Fit the least-squares line with lm()Show the code
# The page's 24 students: first and second test scores
first <- c(75, 74, 68, 62, 43, 94, 83, 89, 58, 49, 71, 66,
52, 57, 53, 65, 46, 55, 63, 76, 58, 80, 59, 68)
second <- c(74, 70, 51, 65, 56, 87, 88, 72, 64, 65, 56, 70,
70, 58, 56, 72, 52, 66, 60, 66, 63, 69, 69, 59)
fit <- lm(second ~ first)
round(coef(fit), 2)
# The slope is r times the ratio of the two SDs
r <- cor(first, second)
round(r * sd(second) / sd(first), 2)
# Smallest sum of squared residuals, then the page's
# starting line, second = 50 + 0.1 x first
round(sum(residuals(fit)^2))
round(sum((second - (50 + 0.1 * first))^2))
# R squared equals the squared correlation
round(c(R2 = summary(fit)$r.squared, r_sq = r^2), 2)Uses base R only.
What least squares gives you
- The least-squares line has the smallest sum of squared residuals of any line. Squaring gives big misses more weight than small ones, so one far-off student can pull the line a long way.
- The line passes through the point of means, and its slope is r × sy / sx. A slope describes an association; reading it as cause and effect needs more than the data on this page.
- R² compares the leftover squares with the squares around the mean. With one predictor and an intercept it equals r², 0.42 for these students.
- In standard units the slope is r, so when r is below 1 the prediction sits closer to the mean than the first score did. That is regression to the mean.
- A regression crosswalk in linking predicts one test from another, and for the same reason its predicted scores are less spread out than the real ones.
Recommended reading
- Regression and Other StoriesAndrew Gelman, Jennifer Hill, and Aki VehtariChapters 8 and 12, least squares and regression to the mean