Concept map / Logistic regression

Share

Logistic regression for pass and fail

Fit an S-shaped curve to pass/fail outcomes, read the slope as a change in log odds, and see that an item curve is the same model.

Prerequisites Regression, LikelihoodNext topics IRT: item curves, DIF

1Forty students, pass or fail

Forty students report how many hours they studied, and each one passed the exam or failed it. A regression line is the obvious tool, but the outcome is 0 or 1, and what you want to predict is a chance, which has to stay between 0 and 1. Logistic regression fits its straight line to the log odds of passing instead, and on the probability scale that line becomes an S-shaped curve.

Move the intercept β₀ and the slope β₁ to fit the curve by eye. The log-likelihood, the likelihood of all 40 outcomes on a log scale, rises as the curve gets better, and the fit button finds the curve where it peaks. The x slider reads off the chance of passing at any number of hours.

00.250.50.7510246810Chance of passingHours of study0.38 at 5.0 h
Student (top row passed, bottom row failed)Your curve

At 5.0 hours the log odds are −2.0 + 0.30 × 5.0 = −0.50, so the odds are 0.61 to 1 and the chance of passing is 38%. Each extra hour multiplies the odds by 1.35. The log-likelihood is −22.11; the best curve reaches −17.01.

2One more hour, at 2 hours or at 5

On the maximum likelihood curve, each extra hour multiplies the odds of passing by 2.25. Student A has studied 2 hours and has a 0.09 chance of passing. Student B has studied 5 hours and has 0.52. Each studies one hour more. Whose chance of passing goes up by more?

00.51246810Chance of passingHours of studyAB

3Slopes live on the log-odds scale

The model is a straight line, but for the log odds of passing, not for the chance itself. Odds are p / (1 − p): a 75% chance is odds of 3 to 1. Log odds run from minus to plus infinity, so a line can never leave their range, and the logistic function bends that line back into an S between 0 and 1.

log⁡p1−p  =  β0+β1 x\log \frac{\htmlClass{term-p}{p}}{1 - \htmlClass{term-p}{p}} \;=\; \htmlClass{term-b}{\beta_0} + \htmlClass{term-a}{\beta_1}\, \htmlClass{term-theta}{x}
p  =  11+e−(β0+β1x)\htmlClass{term-p}{p} \;=\; \frac{1}{1 + e^{-(\htmlClass{term-b}{\beta_0} + \htmlClass{term-a}{\beta_1} \htmlClass{term-theta}{x})}}

At x = 5.0: log odds = −2.0 + 0.30 × 5.0 = −0.50; odds = e−0.50 = 0.61; p = 38%. Point at a symbol to find it on the chart in section 1.

odds at x+1odds at x  =  eβ1\frac{\text{odds at } \htmlClass{term-theta}{x} + 1}{\text{odds at } \htmlClass{term-theta}{x}} \;=\; e^{\htmlClass{term-a}{\beta_1}}

e0.30 = 1.35. This ratio is the same at every x, which is why β₁ is reported as an odds ratio. The change in probability is not the same: it is largest near 50%, where the curve is steepest.

How the fit button finds β₀ and β₁

Each passing student contributes log pi and each failing student log(1 − pi). The sum is the log-likelihood:

ℓ=∑i[ yilog⁡pi+(1−yi)log⁡(1−pi)]\begin{aligned} \ell = \sum_{i} \Big[\, & y_i \log p_i \\ & + (1 - y_i) \log (1 - p_i) \Big] \end{aligned}

No formula gives the maximum directly. The button climbs to it with Newton–Raphson steps, 7 of them here, the same method glm() uses under the name iteratively reweighted least squares. The result is β₀ = −3.971 and β₁ = 0.811, with standard errors 1.29 and 0.24.

4Why a straight line will not do

Least squares will happily fit a line to 0s and 1s. On these students it rises 0.133 per hour everywhere, and it promises a chance of −0.16 at 0 hours and 1.17 at 10. Neither is a probability. The flat slope also misses the shape of the data: almost nobody passes below 3 hours and almost everybody passes above 7.

00.510246810Predicted chanceHours of study−0.161.17
Least-squares lineLogistic curveNot a probability

5An item curve is this same model

Replace hours with ability θ and pass with a correct answer, and the logistic curve becomes the two-parameter item characteristic curve. Only the way it is written changes. The slope is the discrimination a, and the difficulty b is the point where the curve crosses 50%.

P(correct∣θ)  =  11+e−a (θ−b)P(\text{correct} \mid \theta) \;=\; \frac{1}{1 + e^{-\htmlClass{term-a}{a}\,(\theta - \htmlClass{term-c}{b})}}
a=β1,b=− β0/β1\htmlClass{term-a}{a} = \htmlClass{term-a}{\beta_1}, \qquad \htmlClass{term-c}{b} = -\,\htmlClass{term-b}{\beta_0} / \htmlClass{term-a}{\beta_1}

Read as an item, the curve in section 1 has a = 0.30 and b = −(−2.0) / 0.30 = 6.67 hours. mirt reports the intercept itself as d, so its d is β₀.

The one real difference is that ability is never observed. Here hours are measured and the fit is an ordinary regression; in IRT, θ has to be estimated together with a and b, which is what calibration does.

6Check yourself

  • Without the fit button, move β₀ and β₁ until the log-likelihood is within 0.2 of its maximum, −17.01.Now: log-likelihood −22.11, which is 5.09 below the maximum
  • Fit the curve by maximum likelihood, then find the hours of study that give an even chance of passing.Now: your own curve is showing, not the fitted one
  • On your own curve, build an item with discrimination a = 1.5 and difficulty b = 4 hours.Now: a = 0.30, b = 6.67 hours

7Fit the same model with glm()

The snippet types in the lab’s 40 students and fits them with glm(family = binomial). It prints β₀ = −3.971 and β₁ = 0.811, a log-likelihood of −17.01, an odds ratio of 2.25 and a 50% point of 4.90 hours, the numbers the fit button gives. The chances at 2, 3, 5 and 6 hours are the ones in section 2, and the last line is the straight line from section 4.

Fit a logistic regression with glm()Show the code
# The lab's 40 students: hours of study, pass (1) or fail (0)
hours <- c(1, 1.4, 1.5, 1.5, 1.9, 2.3, 2.3, 2.4, 3, 3.1, 3.3,
  3.5, 3.8, 4.2, 4.4, 4.5, 4.6, 4.7, 5, 5.4, 5.4, 5.8, 6.1,
  6.2, 6.5, 6.6, 6.7, 6.9, 7, 7, 7.1, 7.4, 7.9, 8.2, 8.6, 8.7,
  8.9, 9, 9.3, 9.7)
pass <- c(0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0,
  1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
  1, 1)

# Maximum likelihood fit of log odds = b0 + b1 * hours
fit <- glm(pass ~ hours, family = binomial)
round(coef(fit), 3)
logLik(fit)

# Odds ratio for one more hour, and the 50% point -b0 / b1
exp(coef(fit)[["hours"]])
-coef(fit)[[1]] / coef(fit)[[2]]

# Chance of passing at 2, 3, 5 and 6 hours
new <- data.frame(hours = c(2, 3, 5, 6))
round(predict(fit, new, type = "response"), 3)

# A straight line predicts below 0 and above 1
line <- lm(pass ~ hours)
round(predict(line, data.frame(hours = c(0, 10))), 2)

Uses base R only.

What the curve tells you