Concept map / Likelihood
Share
Likelihood and maximum likelihood
Compare candidate probabilities of heads using the likelihood of a fixed sequence of coin tosses.
1Which bias best explains seven heads in ten?
Ten tosses of a coin came up as shown below, 7 heads and 3 tails. Pick a candidate value for p, the chance of heads, and multiply the chances of the ten outcomes at that p. Hold the tosses fixed and treat that product as a function of p, and you have the likelihood.
Slide p to see how well each value explains these tosses, or click a coin to turn it over.
Under each coin: the chance of that outcome if p = 0.50.
If the coin's bias were 0.50, this exact run of flips would turn up about 1 time in 1,024. At the peak, p = 0.70, it is 1 time in 450.
2Same proportion, ten times the flips
This is the likelihood for 7 heads in 10 flips. Keep flipping until you have 70 heads in 100: the same proportion from ten times the data. What happens to the curve?
3Climbing to the peak, one step at a time
The log of the likelihood turns the product into a sum and peaks at the same p. Its slope is zero at the top, unless the flips are all heads or all tails and the top is an endpoint. Software finds the top by climbing. Here you can watch two ways of doing it.
From p = 0.50, press Step or Run. Gradient ascent moves by a fixed multiple of the slope, so its steps shrink as the slope flattens.
At p = 0.50 the slope is 7/0.50 − 3/0.50 = 8.00. Setting the slope to zero gives p = h/n = 0.70. Point at a symbol to see its part of the chart.
4Check yourself
The tasks use the coins in section 1 and the climbing controls in section 3, and tick themselves when you meet them.
- Click coins until the peak sits at exactly 0.30.Now: Peak now at p = 0.70
- In section 3, choose Newton, press "Start from p = 0.10" and step until you reach the peak. Then count the steps gradient ascent needs.Now: 0 steps taken, slope 8.00
- Make the likelihood keep rising all the way to p = 1.Now: 7 heads, 3 tails
5Find the peak in R
The snippet uses the same ten tosses. It prints the likelihood at 0.5 and at the peak, 0.7, then takes five Newton steps from p = 0.10, the start used in the Newton task.
Maximize a Bernoulli likelihoodShow the code
# The page's ten tosses: 1 is heads, 0 is tails
tosses <- c(1, 1, 0, 1, 1, 1, 0, 1, 0, 1)
h <- sum(tosses)
n <- length(tosses)
loglik <- function(p) h * log(p) + (n - h) * log(1 - p)
# Likelihood at the starting value 0.5 and at h / n = 0.7
signif(exp(loglik(c(0.5, 0.7))), 3)
# Relative likelihood of 0.5 compared with the peak
round(exp(loglik(0.5) - loglik(0.7)), 3)
# Maximize numerically
best <- optimize(loglik, c(0.01, 0.99), maximum = TRUE)
round(best$maximum, 3)
# Newton's method from p = 0.1, using slope and curvature
p <- 0.1
for (i in 1:5) {
slope <- h / p - (n - h) / (1 - p)
curve <- -h / p^2 - (n - h) / (1 - p)^2
p <- p - slope / curve
cat(round(p, 4), "")
}
cat("\n")Uses base R only.
What the likelihood tells you
- The likelihood ranks values of p by how probable they make the tosses you saw. It is not the probability that p is right.
- For h heads in n tosses the maximum likelihood estimate is h / n. With all heads or all tails the peak sits at an endpoint, 1 or 0.
- The same proportion from more tosses gives a narrower curve. At 7 heads in 10 a fair coin still reaches 44% of the peak height; at 70 in 100 it reaches 0.03%.
- Software climbs to the peak. Gradient ascent follows the slope; Newton’s method also uses the curvature and can overshoot from a poor start, so programs add safeguards.
- Ability estimation multiplies the chances of a person’s item responses in the same way, with θ in place of p.
Recommended reading
- Statistical InferenceGeorge Casella and Roger L. Berger2nd ed., Point Estimation