Concept map / Likelihood

Share

Likelihood and maximum likelihood

Compare candidate probabilities of heads using the likelihood of a fixed sequence of coin tosses.

1Which bias best explains seven heads in ten?

Ten tosses of a coin came up as shown below, 7 heads and 3 tails. Pick a candidate value for p, the chance of heads, and multiply the chances of the ten outcomes at that p. Hold the tosses fixed and treat that product as a function of p, and you have the likelihood.

Slide p to see how well each value explains these tosses, or click a coin to turn it over.

Under each coin: the chance of that outcome if p = 0.50.

0maxLikelihood of these flips00.20.40.60.81Coin bias p (chance of heads)peak at p = 0.70

If the coin's bias were 0.50, this exact run of flips would turn up about 1 time in 1,024. At the peak, p = 0.70, it is 1 time in 450.

2Same proportion, ten times the flips

This is the likelihood for 7 heads in 10 flips. Keep flipping until you have 70 heads in 100: the same proportion from ten times the data. What happens to the curve?

0max00.20.40.60.81LikelihoodCoin bias p7 heads in 10 flips

3Climbing to the peak, one step at a time

The log of the likelihood turns the product into a sum and peaks at the same p. Its slope is zero at the top, unless the flips are all heads or all tails and the top is an endpoint. Software finds the top by climbing. Here you can watch two ways of doing it.

-6-11-16Log-likelihood ℓ(p)00.20.40.60.81Coin bias p (chance of heads)slope +8.0: go right

From p = 0.50, press Step or Run. Gradient ascent moves by a fixed multiple of the slope, so its steps shrink as the slope flattens.

ℓ(p)  =  h log⁡p  +  (n−h) log⁡(1−p)\ell(\htmlClass{term-p}{p}) \;=\; \htmlClass{term-h}{h}\,\log \htmlClass{term-p}{p} \;+\; \htmlClass{term-t}{(n-h)}\,\log(1-\htmlClass{term-p}{p})
slope  =  hp  −  n−h1−p\text{slope} \;=\; \frac{\htmlClass{term-h}{h}}{\htmlClass{term-p}{p}} \;-\; \frac{\htmlClass{term-t}{n-h}}{1-\htmlClass{term-p}{p}}

At p = 0.50 the slope is 7/0.50 − 3/0.50 = 8.00. Setting the slope to zero gives p = h/n = 0.70. Point at a symbol to see its part of the chart.

4Check yourself

The tasks use the coins in section 1 and the climbing controls in section 3, and tick themselves when you meet them.

  • Click coins until the peak sits at exactly 0.30.Now: Peak now at p = 0.70
  • In section 3, choose Newton, press "Start from p = 0.10" and step until you reach the peak. Then count the steps gradient ascent needs.Now: 0 steps taken, slope 8.00
  • Make the likelihood keep rising all the way to p = 1.Now: 7 heads, 3 tails

5Find the peak in R

The snippet uses the same ten tosses. It prints the likelihood at 0.5 and at the peak, 0.7, then takes five Newton steps from p = 0.10, the start used in the Newton task.

Maximize a Bernoulli likelihoodShow the code
# The page's ten tosses: 1 is heads, 0 is tails
tosses <- c(1, 1, 0, 1, 1, 1, 0, 1, 0, 1)
h <- sum(tosses)
n <- length(tosses)
loglik <- function(p) h * log(p) + (n - h) * log(1 - p)

# Likelihood at the starting value 0.5 and at h / n = 0.7
signif(exp(loglik(c(0.5, 0.7))), 3)
# Relative likelihood of 0.5 compared with the peak
round(exp(loglik(0.5) - loglik(0.7)), 3)

# Maximize numerically
best <- optimize(loglik, c(0.01, 0.99), maximum = TRUE)
round(best$maximum, 3)

# Newton's method from p = 0.1, using slope and curvature
p <- 0.1
for (i in 1:5) {
  slope <- h / p - (n - h) / (1 - p)
  curve <- -h / p^2 - (n - h) / (1 - p)^2
  p <- p - slope / curve
  cat(round(p, 4), "")
}
cat("\n")

Uses base R only.

What the likelihood tells you