Concept map / Multiple comparisons
Share
Multiple comparisons
Compare unadjusted tests, Bonferroni adjustment, and the Benjamini-Hochberg procedure in simulated experiments.
1Twenty-four tests, no real effects
A study measures 24 outcomes and runs a hypothesis test on each at α = 0.05. If none of the outcomes has a real effect, how likely is at least one false alarm? Each test keeps its own 5% rate, but the probability of at least one across the set is much higher.
That probability is the familywise error rate. The false discovery rate is the expected share of false rejections among all rejections, counted as zero when nothing is rejected. Bonferroni’s adjustment controls the first, and the Benjamini–Hochberg procedure controls the second. Each tile below is one test.
Of 24 tests, none has a true effect. With individual tests at α = 0.05, no test is rejected. Simulate 1,000 experiments to estimate this rule's error rates.
More controls
2Twenty true nulls: the chance of a false alarm
A study runs 20 independent tests at α = 0.05, and every null hypothesis is true. What is the chance that at least one test rejects?
3A flat cutoff or a rising line
Sort the p-values from smallest to largest. Bonferroni holds every one to the same small cutoff, α/m. Benjamini–Hochberg (BH) gives the k-th smallest a cutoff that grows with its rank, kα/m, and rejects everything up to the last p-value under that line. Point at a symbol to light it up on the chart.
Here m = 24 and α = 0.05. Bonferroni rejects p ≤ 0.05/24 = 0.0021, which no test meets here. No sorted p-value falls under its BH cutoff, so BH rejects none.
4Check yourself
The tasks use the lab in section 1 and tick themselves when you meet them.
- With no true effects and individual tests, simulate 1,000 experiments. More than half should contain at least one false rejection.Now: No simulation at these settings yet
- Use Bonferroni with at least 48 tests and simulate. The rate of at least one false rejection should stay at or below 6%.Now: 24 tests, Individual tests, no simulation yet
- With at least 6 true effects, find the rule that keeps the mean false discovery proportion at or below 5% and still detects more than half of the true effects.Now: 0 true effects, no simulation with true effects yet
5Adjust p-values in R
p.adjust turns raw p-values into adjusted ones, and each rule rejects where its adjusted value is at or below 0.05. The snippet simulates 24 tests with 6 true effects and counts rejections and false alarms under each rule. In this draw, individual tests reject 6 with 1 false alarm, while Bonferroni and BH each reject 3 with none. The last line gives the prediction’s answer: 0.642 without adjustment and 0.049 with Bonferroni.
Adjust p-values with p.adjustShow the code
# 24 tests: the first 6 have true effects (z shifted by 2.8)
set.seed(524)
m <- 24
effect <- rep(c(TRUE, FALSE), c(6, m - 6))
z <- rnorm(m) + ifelse(effect, 2.8, 0)
p <- 2 * pnorm(-abs(z))
# Rejections at alpha = 0.05 under each rule
rules <- list(
none = p,
bonferroni = p.adjust(p, method = "bonferroni"),
BH = p.adjust(p, method = "BH"))
t(sapply(rules, function(q) c(
rejected = sum(q <= 0.05),
false_alarms = sum(q <= 0.05 & !effect))))
# Chance of at least one false rejection, 20 true nulls
round(c(none = 1 - 0.95^20,
bonferroni = 1 - (1 - 0.05 / 20)^20), 3)Uses base R only.
What the adjustments trade
- With m independent true-null tests, the chance of at least one false alarm is 1 − 0.95m: 64.2% for 20 tests and 70.8% for 24.
- Testing each hypothesis at α / m, the Bonferroni rule, keeps that chance at or below α whatever the dependence between tests.
- The price is power. With 6 true effects among 24 tests, individual tests detect each one about 80% of the time and Bonferroni about 39%.
- Benjamini–Hochberg keeps the expected share of false rejections at or below α for independent tests. It finds more true effects than Bonferroni and accepts an occasional false alarm. With no true effects at all, every rejection is false and the two error rates coincide.
- A differential item functioning screen tests every item for group differences, often 40 or more at once, so the same adjustments decide which flags to trust.
Recommended reading
- An Introduction to Statistical Learning with Applications in RGareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani2nd ed., Multiple Testing
- Controlling the False Discovery RateYoav Benjamini and Yosef HochbergPaper · 1995, the step-up procedure