Concept map / Multiple comparisons

Share

Multiple comparisons

Compare unadjusted tests, Bonferroni adjustment, and the Benjamini-Hochberg procedure in simulated experiments.

1Twenty-four tests, no real effects

A study measures 24 outcomes and runs a hypothesis test on each at α = 0.05. If none of the outcomes has a real effect, how likely is at least one false alarm? Each test keeps its own 5% rate, but the probability of at least one across the set is much higher.

That probability is the familywise error rate. The false discovery rate is the expected share of false rejections among all rejections, counted as zero when nothing is rejected. Bonferroni’s adjustment controls the first, and the Benjamini–Hochberg procedure controls the second. Each tile below is one test.

Null 1p 0.666
Null 2p 0.830
Null 3p 0.061
Null 4p 0.574
Null 5p 0.977
Null 6p 0.247
Null 7p 0.506
Null 8p 0.769
Null 9p 0.830
Null 10p 0.337
Null 11p 0.378
Null 12p 0.587
Null 13p 0.725
Null 14p 0.453
Null 15p 0.936
Null 16p 0.733
Null 17p 0.938
Null 18p 0.085
Null 19p 0.078
Null 20p 0.650
Null 21p 0.586
Null 22p 0.069
Null 23p 0.294
Null 24p 0.779

Of 24 tests, none has a true effect. With individual tests at α = 0.05, no test is rejected. Simulate 1,000 experiments to estimate this rule's error rates.

More controls

2Twenty true nulls: the chance of a false alarm

A study runs 20 independent tests at α = 0.05, and every null hypothesis is true. What is the chance that at least one test rejects?

3A flat cutoff or a rising line

Sort the p-values from smallest to largest. Bonferroni holds every one to the same small cutoff, α/m. Benjamini–Hochberg (BH) gives the k-th smallest a cutoff that grows with its rank, kα/m, and rejects everything up to the last p-value under that line. Point at a symbol to light it up on the chart.

00.0250.050.0750.16121824p-value (axis stops at 0.10)Rank k of the p-value, smallest first
p-value rejected by the current ruleα = 0.05BH line kα/mBonferroni α/m
Bonferroni: reject when pi  ≤  α/m\text{Bonferroni: reject when } \htmlClass{term-pk}{p_i} \;\le\; \htmlClass{term-bonf}{\alpha / m}
BH: reject the k smallest, with k=max⁡{k:p(k)≤km α}\text{BH: reject the } k \text{ smallest, with } k = \max\Big\{k : \htmlClass{term-pk}{p_{(k)}} \le \htmlClass{term-bh}{\tfrac{k}{m}\,\alpha}\Big\}
BH: reject the k smallest, withk=max⁡{k:p(k)≤km α}\begin{gathered} \text{BH: reject the } k \text{ smallest, with} \\[4pt] k = \max\Big\{k : \htmlClass{term-pk}{p_{(k)}} \le \htmlClass{term-bh}{\tfrac{k}{m}\,\alpha}\Big\} \end{gathered}

Here m = 24 and α = 0.05. Bonferroni rejects p ≤ 0.05/24 = 0.0021, which no test meets here. No sorted p-value falls under its BH cutoff, so BH rejects none.

4Check yourself

The tasks use the lab in section 1 and tick themselves when you meet them.

  • With no true effects and individual tests, simulate 1,000 experiments. More than half should contain at least one false rejection.Now: No simulation at these settings yet
  • Use Bonferroni with at least 48 tests and simulate. The rate of at least one false rejection should stay at or below 6%.Now: 24 tests, Individual tests, no simulation yet
  • With at least 6 true effects, find the rule that keeps the mean false discovery proportion at or below 5% and still detects more than half of the true effects.Now: 0 true effects, no simulation with true effects yet

5Adjust p-values in R

p.adjust turns raw p-values into adjusted ones, and each rule rejects where its adjusted value is at or below 0.05. The snippet simulates 24 tests with 6 true effects and counts rejections and false alarms under each rule. In this draw, individual tests reject 6 with 1 false alarm, while Bonferroni and BH each reject 3 with none. The last line gives the prediction’s answer: 0.642 without adjustment and 0.049 with Bonferroni.

Adjust p-values with p.adjustShow the code
# 24 tests: the first 6 have true effects (z shifted by 2.8)
set.seed(524)
m <- 24
effect <- rep(c(TRUE, FALSE), c(6, m - 6))
z <- rnorm(m) + ifelse(effect, 2.8, 0)
p <- 2 * pnorm(-abs(z))

# Rejections at alpha = 0.05 under each rule
rules <- list(
  none = p,
  bonferroni = p.adjust(p, method = "bonferroni"),
  BH = p.adjust(p, method = "BH"))
t(sapply(rules, function(q) c(
  rejected = sum(q <= 0.05),
  false_alarms = sum(q <= 0.05 & !effect))))

# Chance of at least one false rejection, 20 true nulls
round(c(none = 1 - 0.95^20,
        bonferroni = 1 - (1 - 0.05 / 20)^20), 3)

Uses base R only.

What the adjustments trade