Concept map / Hypothesis testing and power

Share

Hypothesis testing and power

Compare a two-sided z test’s p-value, Type I error rate, and power under different settings.

1An effect of 0.38 in 30 people: chance or not?

A study of 30 people finds an average improvement of 0.38 standard deviations. If the true effect were zero, how often would chance alone give a difference that large? A hypothesis test answers by placing the result in its sampling distribution under the null hypothesis of no effect. The width of that distribution is the standard error from the central limit theorem, the same one behind a confidence interval.

Rejecting a true null is a Type I error, and the threshold α sets how often it happens. Power is the probability of rejecting when a given true effect exists. The population standard deviation is taken as known and equal to 1, so the statistic is a z score.

−4−2024DensityStandardized test statistic z
Null and its rejection tailsTrue effect and its missesObserved z and its p-value

With n = 30, an observed effect of 0.38 lies 2.08 standard errors from zero (z = 2.08). The two-sided p-value is 0.037, at or below α = 0.05, so the test rejects the null. If the true effect were 0.30, this design would reject in 37.6% of studies (its power) and miss it in 62.4% (a Type II error).

More controls

2After p = 0.03, will a replication reject?

A study with a two-sided z test finds p = 0.03. Suppose the true effect is exactly the effect it observed. A second study with the same sample size uses α = 0.05. How likely is the second study to reject the null?

3z, its p-value and the power of the test

The test divides the observed effect by its standard error σ/√n. A true effect δ shifts the whole distribution of z by δ√n, and power is the share of that shifted curve beyond the cutoffs ±z*. Point at a symbol to light it up on the chart.

z  =  dˉσ/n,p  =  2 Φ(−∣z∣)z \;=\; \frac{\htmlClass{term-obs}{\bar d}}{\sigma/\sqrt{n}}, \qquad p \;=\; \htmlClass{term-p}{2\,\Phi(-|z|)}

Here z = 0.38 / (1/√30) = 2.08, so p = 2Φ(−2.08) = 0.037.

power=Φ(δn−z∗)+Φ(−δn−z∗)\begin{aligned} \text{power} &= \Phi\big(\htmlClass{term-alt}{\delta\sqrt{n}} - \htmlClass{term-crit}{z^*}\big) \\ &\quad + \Phi\big(-\htmlClass{term-alt}{\delta\sqrt{n}} - \htmlClass{term-crit}{z^*}\big) \end{aligned}

With δ = 0.30 and n = 30, δ√n = 1.64 and z* = 1.96 for α = 0.05. Power = Φ(−0.32) + Φ(−3.60) = 37.6%.

4Check yourself

The tasks use the lab in section 1 and tick themselves when you meet them.

  • Set the observed effect to 0.20, keep α at 5%, and raise the sample size until the test rejects the null.Now: Observed effect 0.38, n = 30, α 5.0%, p = 0.037
  • Set the true effect to 0.30, keep α at 5%, and find a sample size that gives at least 80% power.Now: True effect 0.30, n = 30, α 5.0%, power 37.6%
  • Set the true effect to zero, open More controls, set α to 1% and run 100 studies. With no effect, power equals the Type I error rate.Now: True effect 0.30, α 5.0%, power 37.6%, no studies run yet

5The same test and its power in R

The first lines repeat the lab’s starting calculation: z, its p-value and the z test’s power. power.t.test gives the power of the t test, which estimates the standard deviation and so has a little less power than the z test here. The simulated study at the end has a true effect of 0.30 and still gives p = 0.091. With power near 36%, most studies of this size miss an effect that is really there.

A z test, its power, and power.t.testShow the code
# The page's default: observed effect 0.38 SD with n = 30
n <- 30
z <- 0.38 / (1 / sqrt(n))
p <- 2 * pnorm(-abs(z))
round(c(z = z, p = p), 4)

# Power of the z test at a true effect of 0.30 SD
crit <- qnorm(1 - 0.05 / 2)
shift <- 0.3 * sqrt(n)
round(pnorm(shift - crit) + pnorm(-shift - crit), 4)

# The t test estimates the SD, so its power is a bit lower
power.t.test(n = n, delta = 0.3, sd = 1,
             type = "one.sample")$power
# Sample size for 80% power with the t test
power.t.test(power = 0.8, delta = 0.3, sd = 1,
             type = "one.sample")$n

# One simulated study at the true effect
set.seed(81)
x <- rnorm(n, mean = 0.3, sd = 1)
t.test(x, mu = 0)$p.value

Uses base R only.

What a p-value does and does not say