Concept map / Hypothesis testing and power
Share
Hypothesis testing and power
Compare a two-sided z test’s p-value, Type I error rate, and power under different settings.
1An effect of 0.38 in 30 people: chance or not?
A study of 30 people finds an average improvement of 0.38 standard deviations. If the true effect were zero, how often would chance alone give a difference that large? A hypothesis test answers by placing the result in its sampling distribution under the null hypothesis of no effect. The width of that distribution is the standard error from the central limit theorem, the same one behind a confidence interval.
Rejecting a true null is a Type I error, and the threshold α sets how often it happens. Power is the probability of rejecting when a given true effect exists. The population standard deviation is taken as known and equal to 1, so the statistic is a z score.
With n = 30, an observed effect of 0.38 lies 2.08 standard errors from zero (z = 2.08). The two-sided p-value is 0.037, at or below α = 0.05, so the test rejects the null. If the true effect were 0.30, this design would reject in 37.6% of studies (its power) and miss it in 62.4% (a Type II error).
More controls
2After p = 0.03, will a replication reject?
A study with a two-sided z test finds p = 0.03. Suppose the true effect is exactly the effect it observed. A second study with the same sample size uses α = 0.05. How likely is the second study to reject the null?
3z, its p-value and the power of the test
The test divides the observed effect by its standard error σ/√n. A true effect δ shifts the whole distribution of z by δ√n, and power is the share of that shifted curve beyond the cutoffs ±z*. Point at a symbol to light it up on the chart.
Here z = 0.38 / (1/√30) = 2.08, so p = 2Φ(−2.08) = 0.037.
With δ = 0.30 and n = 30, δ√n = 1.64 and z* = 1.96 for α = 0.05. Power = Φ(−0.32) + Φ(−3.60) = 37.6%.
4Check yourself
The tasks use the lab in section 1 and tick themselves when you meet them.
- Set the observed effect to 0.20, keep α at 5%, and raise the sample size until the test rejects the null.Now: Observed effect 0.38, n = 30, α 5.0%, p = 0.037
- Set the true effect to 0.30, keep α at 5%, and find a sample size that gives at least 80% power.Now: True effect 0.30, n = 30, α 5.0%, power 37.6%
- Set the true effect to zero, open More controls, set α to 1% and run 100 studies. With no effect, power equals the Type I error rate.Now: True effect 0.30, α 5.0%, power 37.6%, no studies run yet
5The same test and its power in R
The first lines repeat the lab’s starting calculation: z, its p-value and the z test’s power. power.t.test gives the power of the t test, which estimates the standard deviation and so has a little less power than the z test here. The simulated study at the end has a true effect of 0.30 and still gives p = 0.091. With power near 36%, most studies of this size miss an effect that is really there.
A z test, its power, and power.t.testShow the code
# The page's default: observed effect 0.38 SD with n = 30
n <- 30
z <- 0.38 / (1 / sqrt(n))
p <- 2 * pnorm(-abs(z))
round(c(z = z, p = p), 4)
# Power of the z test at a true effect of 0.30 SD
crit <- qnorm(1 - 0.05 / 2)
shift <- 0.3 * sqrt(n)
round(pnorm(shift - crit) + pnorm(-shift - crit), 4)
# The t test estimates the SD, so its power is a bit lower
power.t.test(n = n, delta = 0.3, sd = 1,
type = "one.sample")$power
# Sample size for 80% power with the t test
power.t.test(power = 0.8, delta = 0.3, sd = 1,
type = "one.sample")$n
# One simulated study at the true effect
set.seed(81)
x <- rnorm(n, mean = 0.3, sd = 1)
t.test(x, mu = 0)$p.valueUses base R only.
What a p-value does and does not say
- A p-value is the probability, if the true effect were zero, of a statistic at least as far from zero as the one observed. It is not the probability that the null is true.
- The standard error shrinks with √n, so the same effect turns significant with more data. An observed 0.20 gives p = 0.051 with n = 95 and p = 0.046 with n = 100.
- With no true effect the test rejects in a share α of studies, so power at zero effect equals α.
- At a true effect of 0.30 and α = 0.05, 30 people give 37.6% power. The t test needs 90 people for 80%. If a study finds p = 0.03 and its observed effect is the true one, a same-size replication rejects only 58.3% of the time.
- In item response theory, test information sets the standard error of an ability estimate and plays the part of n here.
Recommended reading
- Introductory Statistics 2eBarbara Illowsky and Susan DeanChapter 9, hypothesis tests, p-values, and decision errors
- Introductory Statistics with Randomization and SimulationDavid M. Diez, Christopher D. Barr, and Mine Çetinkaya-RundelChapter 4 resources, sample size and power in the one-sample case