Concept map / Nested and crossed effects

Share

Nested and crossed effects

See how clustering shrinks the effective sample size, how nested and crossed rater designs differ, and why pooling across schools can reverse a slope.

1How many students do 120 students count as?

Students sampled from schools are not independent. Two students from the same school share teachers, a neighborhood and often a rater. The intraclass correlation (ICC) measures how alike they are, as the share of score variance that lies between schools. The higher it is, the less each extra student from the same school tells you, and the fewer independent students your sample is worth. The multilevel models page introduces the ICC through shrinkage.

Here 120 students are split evenly across schools. Change the number of schools and the ICC, and watch the effective sample size.

205080Test scoreSchool (12 schools of 10 students)
StudentSchool mean
Schools

120 students sit in 12 schools of 10. With an ICC of 0.20 the design effect is 2.80, so for estimating the overall mean the 120 students are worth about 43 independent students. The standard error is 1.67 times what a simple random sample of 120 would give.

2Three schools, one line through all of them

Three schools, four students each. Within every school, each extra hour of study goes with 2 more points. But the schools whose students study more have lower averages: each school mean is 9 points lower than the previous one, with 3 more hours of study. If you fit one regression line to all 12 students and ignore the schools, what slope do you get?

20304050600246810Test scoreHours of studySchool 1School 2School 3

3Bigger schools, smaller effective sample

The ICC ρ is the between-school variance τ² as a share of the total τ² + σ². Two students from the same school are correlated by ρ, so each extra student in a school adds less than a student from a new school would. The design effect is how much clustering inflates the variance of the overall mean, compared with a simple random sample of the same size.

DEFF=1+(m−1) ρ,neff=NDEFF,ρ=τ2τ2+σ2\text{DEFF} = 1 + (\htmlClass{term-a}{m} - 1)\,\htmlClass{term-theta}{\rho},\qquad n_{\text{eff}} = \frac{N}{\text{DEFF}},\qquad \htmlClass{term-theta}{\rho} = \frac{\tau^2}{\tau^2 + \sigma^2}
DEFF=1+(m−1) ρ,neff=NDEFF,ρ=τ2τ2+σ2\begin{gathered} \text{DEFF} = 1 + (\htmlClass{term-a}{m} - 1)\,\htmlClass{term-theta}{\rho}, \\[6pt] n_{\text{eff}} = \frac{N}{\text{DEFF}},\qquad \htmlClass{term-theta}{\rho} = \frac{\tau^2}{\tau^2 + \sigma^2} \end{gathered}

Here m = 120 / 12 = 10 and ρ = 0.20, so DEFF = 1 + (10 − 1) × 0.20 = 2.80 and neff = 120 / 2.80 = 42.9. The formula assumes equal school sizes and one ICC shared by all schools.

4Raters nested in schools or crossed with them

Raters are nested in schools when each rater scores essays from one school only, and crossed with schools when the same raters score essays from every school. The sliders use the ICC of 0.20 and the 12 schools from section 1.

School 1R1R2
School 2R3R4
School 3R5R6
School 4R7R8
School 5R9R10
School 6R11R12

Each school has its own 2 raters, used nowhere else. The first 6 of 12 schools are shown.

School differences, τ²0.200
Rater severity in a school mean0.050

Each school mean carries the average severity of its own 2 raters, adding 0.10 / 2 = 0.050 to the variance of school means. That is 20.0% of the apparent school variance. A nested model can estimate it but cannot remove it from any one school’s mean.

5Check yourself

The tasks use the controls in sections 1 and 4 and tick themselves when you meet them.

  • With 12 schools, raise the ICC until the 120 students carry no more information than 40 independent students.Now: 12 schools, ICC 0.20, effective sample size 42.9
  • Keep the ICC at 0.20 and get an effective sample size of at least 80 by changing only the number of schools.Now: ICC 0.20, 12 schools of 10, effective sample size 42.9
  • In section 4, with nested raters, an ICC of 0.20 and rater severity variance 0.10, find the fewest raters per school that keeps rater severity below 10% of the apparent school variance.Now: 2 per school, rater share 20.0% (ICC 0.20, severity 0.10)

6The ICC and both rater designs in lme4

The snippet simulates the lab’s starting design, 12 schools of 10 students with an ICC of 0.20, and fits a random-intercept model with lme4::lmer. The estimated ICC is 0.21, so the design effect is 2.89. It then adds rater severity (variance 10, against 20 for schools and 80 for students), first with two raters nested in each school, (1 | school/rater), then with five raters crossed with schools, (1 | school) + (1 | rater_x). With 12 schools and a handful of raters, these variance estimates are rough.

ICC, design effect, and nested versus crossed raters with lme4Show the code
library(lme4)
set.seed(3)
# 12 schools of 10 students, ICC 0.20 (variances 20 and 80)
d <- data.frame(school = factor(rep(1:12, each = 10)))
u <- rnorm(12, sd = sqrt(20))
d$score <- 50 + u[d$school] + rnorm(120, sd = sqrt(80))
fit <- lmer(score ~ 1 + (1 | school), data = d)
vc <- as.data.frame(VarCorr(fit))$vcov
icc <- vc[1] / sum(vc)
round(c(icc = icc, deff = 1 + (10 - 1) * icc), 2)

# Nested: 2 raters per school, each used in one school only
d$rater <- factor(paste(d$school, rep(1:2, each = 5)))
sev <- rnorm(24, sd = sqrt(10))
d$nested <- d$score + sev[d$rater]
nest <- lmer(nested ~ 1 + (1 | school/rater), data = d)
print(VarCorr(nest), comp = "Variance")

# Crossed: 5 raters, each scoring 2 students in every school
d$rater_x <- factor(rep(1:5, times = 24))
d$crossed <- d$score + rnorm(5, sd = sqrt(10))[d$rater_x]
cross <- lmer(crossed ~ 1 + (1 | school) + (1 | rater_x),
              data = d)
print(VarCorr(cross), comp = "Variance")

Uses lme4. Install once with install.packages(c("lme4")).

What clustering costs