Concept map / Nested and crossed effects
Share
Nested and crossed effects
See how clustering shrinks the effective sample size, how nested and crossed rater designs differ, and why pooling across schools can reverse a slope.
1How many students do 120 students count as?
Students sampled from schools are not independent. Two students from the same school share teachers, a neighborhood and often a rater. The intraclass correlation (ICC) measures how alike they are, as the share of score variance that lies between schools. The higher it is, the less each extra student from the same school tells you, and the fewer independent students your sample is worth. The multilevel models page introduces the ICC through shrinkage.
Here 120 students are split evenly across schools. Change the number of schools and the ICC, and watch the effective sample size.
120 students sit in 12 schools of 10. With an ICC of 0.20 the design effect is 2.80, so for estimating the overall mean the 120 students are worth about 43 independent students. The standard error is 1.67 times what a simple random sample of 120 would give.
2Three schools, one line through all of them
Three schools, four students each. Within every school, each extra hour of study goes with 2 more points. But the schools whose students study more have lower averages: each school mean is 9 points lower than the previous one, with 3 more hours of study. If you fit one regression line to all 12 students and ignore the schools, what slope do you get?
3Bigger schools, smaller effective sample
The ICC ρ is the between-school variance τ² as a share of the total τ² + σ². Two students from the same school are correlated by ρ, so each extra student in a school adds less than a student from a new school would. The design effect is how much clustering inflates the variance of the overall mean, compared with a simple random sample of the same size.
Here m = 120 / 12 = 10 and ρ = 0.20, so DEFF = 1 + (10 − 1) × 0.20 = 2.80 and neff = 120 / 2.80 = 42.9. The formula assumes equal school sizes and one ICC shared by all schools.
4Raters nested in schools or crossed with them
Raters are nested in schools when each rater scores essays from one school only, and crossed with schools when the same raters score essays from every school. The sliders use the ICC of 0.20 and the 12 schools from section 1.
Each school has its own 2 raters, used nowhere else. The first 6 of 12 schools are shown.
Each school mean carries the average severity of its own 2 raters, adding 0.10 / 2 = 0.050 to the variance of school means. That is 20.0% of the apparent school variance. A nested model can estimate it but cannot remove it from any one school’s mean.
5Check yourself
The tasks use the controls in sections 1 and 4 and tick themselves when you meet them.
- With 12 schools, raise the ICC until the 120 students carry no more information than 40 independent students.Now: 12 schools, ICC 0.20, effective sample size 42.9
- Keep the ICC at 0.20 and get an effective sample size of at least 80 by changing only the number of schools.Now: ICC 0.20, 12 schools of 10, effective sample size 42.9
- In section 4, with nested raters, an ICC of 0.20 and rater severity variance 0.10, find the fewest raters per school that keeps rater severity below 10% of the apparent school variance.Now: 2 per school, rater share 20.0% (ICC 0.20, severity 0.10)
6The ICC and both rater designs in lme4
The snippet simulates the lab’s starting design, 12 schools of 10 students with an ICC of 0.20, and fits a random-intercept model with lme4::lmer. The estimated ICC is 0.21, so the design effect is 2.89. It then adds rater severity (variance 10, against 20 for schools and 80 for students), first with two raters nested in each school, (1 | school/rater), then with five raters crossed with schools, (1 | school) + (1 | rater_x). With 12 schools and a handful of raters, these variance estimates are rough.
ICC, design effect, and nested versus crossed raters with lme4Show the code
library(lme4)
set.seed(3)
# 12 schools of 10 students, ICC 0.20 (variances 20 and 80)
d <- data.frame(school = factor(rep(1:12, each = 10)))
u <- rnorm(12, sd = sqrt(20))
d$score <- 50 + u[d$school] + rnorm(120, sd = sqrt(80))
fit <- lmer(score ~ 1 + (1 | school), data = d)
vc <- as.data.frame(VarCorr(fit))$vcov
icc <- vc[1] / sum(vc)
round(c(icc = icc, deff = 1 + (10 - 1) * icc), 2)
# Nested: 2 raters per school, each used in one school only
d$rater <- factor(paste(d$school, rep(1:2, each = 5)))
sev <- rnorm(24, sd = sqrt(10))
d$nested <- d$score + sev[d$rater]
nest <- lmer(nested ~ 1 + (1 | school/rater), data = d)
print(VarCorr(nest), comp = "Variance")
# Crossed: 5 raters, each scoring 2 students in every school
d$rater_x <- factor(rep(1:5, times = 24))
d$crossed <- d$score + rnorm(5, sd = sqrt(10))[d$rater_x]
cross <- lmer(crossed ~ 1 + (1 | school) + (1 | rater_x),
data = d)
print(VarCorr(cross), comp = "Variance")Uses lme4. Install once with install.packages(c("lme4")).
What clustering costs
- The ICC is the correlation between two students from the same school, and also the share of variance that lies between schools.
- With clusters of size m, the variance of a mean is multiplied by the design effect, 1 + (m − 1)ρ. Even a small ICC matters when clusters are large, so for a fixed number of students, more schools beat bigger schools.
- When each school has its own raters, their average severity becomes part of that school’s mean. With one rater per school, rater and school effects cannot be separated at all.
- When the same raters score every school equally often, severity shifts every school alike, and a model with crossed random effects estimates the rater and school variances separately.
- A slope fitted across schools can have the opposite sign from the slope within each school, so decide which question you are asking before you fit one line. Mixed-effects models fit these structures directly, and generalizability theory uses crossed and nested facets to plan rating designs.
Recommended reading
- Data Analysis Using Regression and Multilevel/Hierarchical ModelsAndrew Gelman and Jennifer HillChapter 13, varying slopes and non-nested models