Concept map / Generalizability theory
Share
Generalizability theory
Use variance components from a crossed persons × tasks × raters design to compare G and Φ as the numbers of tasks and raters change.
1More tasks or more raters?
A writing assessment gives each person a few tasks, and raters score every response. Classical test theory puts everything that is not true score into one error term. Generalizability theory splits that error by source, or facet: here, which tasks a person happened to get and which raters happened to score them. Each source has a variance component, its share of the score variance.
The design here is fully crossed: every person does the same tasks, and the same raters score every task for every person. Change how many tasks and raters a score averages over and watch two coefficients, G for ranking people and Φ for comparing them with a fixed cut score.
Error left after averaging over 2 tasks and 2 raters. Green parts count against both coefficients, gold parts only against Φ.
Each person completes 2 tasks, and the same 2 raters score every task, so each score averages 4 ratings. Relative error variance is 0.725, which gives G = 0.58 for ranking people. Absolute error adds task difficulty, rater severity and their interaction, 1.000 in all, which gives Φ = 0.50 for comparing scores with a fixed cut score.
More controls
2Eight ratings per person, three ways to spend them
With the starting components, the person × task variance (0.80) is more than twice the person × rater variance (0.35). You can afford 8 ratings per person. Which design gives the highest G?
3G for ranking, Φ for a cut score
In the crossed persons × tasks × raters design (p × t × r), a G study estimates seven variance components. A D study then divides each error component by the number of tasks, raters or both that a score averages over. Ranking people depends only on components that involve persons. A pass or fail decision also depends on how hard the tasks were and how severe the raters were. Point at a term to light up its bars and controls.
With your settings, σ²δ = 0.80/2 + 0.35/2 + 0.60/4 = 0.725, so G = 1 / (1 + 0.725) = 0.58. Adding 0.30/2 + 0.20/2 + 0.10/4 gives σ²Δ = 1.000 and Φ = 1 / (1 + 1.000) = 0.50. Person variance is fixed at 1, task variance at 0.30, task × rater at 0.10, and the residual at 0.60.
4Check yourself
The tasks use the controls in section 1 and tick themselves when you meet them.
- With the starting variance components, reach G of at least 0.75 using no more than 16 ratings per person.Now: 2 × 2 = 4 ratings, G = 0.58
- With the starting variance components, find the design with the fewest ratings per person that reaches Φ of at least 0.70.Now: 2 tasks × 2 raters = 4 ratings, Φ = 0.50
- Raise rater severity variance (under More controls) to 0.80 or more and still reach Φ of at least 0.70.Now: Rater severity 0.20, Φ = 0.50
5A G study and a D study in R
A G study estimates the variance components. The snippet simulates 150 persons, 6 tasks and 6 raters from the lab’s starting components and fits all six crossed random effects with lme4::lmer. The residual stands for the person × task × rater interaction plus error. It then runs the D study by hand for the lab’s starting design of 2 tasks and 2 raters. With only six tasks and six raters the task and rater components are rough (0.20 and 0.47, against 0.30 and 0.20), yet G and Φ come out at 0.58 and 0.49, close to the lab’s 0.58 and 0.50.
Crossed variance components with lme4, then G and ΦShow the code
library(lme4)
set.seed(5)
# G study: 150 persons x 6 tasks x 6 raters, fully crossed
d <- expand.grid(p = factor(1:150), t = factor(1:6),
r = factor(1:6))
v <- c(p = 1, t = 0.3, r = 0.2, pt = 0.8, pr = 0.35,
tr = 0.1, ptr = 0.6)
eff <- function(f, var) rnorm(nlevels(f), sd = sqrt(var))[f]
d$y <- eff(d$p, v["p"]) + eff(d$t, v["t"]) +
eff(d$r, v["r"]) + eff(d$p:d$t, v["pt"]) +
eff(d$p:d$r, v["pr"]) + eff(d$t:d$r, v["tr"]) +
rnorm(nrow(d), sd = sqrt(v["ptr"]))
fit <- lmer(y ~ 1 + (1 | p) + (1 | t) + (1 | r) + (1 | p:t) +
(1 | p:r) + (1 | t:r), data = d)
s2 <- as.data.frame(VarCorr(fit))
s2 <- setNames(s2$vcov, sub(":", "", s2$grp))
round(s2, 2)
# D study by hand: 2 tasks, the same 2 raters on each
nt <- 2; nr <- 2
rel <- s2["pt"] / nt + s2["pr"] / nr +
s2["Residual"] / (nt * nr)
abs <- rel + s2["t"] / nt + s2["r"] / nr + s2["tr"] / (nt * nr)
G <- s2[["p"]] / (s2[["p"]] + rel[[1]])
Phi <- s2[["p"]] / (s2[["p"]] + abs[[1]])
round(c(G = G, Phi = Phi), 2)Uses lme4. Install once with install.packages(c("lme4")).
Where the error comes from
- Generalizability theory gives each facet and interaction its own variance component, so you can see whether tasks, raters or their combination limit precision.
- Each component is divided by its own count. More tasks shrink the person × task error and more raters shrink the person × rater error, so extra raters cannot fix error that comes from too few tasks.
- Relative error includes only interactions with persons. Absolute error also includes task difficulty, rater severity and their interaction, so Φ is never larger than G. A severe rater who scores everyone leaves the ranking alone but moves scores relative to a cut score.
- These formulas belong to the crossed p × t × r design. If raters were nested within tasks, rater and task × rater variance could not be separated, and the error terms would change.
- Variance components come from mixed-effects models, and reliability as a ratio of variances comes back in rater studies and grouped designs.
Recommended reading
- Generalizability TheoryRobert L. BrennanMultifacet generalizability studies and decision study designs