Concept map / Generalizability theory

Share

Generalizability theory

Use variance components from a crossed persons × tasks × raters design to compare G and Φ as the numbers of tasks and raters change.

1More tasks or more raters?

A writing assessment gives each person a few tasks, and raters score every response. Classical test theory puts everything that is not true score into one error term. Generalizability theory splits that error by source, or facet: here, which tasks a person happened to get and which raters happened to score them. Each source has a variance component, its share of the score variance.

The design here is fully crossed: every person does the same tasks, and the same raters score every task for every person. Change how many tasks and raters a score averages over and watch two coefficients, G for ranking people and Φ for comparing them with a fixed cut score.

0.000.250.500.751.0012345678Coefficient (0 to 1)Tasks per person (each scored by all 2 raters)G 0.58Φ 0.50
G (Eρ²), relative decisionsΦ, absolute decisions
Person × task0.400
Person × rater0.175
Person × task × rater, residual0.150
Task difficulty0.150
Rater severity0.100
Task × rater0.025

Error left after averaging over 2 tasks and 2 raters. Green parts count against both coefficients, gold parts only against Φ.

Each person completes 2 tasks, and the same 2 raters score every task, so each score averages 4 ratings. Relative error variance is 0.725, which gives G = 0.58 for ranking people. Absolute error adds task difficulty, rater severity and their interaction, 1.000 in all, which gives Φ = 0.50 for comparing scores with a fixed cut score.

More controls

2Eight ratings per person, three ways to spend them

With the starting components, the person × task variance (0.80) is more than twice the person × rater variance (0.35). You can afford 8 ratings per person. Which design gives the highest G?

Design (tasks × raters)G coefficient0.000.250.500.758 tasks × 1 rater4 tasks × 2 raters2 tasks × 4 raters

3G for ranking, Φ for a cut score

In the crossed persons × tasks × raters design (p × t × r), a G study estimates seven variance components. A D study then divides each error component by the number of tasks, raters or both that a score averages over. Ranking people depends only on components that involve persons. A pass or fail decision also depends on how hard the tasks were and how severe the raters were. Point at a term to light up its bars and controls.

Eρ2=σp2σp2+σpt2nt+σpr2nr+σptr,e2ntnr⏟σδ2,Φ=σp2σp2+σδ2+σt2nt+σr2nr+σtr2ntnrE\rho^2 = \frac{\sigma^2_p}{\sigma^2_p + \underbrace{\htmlClass{term-a}{\tfrac{\sigma^2_{pt}}{n_t}} + \htmlClass{term-b}{\tfrac{\sigma^2_{pr}}{n_r}} + \htmlClass{term-a}{\tfrac{\sigma^2_{ptr,e}}{n_t n_r}}}_{\sigma^2_\delta}}, \qquad \Phi = \frac{\sigma^2_p}{\sigma^2_p + \sigma^2_\delta + \htmlClass{term-c}{\tfrac{\sigma^2_t}{n_t} + \tfrac{\sigma^2_r}{n_r} + \tfrac{\sigma^2_{tr}}{n_t n_r}}}
Eρ2=σp2σp2+σpt2nt+σpr2nr+σptr,e2ntnr⏟σδ2,Φ=σp2σp2+σδ2+σt2nt+σr2nr+σtr2ntnr\begin{gathered} E\rho^2 = \frac{\sigma^2_p}{\sigma^2_p + \underbrace{\htmlClass{term-a}{\tfrac{\sigma^2_{pt}}{n_t}} + \htmlClass{term-b}{\tfrac{\sigma^2_{pr}}{n_r}} + \htmlClass{term-a}{\tfrac{\sigma^2_{ptr,e}}{n_t n_r}}}_{\sigma^2_\delta}}, \\[8pt] \Phi = \frac{\sigma^2_p}{\sigma^2_p + \sigma^2_\delta + \htmlClass{term-c}{\tfrac{\sigma^2_t}{n_t} + \tfrac{\sigma^2_r}{n_r} + \tfrac{\sigma^2_{tr}}{n_t n_r}}} \end{gathered}

With your settings, σ²δ = 0.80/2 + 0.35/2 + 0.60/4 = 0.725, so G = 1 / (1 + 0.725) = 0.58. Adding 0.30/2 + 0.20/2 + 0.10/4 gives σ²Δ = 1.000 and Φ = 1 / (1 + 1.000) = 0.50. Person variance is fixed at 1, task variance at 0.30, task × rater at 0.10, and the residual at 0.60.

4Check yourself

The tasks use the controls in section 1 and tick themselves when you meet them.

  • With the starting variance components, reach G of at least 0.75 using no more than 16 ratings per person.Now: 2 × 2 = 4 ratings, G = 0.58
  • With the starting variance components, find the design with the fewest ratings per person that reaches Φ of at least 0.70.Now: 2 tasks × 2 raters = 4 ratings, Φ = 0.50
  • Raise rater severity variance (under More controls) to 0.80 or more and still reach Φ of at least 0.70.Now: Rater severity 0.20, Φ = 0.50

5A G study and a D study in R

A G study estimates the variance components. The snippet simulates 150 persons, 6 tasks and 6 raters from the lab’s starting components and fits all six crossed random effects with lme4::lmer. The residual stands for the person × task × rater interaction plus error. It then runs the D study by hand for the lab’s starting design of 2 tasks and 2 raters. With only six tasks and six raters the task and rater components are rough (0.20 and 0.47, against 0.30 and 0.20), yet G and Φ come out at 0.58 and 0.49, close to the lab’s 0.58 and 0.50.

Crossed variance components with lme4, then G and ΦShow the code
library(lme4)
set.seed(5)
# G study: 150 persons x 6 tasks x 6 raters, fully crossed
d <- expand.grid(p = factor(1:150), t = factor(1:6),
                 r = factor(1:6))
v <- c(p = 1, t = 0.3, r = 0.2, pt = 0.8, pr = 0.35,
       tr = 0.1, ptr = 0.6)
eff <- function(f, var) rnorm(nlevels(f), sd = sqrt(var))[f]
d$y <- eff(d$p, v["p"]) + eff(d$t, v["t"]) +
  eff(d$r, v["r"]) + eff(d$p:d$t, v["pt"]) +
  eff(d$p:d$r, v["pr"]) + eff(d$t:d$r, v["tr"]) +
  rnorm(nrow(d), sd = sqrt(v["ptr"]))
fit <- lmer(y ~ 1 + (1 | p) + (1 | t) + (1 | r) + (1 | p:t) +
              (1 | p:r) + (1 | t:r), data = d)
s2 <- as.data.frame(VarCorr(fit))
s2 <- setNames(s2$vcov, sub(":", "", s2$grp))
round(s2, 2)
# D study by hand: 2 tasks, the same 2 raters on each
nt <- 2; nr <- 2
rel <- s2["pt"] / nt + s2["pr"] / nr +
  s2["Residual"] / (nt * nr)
abs <- rel + s2["t"] / nt + s2["r"] / nr + s2["tr"] / (nt * nr)
G <- s2[["p"]] / (s2[["p"]] + rel[[1]])
Phi <- s2[["p"]] / (s2[["p"]] + abs[[1]])
round(c(G = G, Phi = Phi), 2)

Uses lme4. Install once with install.packages(c("lme4")).

Where the error comes from