Concept map / Standard setting

Share

Standard setting

Turn an Angoff panel's judgments into a cut score and inspect how many candidates each cut misclassifies.

1Three judges, eight items, one pass mark

A licensing board needs a pass mark for an 8-item test. Standard setting turns a written description of a competent candidate into that number. In the Angoff method, judges picture a candidate just at the standard and estimate the chance that this candidate answers each item correctly. The item averages, added up, give the recommended cut. Whether the cut then supports the pass or fail decision is a validity question.

Under the judges’ table, 100 simulated candidates sit at their raw scores: 50 who meet the standard and 50 below it. Candidates pass at the cut or above. Move the cut, or make the whole panel stricter, and see who lands on the wrong side.

Chance that a borderline candidate answers correctly
ItemJudge AJudge BJudge CAverage
185%80%90%85%
270%75%70%72%
360%65%55%60%
455%45%50%50%
545%50%40%45%
635%35%40%37%
725%30%25%27%
820%20%25%22%
Panel recommendation (sum of averages)3.97
Below standard but pass25 of 50
Meet standard but fail1 of 50
Misclassified in total26
012345678Candidates, one dot eachRaw score out of 8Meet the standardBelow the standardpassfail012345678Candidates, one dot eachRaw score out of 8Meet the standardBelow the standardpassfail
Meets the standardBelow the standardMisclassified at this cutPanel recommendation 3.97

The panel recommends 3.97 out of 8. With the cut at 4, 74 of 100 candidates pass. 25 of the 50 below the standard pass and 1 of the 50 who meet it fail, so 26 are misclassified, and 49 of the 74 who pass (66%) meet the standard.

2Predict what the same cut does to a weaker cohort

Keep the cut at the panel’s 4. In the cohort above, half the candidates meet the standard, and 49 of the 74 who pass (66%) meet it. Next year only 10 in 100 candidates meet the standard, and each group’s scores are spread as before. Of the candidates who pass, what share will meet the standard?

3The cut is the borderline candidate’s expected score

Judge j gives a probability pij for item i. Averaged over judges, it estimates how often a borderline candidate gets item i right. The sum of those averages is that candidate’s expected raw score, and it becomes the recommended cut. Point at p̄ to light up the averages in the table.

cut  =  ∑i=18pˉi,pˉi  =  13∑j=13pij\text{cut} \;=\; \sum_{i=1}^{8} \htmlClass{term-prior}{\bar p_i}, \qquad \htmlClass{term-prior}{\bar p_i} \;=\; \frac{1}{3}\sum_{j=1}^{3} p_{ij}

Now: 0.85 + 0.72 + 0.60 + 0.50 + 0.45 + 0.37 + 0.27 + 0.22 = 3.97.

Shifts, rounding and the share of passers who meet the standard

Shifting every judgment by s moves each item average by s unless a judgment hits 0% or 100%, so the cut moves by up to 8 × s raw-score points.

The recommendation lies on a continuous scale, but the test reports whole scores, so the board still decides where to round. At a cut of 4 the share of passing candidates who meet the standard is 49 / 74 = 0.66.

4Check yourself

The tasks use the sliders and the button in section 1 and tick themselves when you meet them.

  • Set the cut so that no more than 5 of the 50 candidates below the standard pass.Now: Cut 4: 25 below-standard candidates pass, 1 who meet it fail
  • Find the cut with the fewest misclassified candidates in total.Now: Cut 4: 26 misclassified
  • Shift every judgment until the panel’s recommendation rounds to 5, then adopt it with the “Use the nearest whole score” button.Now: Shift 0 points, recommendation 3.97, adopted cut 4

5The Angoff cut and its consequences in base R

The code computes the Angoff cut in base R, simulates 50 candidates below and 50 meeting the standard from the lab’s score model, and tabulates who passes. It then counts misclassifications at every cut and computes the expected share of passers who meet the standard in an even cohort and a weak one. The recommendation is 3.97, adopted as 4, and the shares are 0.69 and 0.20, as in the prediction. R draws different candidates from the lab, so its counts differ a little: at the cut of 4, 20 below-standard candidates pass and 2 who meet the standard fail, and a cut of 5 again gives the fewest errors (16).

Angoff cut and classification consequences in base RShow the code
# Angoff judgments: 8 items (rows) by 3 judges (columns)
angoff <- matrix(c(0.85, 0.80, 0.90,  0.70, 0.75, 0.70,
                   0.60, 0.65, 0.55,  0.55, 0.45, 0.50,
                   0.45, 0.50, 0.40,  0.35, 0.35, 0.40,
                   0.25, 0.30, 0.25,  0.20, 0.20, 0.25),
                 ncol = 3, byrow = TRUE)
# The recommended cut is the sum of the item averages
cut_raw <- sum(rowMeans(angoff))
cut <- round(cut_raw)
c(recommended = cut_raw, adopted = cut)

# 50 candidates below the standard and 50 who meet it
set.seed(98)
draw <- function(mu)
  pmin(8, pmax(0, round(mu + 1.25 * rnorm(50))))
score <- c(draw(3.25), draw(5.6))
meets <- rep(c(FALSE, TRUE), each = 50)
table(meets, pass = score >= cut)

# Misclassified candidates at every possible cut
errors <- sapply(0:8, function(k)
  sum(!meets & score >= k) + sum(meets & score < k))
setNames(errors, 0:8)

# Expected share of passers who meet the standard
p_meet <- 1 - pnorm((cut - 0.5 - 5.6) / 1.25)
p_below <- 1 - pnorm((cut - 0.5 - 3.25) / 1.25)
base <- c(half = 0.5, one_in_ten = 0.1)
round(base * p_meet / (base * p_meet + (1 - base) * p_below), 2)

Uses base R only.

What a cut score rests on