Concept map / Score reporting

Share

Interpreting reported scores

Read a score report with a scale score, a standard error band and a performance level, and see how often measurement error alone puts a student in the wrong level.

1One report, many possible sittings

A score report turns one test sitting into a scale score, a band around it and a performance level set by cut scores. All of these rest on a score that contains measurement error. The standard error of measurement (SEM) is the typical size of that error in scale points. It comes from the test’s reliability, as in classical test theory, and an ability estimate carries one too.

The scale here runs from 300 to 700 with a mean of 500 and an SD of 50. Cuts at 450, 500 and 560 make four levels. Set a student’s true score and the test’s reliability, then give the test again and watch the report change.

Reading score reportShowing the true score until you give the test
Scale score510
95% band471 to 549
Performance levelProficient
Percentile rank58
Below BasicBasicProficientAdvanced300400450500560600700Chance of each reported score for this studentScale score
True scoreReported scores over many sittingsReported in the wrong level95% band around the reported scoreAll students
No sittings yet
Below Basic0.1%
Basic30.7%
Proficient68.5%
Advanced0.6%

The student’s true score is 510, which is Proficient. With a standard error of 20.0 points, a report shows that level 68.5% of the time and a different level 31.5% of the time. The current band from 471 to 549 crosses the cut at 500. Across all students at this reliability, 73% are reported in their true level.

More controls

2Predict the risk for a student 10 points above a cut

Proficient starts at 500. A student’s true score is 510, so they are Proficient. The test has reliability 0.84, which gives a standard error of 20 points. If this student took the test many times, how often would the report call them below Proficient?

3Reliability sets the SEM, the SEM sets the risk

The standard error of measurement turns reliability into scale points. The chance of landing on the other side of a cut score depends only on how many standard errors the true score sits from it. Φ is the standard normal cumulative probability.

SEM=SD1−ρXX′P(reported below c∣T)=Φ ⁣(c−TSEM)\htmlClass{term-sem}{\mathrm{SEM}} = \mathrm{SD}\sqrt{1 - \rho_{XX'}} \qquad P(\text{reported below } \htmlClass{term-cut}{c} \mid T) = \Phi\!\left(\frac{\htmlClass{term-cut}{c} - T}{\htmlClass{term-sem}{\mathrm{SEM}}}\right)
SEM=SD1−ρXX′P(reported below c∣T)=Φ ⁣(c−TSEM)\begin{gathered} \htmlClass{term-sem}{\mathrm{SEM}} = \mathrm{SD}\sqrt{1 - \rho_{XX'}} \\[4pt] P(\text{reported below } \htmlClass{term-cut}{c} \mid T) = \Phi\!\left(\frac{\htmlClass{term-cut}{c} - T}{\htmlClass{term-sem}{\mathrm{SEM}}}\right) \end{gathered}

Now: SEM = 50 × √(1 − 0.84) = 20.00. The nearest cut is 500, which is 0.50 standard errors from the true score, so a single sitting lands on the other side of it 30.9% of the time. The 95% band is the reported score ± 1.96 × SEM = ± 39.2 points.

4Check yourself

The tasks use the sliders in section 1 and tick themselves when you meet them.

  • Keep the reliability at 0.90 or higher and find a true score that is reported in the wrong level more than 40% of the time.Now: reliability 0.84, wrong level 31.5% of the time
  • With reliability no higher than 0.90, place a Proficient student so that the report shows Proficient at least 90% of the time.Now: reliability 0.84, true level Proficient, reported correctly 68.5% of the time
  • Raise the reliability until at least 85% of all students are reported in their true level.Now: reliability 0.84, 73% of all students in their true level

5Bands and misclassification in base R

Base R is enough. The snippet computes the standard error, the chance that a student with a true score of 510 is reported below Proficient (0.309), and the 95% band (470.8 to 549.2). It then simulates a million students and prints the share reported in their true level (0.731; the lab shows 73%) and the full table of true against reported levels.

Standard error, bands and misclassificationShow the code
# Scale with mean 500 and SD 50; reliability 0.84
sd_x <- 50
rel <- 0.84
sem <- sd_x * sqrt(1 - rel)
sem

# Student with true score 510; Proficient starts at 500
cuts <- c(450, 500, 560)
true <- 510
# Chance a report puts them below Proficient
pnorm(500, mean = true, sd = sem)
# 95% band around a reported score of 510
510 + c(-1.96, 1.96) * sem

# Whole population: true level versus reported level
set.seed(1)
t <- rnorm(1e6, 500, sd_x * sqrt(rel))
x <- t + rnorm(1e6, 0, sem)
levels <- c("Below Basic", "Basic", "Proficient", "Advanced")
lv <- function(s) factor(findInterval(s, cuts), 0:3, levels)
mean(lv(t) == lv(x))
round(prop.table(table(true = lv(t), reported = lv(x)), 1), 2)

Uses base R only.

What a reported score promises