Concept map / Reliability
Share
Reliability in classical test theory
Use two parallel test forms to connect true-score variance, error variance, and the standard error of measurement.
1Two forms of one test, 120 people
A test score is never exactly the score a person deserves. Classical test theory splits it into a true score, the average the person would get over many equivalent sittings, and an error that pushes this one sitting up or down.
Here 120 simulated people each take two parallel forms, with independent errors on each. A perfect test would put every dot on the dashed line. At a reliability of 0.80 the two forms correlate 0.78 in this group, close to 0.80 but not equal to it.
With 20 items the reliability is 0.80, so 80% of the score variance is true-score variance and 20% is error. In this group of 120 the two forms correlate 0.78. One score is off by about 5.0 points, the standard error of measurement.
More controls
A new group moves the correlation a little. The reliability stays where the sliders put it.
2One person, many possible scores
The orange dot in the scatter is one person. Each sitting adds an error to their true score, as likely up as down, and the curve shows where a score could land. The shaded band runs one standard error of measurement (SEM) either side of the true score. Click any dot to follow someone else.
Person 32 has a true score of 54.9. They scored 63.5 on form A and 47.7 on form B. On this test the SEM is 5.0 points, so about 68% of their possible scores fall between 49.9 and 59.9. A retest draws a fresh score.
More controls
3Twice the items: how far does 0.60 rise?
A 20-item test has reliability 0.60. You add 20 parallel items whose errors are unrelated to the first set's. What reliability does the Spearman-Brown formula give the 40-item test?
4Every number here comes from one split of the variance
Classical test theory starts from observed score X = true score T + error E, with E unrelated to T. Unrelated parts have variances that add, so the total splits cleanly into a true part and an error part. Point at a colored symbol to light up its part of the charts.
Reliability: 100 / (100 + 25.0) = 0.80. It is also the expected correlation between two parallel forms, which is why the scatter estimates it.
SEM: σX = √125.0 = 11.18, so SEM = 11.18 × √(1 − 0.80) = 5.00 points, the standard deviation of the error, √25.0.
Show reliability against test length
Spearman-Brown gives the reliability of a test k times as long as one with reliability ρ1. Here k = 20 / 20 = 1.00 and ρ1 = 0.80, so (1.00 × 0.80) / (1 + (1.00 − 1) × 0.80) = 0.800.
The curve climbs fast and then flattens. From 20 items, 20 more would add 0.089.
5Check yourself
The tasks use the sliders in section 1 and the retest buttons in section 2.
- Keep ρ₁ at 0.80 and lengthen the test until its reliability reaches 0.90. How many items does that take?Now: 20-item reliability 0.80, 20 items, reliability 0.800
- Make the error variance larger than the true variance, so reliability drops below 0.50, and look at the cloud.Now: True variance 100, error variance 25.0, reliability 0.80
- Retest one person at least 50 times. About two thirds of the scores should land inside the band.Now: No retests yet
6Alpha and Spearman-Brown in psych
The lab has no item responses, so the code simulates 1000 people answering 20 items at the lab’s starting reliability. Coefficient alpha from psych::alpha comes out at 0.80. Spearman-Brown then gives 0.889 for 40 items and 0.900 for 45, the length the first task asks for. The last line prints the starting margin of error, 5 points.
Estimate alpha and apply Spearman-Brown with psychShow the code
library(psych)
# 1000 people answer 20 items whose correlations are all 1/6,
# so the 20-item test has reliability 0.80, as on the page
set.seed(22)
n <- 1000
true <- rnorm(n)
items <- sapply(1:20, function(i)
sqrt(1 / 6) * true + sqrt(5 / 6) * rnorm(n))
rel <- alpha(items, discrete = FALSE)$total$raw_alpha
round(rel, 2)
# Spearman-Brown: reliability of a test k times as long
sb <- function(rho, k) k * rho / (1 + (k - 1) * rho)
# 40 and 45 items are 2 and 2.25 times the 20-item test
round(sb(0.80, c(40, 45) / 20), 3)
# SEM on the page's scale, where true scores have SD 10
sd_x <- sqrt(10^2 / 0.80)
round(sd_x * sqrt(1 - 0.80), 2)Uses psych. Install once with install.packages(c("psych")).
What reliability measures
- Reliability is true-score variance divided by observed-score variance. At 0.80, four fifths of the spread in scores comes from real differences between people.
- When two forms are parallel, their correlation equals the reliability. The correlation in one sample is an estimate of it and moves from group to group.
- The standard error of measurement is the observed standard deviation times √(1 − reliability). About 68% of a person’s retest scores land within one SEM of their true score.
- Doubling a test with reliability 0.60 raises it to 0.75, not 0.80. Getting from 0.80 to 0.90 takes 2.25 times as many items.
- The same error variance gives a lower reliability in a group whose true scores vary less. Item information goes further and lets precision change from one ability level to the next.
Recommended reading
- Reliability in Educational MeasurementWon-Chan Lee and Deborah J. HarrisEducational Measurement, 5th ed., chapter 5