Concept map / Reliability

Share

Reliability in classical test theory

Use two parallel test forms to connect true-score variance, error variance, and the standard error of measurement.

1Two forms of one test, 120 people

A test score is never exactly the score a person deserves. Classical test theory splits it into a true score, the average the person would get over many equivalent sittings, and an error that pushes this one sitting up or down.

Here 120 simulated people each take two parallel forms, with independent errors on each. A perfect test would put every dot on the dashed line. At a reliability of 0.80 the two forms correlate 0.78 in this group, close to 0.80 but not equal to it.

02020404060608080100100Score on form BScore on form Ar = 0.78Variance of scoresTrue100 (80%)Error25.0 (20%)
Each dot is one personPerson followed in section 2Dashed line: same score on both forms
Reliability of this test0.80
Correlation of the two forms in this group0.78
Margin of error for one score (SEM)5.0 points

With 20 items the reliability is 0.80, so 80% of the score variance is true-score variance and 20% is error. In this group of 120 the two forms correlate 0.78. One score is off by about 5.0 points, the standard error of measurement.

More controls

A new group moves the correlation a little. The reliability stays where the sliders put it.

2One person, many possible scores

The orange dot in the scatter is one person. Each sitting adds an error to their true score, as likely up as down, and the curve shows where a score could land. The shaded band runs one standard error of measurement (SEM) either side of the true score. Click any dot to follow someone else.

020406080100Taller = more likelyScore this person could gettrue 54.9 ± 5.0
True scoreBand of ± 1 SEMTheir form A and form B scoresRetests

Person 32 has a true score of 54.9. They scored 63.5 on form A and 47.7 on form B. On this test the SEM is 5.0 points, so about 68% of their possible scores fall between 49.9 and 59.9. A retest draws a fresh score.

More controls

3Twice the items: how far does 0.60 rise?

A 20-item test has reliability 0.60. You add 20 parallel items whose errors are unrelated to the first set's. What reliability does the Spearman-Brown formula give the 40-item test?

4Every number here comes from one split of the variance

Classical test theory starts from observed score X = true score T + error E, with E unrelated to T. Unrelated parts have variances that add, so the total splits cleanly into a true part and an error part. Point at a colored symbol to light up its part of the charts.

ρ  =  σT2σT2+σE2\rho \;=\; \frac{\htmlClass{term-tru}{\sigma_T^2}}{\htmlClass{term-tru}{\sigma_T^2} + \htmlClass{term-err}{\sigma_E^2}}
SEM  =  σX1−ρ\htmlClass{term-err}{\text{SEM}} \;=\; \sigma_X \sqrt{1 - \rho}
ρk  =  k ρ11+(k−1) ρ1\rho_{\htmlClass{term-len}{k}} \;=\; \frac{\htmlClass{term-len}{k}\,\rho_1}{1 + (\htmlClass{term-len}{k} - 1)\,\rho_1}

Reliability: 100 / (100 + 25.0) = 0.80. It is also the expected correlation between two parallel forms, which is why the scatter estimates it.

SEM: σX = √125.0 = 11.18, so SEM = 11.18 × √(1 − 0.80) = 5.00 points, the standard deviation of the error, √25.0.

Show reliability against test length

Spearman-Brown gives the reliability of a test k times as long as one with reliability ρ1. Here k = 20 / 20 = 1.00 and ρ1 = 0.80, so (1.00 × 0.80) / (1 + (1.00 − 1) × 0.80) = 0.800.

0.250.500.751.00020406080100120140160ReliabilityNumber of items20 items: 0.800

The curve climbs fast and then flattens. From 20 items, 20 more would add 0.089.

5Check yourself

The tasks use the sliders in section 1 and the retest buttons in section 2.

  • Keep ρ₁ at 0.80 and lengthen the test until its reliability reaches 0.90. How many items does that take?Now: 20-item reliability 0.80, 20 items, reliability 0.800
  • Make the error variance larger than the true variance, so reliability drops below 0.50, and look at the cloud.Now: True variance 100, error variance 25.0, reliability 0.80
  • Retest one person at least 50 times. About two thirds of the scores should land inside the band.Now: No retests yet

6Alpha and Spearman-Brown in psych

The lab has no item responses, so the code simulates 1000 people answering 20 items at the lab’s starting reliability. Coefficient alpha from psych::alpha comes out at 0.80. Spearman-Brown then gives 0.889 for 40 items and 0.900 for 45, the length the first task asks for. The last line prints the starting margin of error, 5 points.

Estimate alpha and apply Spearman-Brown with psychShow the code
library(psych)
# 1000 people answer 20 items whose correlations are all 1/6,
# so the 20-item test has reliability 0.80, as on the page
set.seed(22)
n <- 1000
true <- rnorm(n)
items <- sapply(1:20, function(i)
  sqrt(1 / 6) * true + sqrt(5 / 6) * rnorm(n))
rel <- alpha(items, discrete = FALSE)$total$raw_alpha
round(rel, 2)

# Spearman-Brown: reliability of a test k times as long
sb <- function(rho, k) k * rho / (1 + (k - 1) * rho)
# 40 and 45 items are 2 and 2.25 times the 20-item test
round(sb(0.80, c(40, 45) / 20), 3)

# SEM on the page's scale, where true scores have SD 10
sd_x <- sqrt(10^2 / 0.80)
round(sd_x * sqrt(1 - 0.80), 2)

Uses psych. Install once with install.packages(c("psych")).

What reliability measures