Concept map / Validity and test design
Share
Validity and test design
Build a placement test from a blueprint and examine content, correlation, consistency and consequence evidence for using it to assign reading support.
1Should a quick speed task count toward reading support?
A school district wants a short test to decide which students get extra reading support. A colleague suggests adding a one-minute reading-speed task, because it is quick and very consistent. Would that make the scores a better basis for the decision? Validity is the degree to which evidence and theory support a proposed interpretation and use of scores, so it belongs to a use, not to a test.
Each of 200 simulated students has a comprehension skill, a vocabulary skill that correlates 0.7 with it, and a reading speed that correlates 0.3 with it. The 50 lowest placement scores are flagged. The 50 lowest on an independent end-of-year comprehension assessment are the students who really needed support. Jordan and Sam, the two large rings, have the same comprehension and vocabulary, but Sam reads slowly.
This form has 5 tasks: 3 comprehension, 2 vocabulary and 0 speed. Its scores correlate 0.69 with the end-of-year assessment. Of the 50 students who needed support, it flags 31 and misses 19. Jordan scores 1.80 and Sam 1.80; neither is flagged.
2Add the speed task: who trades places?
Start from the five-task form and add the one-minute speed task. The correlation with the end-of-year assessment hardly moves, from 0.69 to 0.67. The test still flags its 50 lowest scorers. How many of those 50 students change?
3Five kinds of evidence about your form
The Standards for Educational and Psychological Testing name five sources of validity evidence. Each checks a different step from task scores to the support decision. Choose one to read what it says about the form you built.
This form gives 60% of its tasks to comprehension, 40% to vocabulary and 0% to speed. The blueprint for a comprehension placement calls for 60%, 40% and 0%. Experts would also check that each task matches its blueprint category.
Two sources come with a number. The correlation between the placement score X and the end-of-year score Y is evidence from relations to other variables:
For this form, cov(X, Y) = 4.33, sX = 0.428 and sY = 14.71, so r = 4.33 / (0.428 × 14.71) = 0.69.
Show the alpha formula
Coefficient alpha, from classical test theory, is evidence about internal structure. It compares the k task variances si² with the variance of the total score T:
With k = 5, the task variances sum to 1.722 and the total-score variance is 4.590, so α = 5/4 × (1 − 1.722 / 4.590) = 0.78. Alpha says how consistently the tasks agree with each other, not whether they measure comprehension.
4Check yourself
The tasks use the checkboxes in section 1.
- Add the speed task and watch Sam, whose comprehension and vocabulary match Jordan’s.Now: Speed task out. Jordan is not flagged; Sam is not flagged.
- Find a form whose scores correlate at least 0.70 with the end-of-year comprehension assessment.Now: r = 0.69 with 5 tasks
- Find a form with coefficient alpha of at least 0.60 whose correlation with the end-of-year assessment is below 0.45.Now: alpha 0.78, r = 0.69
5The same checks in base R
The code simulates a class of 200 from the same model. It prints the blueprint coverage of the five-task and six-task forms, their correlations with the end-of-year assessment and their coefficient alpha. Then it counts how many flagged students change and how many who need support are missed. R draws its own class, so its numbers differ a little from the page’s: the correlations are 0.72 and 0.70, alpha is 0.81 and 0.76, 4 flagged students change, and both forms miss 19 of the 50.
Blueprint coverage, criterion correlation and flagging in base RShow the code
# 200 students: comprehension, vocabulary and reading speed
set.seed(7)
n <- 200
comp <- rnorm(n)
vocab <- 0.7 * comp + sqrt(1 - 0.7^2) * rnorm(n)
speed <- 0.3 * comp + sqrt(1 - 0.3^2) * rnorm(n)
task <- function(skill, l)
pmin(4, pmax(0, 2 + 0.6 * (l * skill + sqrt(1 - l^2) * rnorm(n))))
scores <- cbind(sapply(1:3, function(j) task(comp, 0.75)),
sapply(1:2, function(j) task(vocab, 0.75)),
task(speed, 0.9))
area <- factor(c(rep("Comprehension", 3), rep("Vocabulary", 2),
"Speed"), levels = c("Comprehension",
"Vocabulary", "Speed"))
year_end <- 100 + 15 * (0.85 * comp + sqrt(1 - 0.85^2) * rnorm(n))
# Blueprint coverage of the five-task and six-task forms
rbind(five = prop.table(table(area[1:5])),
six = prop.table(table(area)))
# Validity coefficient and coefficient alpha for each form
alpha <- function(m) ncol(m) / (ncol(m) - 1) *
(1 - sum(apply(m, 2, var)) / var(rowSums(m)))
x5 <- rowMeans(scores[, 1:5])
x6 <- rowMeans(scores)
round(rbind(r = c(cor(x5, year_end), cor(x6, year_end)),
alpha = c(alpha(scores[, 1:5]), alpha(scores))), 2)
# Flag the 50 lowest scorers; compare with the 50 lowest at year end
flag5 <- rank(x5, ties.method = "first") <= 50
flag6 <- rank(x6, ties.method = "first") <= 50
need <- rank(year_end, ties.method = "first") <= 50
c(changed = sum(flag5 & !flag6),
missed_five = sum(need & !flag5), missed_six = sum(need & !flag6))Uses base R only.
What the speed task shows about validity
- The same scores can be well supported for one decision and poorly supported for another. Ask whether the evidence supports this interpretation for this purpose.
- Content evidence asks whether the tasks represent the skills the interpretation names. A speed task on a comprehension placement test is outside the blueprint, however quick it is.
- Coefficient alpha says how well the tasks agree with each other. The two vocabulary tasks agree well with each other yet correlate much less with comprehension than the comprehension tasks do.
- A correlation of about 0.7 with an independent comprehension measure supports the interpretation, and the test still misses many students who need support. Adding the speed task barely moves that correlation but swaps students in and out of support by how fast they read.
- Measurement invariance asks whether scores mean the same thing across groups, and standard setting asks whether a cut score supports the decision it drives.
Recommended reading
- Validity and ValidationSuzanne Lane and Scott F. MarionEducational Measurement, 5th ed., chapter 4
- Designing and Developing Educational AssessmentsKristen Huff, Paul Nichols, and M. Christina SchneiderEducational Measurement, 5th ed., chapter 7