Concept map / Adaptive testing
Share
Computerized adaptive testing
Track the ability estimate and its posterior uncertainty as the test selects items in response to answers.
A fixed test gives everyone the same items, so a strong student spends time on easy items that say little about them. A computerized adaptive test (CAT) uses the answers so far to choose the next item. How many well-chosen items does it take to match the precision of a 20-item fixed form? The page builds on item information and ability estimation.
1Eleven questions, one answer at a time
You decide whether the student gets each question right. The questions differ only in difficulty b. The green dot is the estimate, the posterior mean under a standard normal prior, and the band is the 95% credible interval. After each answer the test picks the unused question with the most information across the current posterior.
Question 1 of 11
Choose the student’s response.
Current ability estimate
0.00Ability scale θ
Before any responses, the estimate is the prior mean, 0.00, and the 95% credible interval runs from −1.96 to 1.96. The first question has difficulty 0.0, where a student at the prior mean has a 50% chance of answering correctly.
2A 40-item bank and a stopping rule
This bank has 40 two-parameter logistic (2PL) items with different discriminations, and their difficulties bunch near the middle, as in most real banks. After each answer the simulator updates the expected a posteriori (EAP) estimate and gives the unused item with the most information at that estimate. It stops when the posterior standard deviation, the margin of error, falls to 0.40, or after 30 items.
Set the person’s real ability and simulate. Answering by hand and the stopping rule are under More controls.
More controls
Before any answers the estimate is 0 with a posterior SD of 1. The first item is item 16 (difficulty −0.35), the most informative item at ability 0.
3Who gets the longest test?
Three people take this adaptive test. It stops once the margin of error is down to ± 0.40, or after 30 items if it never gets there. Who usually gets the longest test?
4How many items match a 20-item fixed form
The fixed form here is 20 items spread evenly across the bank. Its curve is an approximation from the information at the person's real ability plus the prior. The adaptive curve is the posterior SD of the run above.
At ability 1.0 the fixed form ends with a posterior SD of about 0.40 after all 20 items. Run the adaptive test above to compare.
5Why the next item is the one it is
Each stem is one bank item, placed at its difficulty. Its height is the item's information at the current estimate, the green line. The test gives the tallest unused stem.
At the estimate 0.00, item 16 gives 2.1² × 0.68 × 0.32 = 0.97, the most of any unused item. The orange factor grows with the item's discrimination. The blue factor is largest when the chance of a right answer is 0.50.
6Check yourself
The tasks use the controls in section 2. Each ticks itself when you meet it.
- Set the person's ability between −0.5 and 0.5, start over, and simulate a whole test. It should reach the stopping rule in 12 items or fewer.Now: Ability 1.00, 0 items, posterior SD 1.00 (target 0.40)
- Start over, open More controls and answer by hand: mark the first five items right. Watch how far each right answer pushes the next item's difficulty.Now: No items yet
- Set the ability to 2.5 or higher, start over, and simulate a whole test. Count how many items it takes compared with an average person.Now: Ability 1.00, 0 items
7Run the same test with mirt
The code calibrates the page’s 40-item bank with mirt on simulated data and runs the same loop as the simulator: pick the unused item with the most information at the current estimate, simulate an answer from a person at ability 1, and update the EAP estimate with fscores. It stops at a posterior standard deviation of 0.40. With this seed that takes 11 items, with an estimate of 1.12 and a standard deviation of 0.399. The first item is item 16, the same one the simulator picks.
Run an adaptive test on a 2PL bank with mirtShow the code
library(mirt)
# The page's 40-item 2PL bank, sorted from easiest to hardest
k <- 1:40
b <- round(pmin(3, pmax(-3, 0.775 * qlogis((k - 0.5) / 40))), 2)
a <- c(1.3, 1.4, 1.1, 1.3, 1.4, 2.0, 2.0, 1.2, 1.4, 1.9,
1.0, 1.2, 1.6, 1.2, 0.8, 2.1, 1.0, 1.5, 1.9, 0.8,
1.4, 1.0, 1.5, 1.4, 0.8, 1.7, 1.9, 2.0, 1.7, 1.6,
1.0, 1.2, 1.3, 1.2, 1.7, 1.3, 1.3, 2.1, 1.6, 1.8)
# Calibrate the bank on 2000 simulated examinees
set.seed(1)
dat <- simdata(matrix(a), matrix(-a * b), 2000,
itemtype = "dich")
bank <- mirt(dat, 1, itemtype = "2PL", verbose = FALSE)
est_pars <- coef(bank, IRTpars = TRUE, simplify = TRUE)$items
ah <- est_pars[, "a"]
bh <- est_pars[, "b"]
# Adaptive test for a person at ability 1, stopping at SD 0.40
resp <- rep(NA, 40)
f <- cbind(F1 = 0, SE_F1 = 1)
while (f[1, "SE_F1"] > 0.40 && sum(!is.na(resp)) < 30) {
p <- plogis(ah * (f[1, "F1"] - bh))
# Information a^2 P (1 - P) at the estimate; used items get -1
i <- which.max(ifelse(is.na(resp), ah^2 * p * (1 - p), -1))
resp[i] <- rbinom(1, 1, plogis(a[i] * (1 - b[i])))
f <- fscores(bank, response.pattern = resp, method = "EAP")
cat(sprintf("item %2d b %5.2f u %d EAP %5.2f SD %.3f\n",
i, bh[i], resp[i], f[1, "F1"], f[1, "SE_F1"]))
}Uses mirt. Install once with install.packages(c("mirt")).
What adaptive testing buys
- After each answer the test multiplies the current ability distribution by the chance of that answer at every ability. A right answer moves the estimate up and a wrong one moves it down.
- The next item is the unused one with the largest a² P(1 − P) at the current estimate, which favors discriminating items with a difficulty close to the estimate.
- With a stopping rule on the posterior standard deviation, people take different numbers of items but finish with about the same margin of error.
- Near the middle of this bank the adaptive test reaches 0.40 in about 9 items. The 20-item fixed form needs all 20 to reach 0.37 at ability 0, and it ends at 0.55 at ability 2.5.
- Where the bank is thin, as above 1.5 here, the test runs longer and can reach the item limit. Operational CATs also control item exposure and balance content, and they rely on measurement invariance so that one bank serves every group.
Recommended reading
- Technology-Based Assessment: Validity, Modeling, and Analysis IssuesRandy E. Bennett, Michelle LaMar, and John MazzeoEducational Measurement, 5th ed., chapter 9