Concept map / Adaptive testing

Share

Computerized adaptive testing

Track the ability estimate and its posterior uncertainty as the test selects items in response to answers.

A fixed test gives everyone the same items, so a strong student spends time on easy items that say little about them. A computerized adaptive test (CAT) uses the answers so far to choose the next item. How many well-chosen items does it take to match the precision of a 20-item fixed form? The page builds on item information and ability estimation.

1Eleven questions, one answer at a time

You decide whether the student gets each question right. The questions differ only in difficulty b. The green dot is the estimate, the posterior mean under a standard normal prior, and the band is the 95% credible interval. After each answer the test picks the unused question with the most information across the current posterior.

Question 1 of 11

Question difficulty b0.0

Choose the student’s response.

Current ability estimate

0.00
95% credible interval−1.96 to 1.96
Before any responsesEstimate 0.00. 95% credible interval −1.96 to 1.96.

Ability scale θ

Before any responses, the estimate is the prior mean, 0.00, and the 95% credible interval runs from −1.96 to 1.96. The first question has difficulty 0.0, where a student at the prior mean has a 50% chance of answering correctly.

2A 40-item bank and a stopping rule

This bank has 40 two-parameter logistic (2PL) items with different discriminations, and their difficulties bunch near the middle, as in most real banks. After each answer the simulator updates the expected a posteriori (EAP) estimate and gives the unused item with the most information at that estimate. It stops when the posterior standard deviation, the margin of error, falls to 0.40, or after 30 items.

Set the person’s real ability and simulate. Answering by hand and the stopping rule are under More controls.

-4-202405101520Ability estimate after each itemItems given
Item answered right, at its difficultyItem answered wrongEstimate ± one posterior SDThe person's real ability
More controls

Before any answers the estimate is 0 with a posterior SD of 1. The first item is item 16 (difficulty −0.35), the most informative item at ability 0.

3Who gets the longest test?

Three people take this adaptive test. It stops once the margin of error is down to ± 0.40, or after 30 items if it never gets there. Who usually gets the longest test?

4How many items match a 20-item fixed form

The fixed form here is 20 items spread evenly across the bank. Its curve is an approximation from the information at the person's real ability plus the prior. The adaptive curve is the posterior SD of the run above.

00.250.50.751051015202530Posterior standard deviationItems givenstopping rule
Adaptive test (this run)Fixed 20-item form (approximate)

At ability 1.0 the fixed form ends with a posterior SD of about 0.40 after all 20 items. Run the adaptive test above to compare.

5Why the next item is the one it is

Each stem is one bank item, placed at its difficulty. Its height is the item's information at the current estimate, the green line. The test gives the tallest unused stem.

00.51-4-3-2-101234Information at the current estimateItem difficulty bnext: item 16
Faded stems are items already given.
next item  =  the unused item with the largest    ai 2  Pi(θ^) [1−Pi(θ^)]\text{next item} \;=\; \text{the unused item with the largest}\;\; \htmlClass{term-a}{a_i^{\,2}}\;\htmlClass{term-pq}{P_i(\hat\theta)\,\bigl[1-P_i(\hat\theta)\bigr]}
next item  =  the unused itemwith the largestai 2  Pi(θ^) [1−Pi(θ^)]\begin{gathered} \text{next item} \;=\; \text{the unused item} \\ \text{with the largest} \\ \htmlClass{term-a}{a_i^{\,2}}\;\htmlClass{term-pq}{P_i(\hat\theta)\,\bigl[1-P_i(\hat\theta)\bigr]} \end{gathered}

At the estimate 0.00, item 16 gives 2.1² × 0.68 × 0.32 = 0.97, the most of any unused item. The orange factor grows with the item's discrimination. The blue factor is largest when the chance of a right answer is 0.50.

6Check yourself

The tasks use the controls in section 2. Each ticks itself when you meet it.

  • Set the person's ability between −0.5 and 0.5, start over, and simulate a whole test. It should reach the stopping rule in 12 items or fewer.Now: Ability 1.00, 0 items, posterior SD 1.00 (target 0.40)
  • Start over, open More controls and answer by hand: mark the first five items right. Watch how far each right answer pushes the next item's difficulty.Now: No items yet
  • Set the ability to 2.5 or higher, start over, and simulate a whole test. Count how many items it takes compared with an average person.Now: Ability 1.00, 0 items

7Run the same test with mirt

The code calibrates the page’s 40-item bank with mirt on simulated data and runs the same loop as the simulator: pick the unused item with the most information at the current estimate, simulate an answer from a person at ability 1, and update the EAP estimate with fscores. It stops at a posterior standard deviation of 0.40. With this seed that takes 11 items, with an estimate of 1.12 and a standard deviation of 0.399. The first item is item 16, the same one the simulator picks.

Run an adaptive test on a 2PL bank with mirtShow the code
library(mirt)

# The page's 40-item 2PL bank, sorted from easiest to hardest
k <- 1:40
b <- round(pmin(3, pmax(-3, 0.775 * qlogis((k - 0.5) / 40))), 2)
a <- c(1.3, 1.4, 1.1, 1.3, 1.4, 2.0, 2.0, 1.2, 1.4, 1.9,
       1.0, 1.2, 1.6, 1.2, 0.8, 2.1, 1.0, 1.5, 1.9, 0.8,
       1.4, 1.0, 1.5, 1.4, 0.8, 1.7, 1.9, 2.0, 1.7, 1.6,
       1.0, 1.2, 1.3, 1.2, 1.7, 1.3, 1.3, 2.1, 1.6, 1.8)

# Calibrate the bank on 2000 simulated examinees
set.seed(1)
dat <- simdata(matrix(a), matrix(-a * b), 2000,
               itemtype = "dich")
bank <- mirt(dat, 1, itemtype = "2PL", verbose = FALSE)
est_pars <- coef(bank, IRTpars = TRUE, simplify = TRUE)$items
ah <- est_pars[, "a"]
bh <- est_pars[, "b"]

# Adaptive test for a person at ability 1, stopping at SD 0.40
resp <- rep(NA, 40)
f <- cbind(F1 = 0, SE_F1 = 1)
while (f[1, "SE_F1"] > 0.40 && sum(!is.na(resp)) < 30) {
  p <- plogis(ah * (f[1, "F1"] - bh))
  # Information a^2 P (1 - P) at the estimate; used items get -1
  i <- which.max(ifelse(is.na(resp), ah^2 * p * (1 - p), -1))
  resp[i] <- rbinom(1, 1, plogis(a[i] * (1 - b[i])))
  f <- fscores(bank, response.pattern = resp, method = "EAP")
  cat(sprintf("item %2d b %5.2f u %d EAP %5.2f SD %.3f\n",
              i, bh[i], resp[i], f[1, "F1"], f[1, "SE_F1"]))
}

Uses mirt. Install once with install.packages(c("mirt")).

What adaptive testing buys