Concept map / DCM
Share
Diagnostic classification models
Use simulated fraction responses to estimate mastery profiles and check which profiles the item set can distinguish.
1Which fraction skills does this student have?
A diagnostic classification model places a student in a profile of skills, called attributes, rather than at a point on a scale. Three fraction skills, each mastered or not, give eight profiles.
The Q-matrix lists the attributes each item requires. Under the DINA model, a student with all of them answers correctly with probability 1 − slip (s); a student missing even one succeeds only by a guess (g). Here one g and one s cover all seven items, where real tests estimate a pair per item.
Open an item below to change its answer or what it requires.
A = add fractions that share a denominatorB = rewrite a fraction with a new denominatorC = simplify to lowest terms
- Requires
More controls
π is the share of students who have each attribute, taken as independent. At 0.50 the eight profiles start equal.
The most probable profile is AB (110): can add and rewrite, cannot simplify. Its probability is 0.86. Probability of mastery: A (add) 0.93, B (rewrite) 0.93, C (simplify) 0.01.
Over every possible answer pattern with these settings, 56.0% of students would land in exactly the right profile. One attribute at a time, the call is right for A 81.8%, B 81.8% and C 81.8% of them.
2What dropping two items costs
To save time, a teacher drops items 2 and 6, the two that ask for rewriting without adding. In the five items left, every item that requires B (rewrite) also requires A (add).
| Item kept | A | B | C |
|---|---|---|---|
| 1. 1/5 + 2/5 | A | · | · |
| 3. Simplify 6/9 | · | · | C |
| 4. 1/2 + 1/3 | A | B | · |
| 5. 1/8 + 3/8, simplified | A | · | C |
| 7. 1/6 + 1/3, simplified | A | B | C |
Bo can rewrite a fraction with a new denominator but cannot add or simplify: profile B only (010). Nel has none of the three skills (000). Guess and slip are both 0.10, so answers are fairly trustworthy. How well can the five-item test tell Bo’s profile from Nel’s?
3From item chances to a posterior
DINA stands for “deterministic inputs, noisy and gate”. The deterministic part is η: 1 when profile α has every attribute the item’s Q-matrix row requires, 0 otherwise. Slip and guess are the noise. Point at a colored term and the matching slider or chart line lights up.
q is a Q-matrix entry, x is 1 for a right answer and 0 for a wrong one, and L is the likelihood: the chance of the whole answer pattern if the profile were α.
Profile AB (110) can do 3 of the 7 items. You marked 3 of those right and 0 wrong, and 0 right and 4 wrong among the rest. So L = 0.803 × 0.200 × 0.200 × 0.804 = 0.2097. Prior 0.1250 × L = 0.0262. Added over all eight profiles the product is 0.0303, so the posterior is 0.0262 / 0.0303 = 0.865.
Show the item-by-item table
| Item | η for AB | P(right) | Answer | Factor |
|---|---|---|---|---|
| 1. 1/5 + 2/5 needs A | 1 | 1 − s = 0.80 | right | 0.80 |
| 2. 2/3 = ?/12 needs B | 1 | 1 − s = 0.80 | right | 0.80 |
| 3. Simplify 6/9 needs C | 0 | g = 0.20 | wrong | 0.80 |
| 4. 1/2 + 1/3 needs A and B | 1 | 1 − s = 0.80 | right | 0.80 |
| 5. 1/8 + 3/8, simplified needs A and C | 0 | g = 0.20 | wrong | 0.80 |
| 6. 4/6 = ?/9 needs B and C | 0 | g = 0.20 | wrong | 0.80 |
| 7. 1/6 + 1/3, simplified needs A, B and C | 0 | g = 0.20 | wrong | 0.80 |
4Check yourself
Open an item in the first panel to flip its answer or change what it requires. Each task ticks itself off when you meet it.
- Give the answers you would expect from a student who can add and simplify but cannot rewrite, until profile AC (101) is the most probable at 0.80 or more.Now: Most probable: AB (110) at 0.86; AC is at 0.00
- Change the Q-matrix so the test cannot tell "B only" (010) from "none" (000), whatever the answers, while at least one item still requires B.Now: 1 item separates B only from none
- Make the test put at least 90% of students in exactly the right profile. Think about what makes an answer trustworthy. If you changed the Q-matrix, reset it first under More controls.Now: 56.0% of students land in exactly the right profile (guess 0.20, slip 0.20)
5The DINA posterior in R
The posterior needs only base R: one likelihood for each of the eight profiles, multiplied by the prior and normalized. With the lab’s starting answers it prints 0.865 for profile 110 (AB) and mastery probabilities of 0.926, 0.926 and 0.014. The lab rounds these to 0.86, 0.93, 0.93 and 0.01.
DINA posterior in base RShow the code
# Q-matrix: which skills each item needs, from
# A (add), B (rewrite) and C (simplify)
Q <- rbind(c(1, 0, 0), c(0, 1, 0), c(0, 0, 1), c(1, 1, 0),
c(1, 0, 1), c(0, 1, 1), c(1, 1, 1))
x <- c(1, 1, 0, 1, 0, 0, 0) # the page's starting answers
g <- 0.2 # guess
s <- 0.2 # slip
# The eight profiles; each skill is held with probability 0.5
prof <- expand.grid(A = 0:1, B = 0:1, C = 0:1)
prior <- rep(1 / 8, 8)
# DINA: 1 - s if the profile has every skill the item needs
lik <- apply(prof, 1, function(al) {
eta <- apply(Q, 1, function(q) all(al[q == 1] == 1))
p <- ifelse(eta, 1 - s, g)
prod(p^x * (1 - p)^(1 - x))
})
post <- prior * lik / sum(prior * lik)
names(post) <- apply(prof, 1, paste, collapse = "")
round(post, 3)
# Probability of mastery: add the profiles holding each skill
round(colSums(prof * post), 3)Uses base R only.
What the Q-matrix decides, in brief
- The Q-matrix is an assumption about what each item requires, and it settles which profiles the test can ever tell apart.
- Under DINA a student with every required attribute succeeds with probability 1 − slip. Everyone else succeeds with the guessing probability, whether they lack one attribute or all of them.
- The posterior over profiles comes from Bayes’ rule. Add up the profiles that contain an attribute and you get the probability of mastering it.
- Two profiles with the same chance on every item have the same likelihood for any answer pattern. More items of the same kinds will not separate them; an item with a different set of requirements can.
- A single profile label can hide real doubt about one attribute, so report the mastery probabilities beside it. Multidimensional item response theory treats the skills as continuous instead.
Recommended reading
- Diagnostic Measurement: Theory, Methods, and ApplicationsAndré A. Rupp, Jonathan Templin, and Robert A. HensonCognitive diagnosis and diagnostic classification models