Concept map / DCM

Share

Diagnostic classification models

Use simulated fraction responses to estimate mastery profiles and check which profiles the item set can distinguish.

Prerequisites IRT: item curves, Bayes

1Which fraction skills does this student have?

A diagnostic classification model places a student in a profile of skills, called attributes, rather than at a point on a scale. Three fraction skills, each mastered or not, give eight profiles.

The Q-matrix lists the attributes each item requires. Under the DINA model, a student with all of them answers correctly with probability 1 − slip (s); a student missing even one succeeds only by a guess (g). Here one g and one s cover all seven items, where real tests estimate a pair per item.

Open an item below to change its answer or what it requires.

A = add fractions that share a denominatorB = rewrite a fraction with a new denominatorC = simplify to lowest terms

  • Requires
00.250.50.751Probability of each profile, given the answers0.01none0000.200.05A1000.200.05B0100.200.00C0010.200.86AB1100.800.00AC1010.200.00BC0110.200.00ABC1110.80Attribute profile, and its chance of a right answer on item 4
PosteriorPriorItem 4 needs A and B: 0.80 for profiles that have it all, 0.20 for the rest
Mastery of A (add)0.93master
Mastery of B (rewrite)0.93master
Mastery of C (simplify)0.01non-master
More controls

π is the share of students who have each attribute, taken as independent. At 0.50 the eight profiles start equal.

The most probable profile is AB (110): can add and rewrite, cannot simplify. Its probability is 0.86. Probability of mastery: A (add) 0.93, B (rewrite) 0.93, C (simplify) 0.01.

Over every possible answer pattern with these settings, 56.0% of students would land in exactly the right profile. One attribute at a time, the call is right for A 81.8%, B 81.8% and C 81.8% of them.

2What dropping two items costs

To save time, a teacher drops items 2 and 6, the two that ask for rewriting without adding. In the five items left, every item that requires B (rewrite) also requires A (add).

Item keptABC
1. 1/5 + 2/5A··
3. Simplify 6/9··C
4. 1/2 + 1/3AB·
5. 1/8 + 3/8, simplifiedA·C
7. 1/6 + 1/3, simplifiedABC

Bo can rewrite a fraction with a new denominator but cannot add or simplify: profile B only (010). Nel has none of the three skills (000). Guess and slip are both 0.10, so answers are fairly trustworthy. How well can the five-item test tell Bo’s profile from Nel’s?

3From item chances to a posterior

DINA stands for “deterministic inputs, noisy and gate”. The deterministic part is η: 1 when profile α has every attribute the item’s Q-matrix row requires, 0 otherwise. Slip and guess are the noise. Point at a colored term and the matching slider or chart line lights up.

P(right on item j∣α)  =  (1−s)ηj  g 1−ηj,ηj=∏kαk qjkP(\text{right on item } j \mid \alpha) \;=\; \htmlClass{term-b}{(1-s)^{\eta_j}}\;\htmlClass{term-a}{g^{\,1-\eta_j}}, \qquad \eta_j = \prod_k \alpha_k^{\,q_{jk}}
P(right on item j∣α)  =  (1−s)ηj  g 1−ηjηj=∏kαk qjk\begin{gathered} P(\text{right on item } j \mid \alpha) \;=\; \htmlClass{term-b}{(1-s)^{\eta_j}}\;\htmlClass{term-a}{g^{\,1-\eta_j}} \\[6pt] \eta_j = \prod_k \alpha_k^{\,q_{jk}} \end{gathered}
P(α∣x)  =  P(α)  L(α)∑α′P(α′)  L(α′),L(α)=∏jPj xj (1−Pj)1−xjP(\alpha \mid x) \;=\; \frac{\htmlClass{term-prior}{P(\alpha)}\; L(\alpha)}{\sum_{\alpha'} \htmlClass{term-prior}{P(\alpha')}\; L(\alpha')}, \qquad L(\alpha) = \prod_j P_j^{\,x_j}\,(1-P_j)^{1-x_j}
P(α∣x)  =  P(α)  L(α)∑α′P(α′)  L(α′)L(α)=∏jPj xj (1−Pj)1−xj\begin{gathered} P(\alpha \mid x) \;=\; \frac{\htmlClass{term-prior}{P(\alpha)}\; L(\alpha)}{\sum_{\alpha'} \htmlClass{term-prior}{P(\alpha')}\; L(\alpha')} \\[6pt] L(\alpha) = \prod_j P_j^{\,x_j}\,(1-P_j)^{1-x_j} \end{gathered}

q is a Q-matrix entry, x is 1 for a right answer and 0 for a wrong one, and L is the likelihood: the chance of the whole answer pattern if the profile were α.

Profile AB (110) can do 3 of the 7 items. You marked 3 of those right and 0 wrong, and 0 right and 4 wrong among the rest. So L = 0.803 × 0.200 × 0.200 × 0.804 = 0.2097. Prior 0.1250 × L = 0.0262. Added over all eight profiles the product is 0.0303, so the posterior is 0.0262 / 0.0303 = 0.865.

Show the item-by-item table
Itemη for ABP(right)AnswerFactor
1. 1/5 + 2/5 needs A11 − s = 0.80right0.80
2. 2/3 = ?/12 needs B11 − s = 0.80right0.80
3. Simplify 6/9 needs C0g = 0.20wrong0.80
4. 1/2 + 1/3 needs A and B11 − s = 0.80right0.80
5. 1/8 + 3/8, simplified needs A and C0g = 0.20wrong0.80
6. 4/6 = ?/9 needs B and C0g = 0.20wrong0.80
7. 1/6 + 1/3, simplified needs A, B and C0g = 0.20wrong0.80

4Check yourself

Open an item in the first panel to flip its answer or change what it requires. Each task ticks itself off when you meet it.

  • Give the answers you would expect from a student who can add and simplify but cannot rewrite, until profile AC (101) is the most probable at 0.80 or more.Now: Most probable: AB (110) at 0.86; AC is at 0.00
  • Change the Q-matrix so the test cannot tell "B only" (010) from "none" (000), whatever the answers, while at least one item still requires B.Now: 1 item separates B only from none
  • Make the test put at least 90% of students in exactly the right profile. Think about what makes an answer trustworthy. If you changed the Q-matrix, reset it first under More controls.Now: 56.0% of students land in exactly the right profile (guess 0.20, slip 0.20)

5The DINA posterior in R

The posterior needs only base R: one likelihood for each of the eight profiles, multiplied by the prior and normalized. With the lab’s starting answers it prints 0.865 for profile 110 (AB) and mastery probabilities of 0.926, 0.926 and 0.014. The lab rounds these to 0.86, 0.93, 0.93 and 0.01.

DINA posterior in base RShow the code
# Q-matrix: which skills each item needs, from
# A (add), B (rewrite) and C (simplify)
Q <- rbind(c(1, 0, 0), c(0, 1, 0), c(0, 0, 1), c(1, 1, 0),
           c(1, 0, 1), c(0, 1, 1), c(1, 1, 1))
x <- c(1, 1, 0, 1, 0, 0, 0)  # the page's starting answers
g <- 0.2                     # guess
s <- 0.2                     # slip
# The eight profiles; each skill is held with probability 0.5
prof <- expand.grid(A = 0:1, B = 0:1, C = 0:1)
prior <- rep(1 / 8, 8)

# DINA: 1 - s if the profile has every skill the item needs
lik <- apply(prof, 1, function(al) {
  eta <- apply(Q, 1, function(q) all(al[q == 1] == 1))
  p <- ifelse(eta, 1 - s, g)
  prod(p^x * (1 - p)^(1 - x))
})
post <- prior * lik / sum(prior * lik)
names(post) <- apply(prof, 1, paste, collapse = "")
round(post, 3)
# Probability of mastery: add the profiles holding each skill
round(colSums(prof * post), 3)

Uses base R only.

What the Q-matrix decides, in brief