Item Response Theory (IRT)

Suppose you want to assess how much important security knowledge employees actually picked up after completing cybersecurity training. Even if everyone took the same test, the number of correct answers alone isn't enough to compare individual ability.

That's because some items almost anyone gets right, while others only people with solid security knowledge can answer. Even two employees with the same number of correct answers may differ in how well they understand the material, depending on which questions they got right.

Item response theory (IRT) analyzes these differences by considering the characteristics of the items and the respondents' latent ability together. Instead of simply adding up correct answers, it estimates each item's difficulty, discrimination and so on, and models the probability that a person with a given ability answers each item correctly.

In this article we'll cover the basic principles of IRT, the 1PL, 2PL and 3PL models, item characteristic curves and information functions, ability estimation, and model diagnostics and reporting.

On a test it might be knowledge or ability; on a psychological or attitude survey it might be anxiety or a particular disposition. In this article we'll treat cybersecurity knowledge as the latent ability.

This latent ability is usually written with the Greek letter θ (theta), and analyses often scale it to have mean 0 and standard deviation 1. θ = 0 doesn't mean no knowledge at all; it means an average level in the chosen reference group. θ = 1 means an ability level one standard deviation above the mean of that reference group.

The items cover spotting phishing emails, multi-factor authentication, file-sharing permissions, incident reporting procedures and so on, and every item is scored 1 for correct and 0 for incorrect. To evaluate the training, the company wants to check not only each employee's knowledge level but also each item's difficulty.

These results can also inform later revisions of items or the design of level-based training. But first you should check whether the actual content of the items measures a single ability.

A high proportion correct suggests an easy item, and a low one a hard item. But the proportion correct is also affected by the ability level of the group that took the test.

The difficulty parameter (b) in IRT shows where an item sits on the chosen latent ability scale. In the 2PL model, b is the ability level at which the probability of a correct answer is 50%. For example, an item with b = −1 gives a 50% chance of a correct answer even at a fairly low ability level, while an item with b = 1 requires higher ability.

That doesn't mean IRT difficulty is completely invariant across every sample. It requires conditions such as an appropriate model and items that function the same way across groups. And in the 3PL model, the guessing parameter means b can't simply be read as the 50%-correct point.

The number in the model name is the number of main parameters estimated for each item.

1PL is simple and easy to interpret, but it imposes the strong restriction that every item distinguishes ability in a similar way. 2PL lets discrimination differ from item to item, and 3PL captures the chance of a correct answer at low ability due to guessing, especially on multiple-choice tests.

Note, though, that c in 3PL is not a directly observed "guessing rate" of the respondents. As the number of parameters grows, estimation can require more data and stronger identification conditions.

Here a is the discrimination and b the difficulty. For example, suppose an item on spotting phishing emails has discrimination a = 1.5 and difficulty b = 0.5. The probability that an average employee (θ = 0) answers correctly is about 0.321, and for an employee one standard deviation above average (θ = 1) it is about 0.679.

These are probabilities calculated from example parameters, not the observed proportions correct of actual employees. Also, software can differ in how it scales discrimination, so when comparing parameters or results, check which model formula was used.

The higher the difficulty b, the further the curve shifts to the right; the higher the discrimination a, the more steeply the curve rises around the difficulty.

For example, on an easy item (b = −1), even employees with below-average ability can have a high probability of answering correctly. On a hard item (b = 1.5), by contrast, the probability only gets large once ability is high enough. On an item with low discrimination, the probability of a correct answer differs relatively little across ability levels, so it may not separate respondents well in any particular ability range.

When looking at ICCs, it's important not to aim only for a high proportion correct. A test that checks minimum essential knowledge after training needs easy items, and a test meant to distinguish experts needs hard items.

A good item is not necessarily a hard one; it's one that provides the information you need in the ability range that matches the purpose of measurement.

Q2 has relatively high discrimination at 1.80, so it can be useful for distinguishing employees whose ability is around its difficulty of 0.20. Q5, with difficulty −1.10, is closer to an item that checks basic knowledge, and Q8, with difficulty 1.20, can provide more useful information in the high ability range.

Q11 has low discrimination at 0.40, so it's worth checking whether its wording is ambiguous or whether it mixes several concepts. But rather than deleting it right away just because its discrimination is low, consider the training goals and the importance of the item as well.

In the 2PL model, the information of one item is:

By this formula, at the same ability position, an item with higher discrimination can provide more information. Also, a 2PL item generally provides the most information near its difficulty, where the probability of a correct answer is 0.5.

Taking the earlier item with a = 1.5 and b = 0.5 as an example, the probability of a correct answer at θ = 0.5 is 0.5. The information there is 1.5² × 0.5 × 0.5 = 0.5625.

Under conditions such as local independence, the information function for the whole test can be calculated by summing the item information functions. So by placing items of different difficulties appropriately, you can measure precisely over a wider ability range.

Common methods include maximum likelihood estimation (MLE), expected a posteriori (EAP) and maximum a posteriori (MAP). The estimates can differ depending on the method, especially for extreme response patterns.

For example, suppose employees A and B both got 8 of the 12 items right. A may have gotten several hard items right and missed some easy ones, while B got most of the easy items right and missed the hard ones. Taking the items' characteristics into account, IRT can give the two employees different ability estimates.

This table is an example to illustrate differences in response patterns; the values were not produced by an actual estimation.

An ability estimate of 0.65 means a level 0.65 standard deviations above the mean on the chosen scale. The standard error of the estimate shows the uncertainty in that score, and it can differ by ability level.

Under suitable conditions, the local standard error of an ability estimate is roughly related to it as follows.

For example, if the test information near θ = 0 is 9, the standard error there is about 0.33. If the test information at θ = 2 is 4, the standard error is about 0.50. In other words, the same test can provide different amounts of information for average employees and for very high-ability employees.

When designing a test, you don't need to aim for the same precision across all ability ranges. If the purpose is to judge a minimum standard of essential security knowledge, it's useful to increase precision around the passing score; if the purpose is to select advanced security specialists, it's important to increase information in the high ability range.

The usual checks are unidimensionality, local independence, item fit and how items function across groups.

Unidimensionality

In a basic single-factor IRT model, this is the assumption that response patterns are explained mainly by one latent trait. If, on a cybersecurity test, phishing knowledge and network configuration skills are strongly separate traits, you may need to consider multidimensional IRT.

Local independence

The assumption that, given latent ability, there is no additional dependence between item responses. For example, if the answer to an earlier item gives a hint for a later one, this assumption can be violated.

Item fit

Assesses how well an item's actual response pattern matches the pattern the model predicts. For items that fit poorly, check the wording, the answer key, the possibility of more than one correct answer and so on. Don't decide whether to delete an item based on a single fit statistic.

Differential item functioning (DIF)

Checks whether people with the same ability have different probabilities of answering a particular item correctly depending on the group they belong to. For example, you can look at cases where familiarity with terminology differs by department or job. Finding DIF doesn't necessarily mean the item is unfair, but it is an important clue for reviewing the content.

IRT, by contrast, uses a model that directly links latent ability to the probability of each item response.

IRT is also used for item banks and computerized adaptive testing (CAT). For example, you can design a test that chooses as the next item the one that provides the most information at the employee's current ability estimate, reaching the same measurement goal with fewer items.

Implementing this, however, requires enough item calibration data, stable parameter estimates and management of item exposure, among other things.

If you used 3PL, state how the lower asymptote was estimated; if you compared groups, state how DIF was examined.

An actual report should also present the necessary diagnostic evidence, such as standard errors of the estimates, item fit results, the ability estimation method and checks of unidimensionality and local independence.

Unlike simply adding up correct answers, it can estimate latent ability by taking into account which items were answered and how.

In particular, item characteristic curves are useful for understanding difficulty and discrimination, and item and test information functions for assessing measurement precision by ability level. But unless you check the model's assumptions, item fit and how items function across groups, the estimates are hard to trust.