Propensity Score Matching (PSM)

Suppose the government runs a job training program for job seekers. The employment rate of people who took part in the program was 72%, and that of people who did not was 55%. Can we conclude that the job training program raised the employment rate by 17 percentage points?

That is hard to say. People who took part may have been more motivated to find work, younger, or had more previous work experience than those who did not. Such differences can affect not only whether someone joins the program but also how likely they are to find a job.

So if you simply compare the employment rates of participants and nonparticipants, differences in individual characteristics that existed from the start can end up counted as part of the program effect. A classic method used to reduce this problem is propensity score matching (PSM).

PSM compares people by pairing up those who were about equally likely to take part in the program. This lets you analyze how much outcomes differ between participants and nonparticipants with similar observable characteristics.

In this article we cover the basic idea of PSM, estimating propensity scores, matching methods, diagnosing balance between groups, calculating the treatment effect, and interpreting the results.

Here, treatment does not necessarily mean medical treatment. It broadly means any intervention whose effect the researcher wants to evaluate, such as a government policy, participation in an education program, or use of a company support program. For example, participation in a job training program can be set as the treatment variable.

• Treatment group: job training program participants
• Comparison group: job training program nonparticipants

The problem is that the two groups were not randomly assigned. If individuals chose whether to participate, participants and nonparticipants may differ in age, education, previous experience, length of unemployment, and so on. PSM uses these observable characteristics to calculate each individual’s likelihood of participating in the program.

This likelihood of participating is called the propensity score.

Next, participants and nonparticipants with similar propensity scores are matched and their employment rates compared. Creating similar groups with PSM, however, does not make the study fully equivalent to a randomized experiment. Differences in factors not included in the data can still remain.

The formula is as follows.

Here T is whether the person participated in the program, and X is the individual’s characteristics measured before participation. For example, suppose there are two job seekers.

The two people differ in whether they actually participated, but the propensity scores calculated from their pre-participation characteristics are similar. So PSM can link these two people as a comparison pair.

Conversely, a nonparticipant with a propensity score of 0.20, who differs greatly in age, education, experience, and so on, may not be a suitable comparison for a participant with a propensity score of 0.71. Propensity scores are often estimated with logistic regression, but other statistical models or machine learning methods can also be used.

A particularly important step in this process is checking covariate balance after matching. Similar propensity scores alone cannot guarantee that every individual characteristic has become sufficiently similar. So when evaluating the matching results, check the balance of characteristics between groups before looking at the difference in employment rates.

Of these, 400 took part in a job training program and 600 did not. The researchers recorded whether each person found a job within 6 months after the program ended.

Study data

What matters here is that the individual characteristics used for matching must be measured before program participation. For example, if you include in the matching variables a certificate earned after job training or job skills improved through the program, you can end up controlling away part of the program’s effect.

You also need to define, for nonparticipants, the same reference date that allows comparison with participants. The research question is as follows.

Employment rate before matching

The participants’ employment rate was 17 percentage points higher than that of nonparticipants. But the participants may have had characteristics that favored employment from the start. For example, suppose the average characteristics of the two groups before matching were as follows.

When participants and nonparticipants differ like this, it is hard to interpret the overall 17-percentage-point gap in employment rates as the effect of job training as is. So we need to compare people with similar characteristics.

The main matching methods are as follows.

For example, suppose participant A has a propensity score of 0.72. If there are nonparticipants with propensity scores of 0.71, 0.45, and 0.29, nearest neighbor matching considers the person with 0.71 first. But it may not always be appropriate to match someone simply because they are the closest.

If a participant’s propensity score is 0.95 but the closest nonparticipant’s is 0.50, the two people may not be similar enough. To reduce this problem, you can use a caliper, which limits the maximum distance allowed for a match.

Choose the matching method in light of the research goal set before the analysis and the characteristics of the data.

If the treatment and comparison groups still differ greatly in age, experience, and so on after matching, the program effect may be hard to trust. A standard measure for assessing covariate balance is the standardized mean difference (SMD).

The SMD expresses a difference between groups in standardized form, so it can be compared across variables measured in different units. An absolute value below 0.1 is commonly used as a reference for good balance. The 0.1 threshold, however, does not guarantee sufficient balance in every study.

Covariate balance before and after matching

Before matching, the SMD for length of unemployment was 0.61, a large difference between groups, but after matching it fell to 0.08. The differences in age, education, and previous experience also shrank. This means matching improved the balance of the observed characteristics.

However, an improvement in covariate balance alone does not mean unobserved confounders have been removed as well.

This is called the common support or overlap condition. For example, if most job training participants have propensity scores of 0.80–0.95 while most nonparticipants fall between 0.10 and 0.30, it is hard to find suitable comparisons.

Conversely, if both groups’ propensity scores are spread sufficiently over a similar range, matching becomes more feasible.

When common support is lacking, the following problems can arise.

• Some participants cannot be matched to suitable comparisons
• People with different characteristics are forced into pairs
• Uncertainty in the treatment effect estimate increases
• The estimate is hard to generalize to all participants

So in a real PSM analysis, it is good to check the propensity score distributions before and after matching with a graph. And if observations outside the common support were excluded, report how many were excluded.

This matters because it is not simple data cleaning but a choice that can change the very population for which the treatment effect is estimated.

For example, suppose 1:1 nearest neighbor matching linked 350 participants with 350 nonparticipants. In this example matching was done without replacement, and each participant was linked to one comparison unit.

Employment rate after matching

After matching, the employment rate was 72% for participants and 60% for nonparticipants. So the difference in employment rates in the matched sample is as follows.

This means that among people matched to have similar observed characteristics, job training participants had an employment rate 12 percentage points higher. The simple difference before matching was 17 percentage points, but after matching it shrank to 12 percentage points. This change suggests that differences in the two groups’ observed characteristics may have been related to part of the employment gap before matching.

But the 5-percentage-point reduction should not be interpreted as an exact decomposition into the confounding effects of specific variables, because the analysis sample itself also changed during matching.

The two main ones are the ATT and the ATE.

ATT(Average Treatment Effect on the Treated)

The average effect of the treatment among those who actually received it. For example, it estimates how the program affected, on average, the people who took part in job training.

ATE(Average Treatment Effect)

The average effect if the treatment were applied to the entire target population. For example, it means the average effect that would occur if all the young adults in the study took part in the job training program.

The 1:1 matching example above linked nonparticipants to participants, so it was an analysis designed to estimate the ATT. But if 50 of the 400 participants could not be matched and were excluded, the result cannot be guaranteed to equal the ATT for all 400 original participants.

In that case, the actual estimand may be limited to the group of participants matched within the common support. So report the observations excluded after matching along with the characteristics of the final analysis sample.

For example, suppose the difference in employment rates in the matched sample was 12 percentage points, and applying an appropriate inference method for matching gave the following results.

Not calculated from an actual sample.

Because the confidence interval does not include 0, this is consistent with a statistically significant difference in a conventional two-sided test. Still, you need to be careful when calculating the uncertainty of matched data.

So choose a variance estimation or inference method suited to the matching method and estimator you used. And even when a statistically significant result appears, the possibility of unobserved confounders remains.

It cannot remove unobserved confounders.

If there are characteristics that affect the outcome but were not measured, such as motivation to find work or personal networks, bias can remain even after matching.

Use pretreatment variables.

Including in the matching variables that changed after program participation can remove part of the treatment effect or create new bias. So choose the variables for propensity score estimation with their causal relationships and measurement timing in mind.

Always check balance after matching.

High predictive accuracy of the propensity score model does not mean good matching quality. In real studies, check covariate balance using SMDs, propensity score distributions, and so on.

Not every participant can be matched.

Participants outside the common support may not find suitable comparisons. Excluding them can change the population to which the results can be generalized.

PSM is not always the best method.

Other methods can also be used, such as inverse probability of treatment weighting (IPTW) with propensity scores, covariate-adjusted regression, and doubly robust estimation. Which method is more appropriate depends on the research goal, the characteristics of the data, and the distribution of confounders.

And where a randomized experiment is possible, random assignment can be better for causal identification than matching on observational data.

Using the job training program example above, you could write it as follows.

In real studies, it is good to also report the specification of the propensity score model, whether a caliper was applied, whether matching allowed replacement, the results of the common support check, and the method used to estimate standard errors appropriately. If possible, you can also add a sensitivity analysis for unobserved confounding to assess the robustness of the results.

Participants and nonparticipants in a job training program can differ in age, education, previous experience, length of unemployment, and so on. PSM uses these characteristics to estimate the probability of program participation and matches people with similar propensity scores. You can then calculate the difference in employment rates between the matched groups to analyze the relationship between program participation and the outcome.

But you need to check whether covariate balance improved sufficiently after matching, and when comparable units are scarce, the analysis population may be restricted. And because it does not remove unobserved confounders, applying PSM does not automatically prove a causal relationship.