Causal inference guide
To tell whether a change was caused by a policy or treatment, what matters most is how you chose your comparison group and your time points. Use this guide to match a causal inference method to your research design.
Did a scholarship program raise students’ grades? Did a new policy change a region’s employment rate? Grades or employment shifting after a policy takes effect isn’t enough to call it the policy’s doing. Something else may have changed over the same period.
Causal inference measures an effect against what would have happened without the policy or treatment. The trouble is that you can never observe the same unit both treated and untreated at the same time. So the core job is to find a credible comparison, using what you know about the data and about how the program was set up.
This guide leaves the calculations and output tables to the individual chapters. Instead, it focuses on which methods to consider given the data and research design you have.
Check four things before you choose a method
Before you think about techniques, look at how the data came to be.
Who got the treatment?
How treatment was assigned determines whether your comparison group holds up.
Do you have before-and-after data?
If you can see before-and-after change and a comparison group, DID and related designs are on the table.
Is there a comparison group?
Ask in what sense the untreated units are really comparable.
What kind of bias worries you?
Methods differ in whether they can also handle factors you didn’t observe.
You can’t pick a causal method just from the shape of your data. What matters most is how treatment was assigned and how convincing your identifying assumptions are.
A map for choosing a causal method
Which method to look at first, based on the features of your research design.
Use the map as a starting point, not a verdict. Having before-and-after data doesn’t automatically make DID valid, for example. You also need a convincing case that, without the treatment, the two groups would have followed similar trends.
Comparing change before and after a policy
Methods that use both change over time and a comparison group.
When one group got the policy and another didn’t
If one region adopted a policy and another didn’t, and you have outcomes for both before and after, consider difference-in-differences (DID). DID takes the change in the treated group and subtracts the change in the comparison group.
| Situation | Consider first | Key condition |
|---|---|---|
| Two groups compared before and after a policy | DID | Parallel trends: without treatment, the groups would have moved in parallel |
| A policy in one region, with several possible comparison regions | Synthetic control | A weighted comparison group that closely reproduces the pre-policy trend |
| Tracking how a policy’s effect changes over time | Event study | Period-by-period trends and other changes happening at the same time |
When only one region got the policy
When a policy was adopted by just one city or country, synthetic control can help. It builds a comparison unit from a weighted combination of other regions. You need enough suitable donor regions, though, and you have to check that the synthetic unit tracks the pre-policy trend closely.
When there’s an eligibility cutoff or a similar comparison group
The right method depends on how people ended up being treated.
When a score or cutoff decides who gets the program
Maybe households qualify for a subsidy when their income falls below a set amount, or students get into a program when their test score clears a bar. In cases like these, consider a regression discontinuity design (RDD). Because it compares people just on either side of the cutoff, it generally estimates a local effect for people near that cutoff.
When treated and untreated people differ
If the two groups differ on observed characteristics such as age, income, or past performance, consider propensity score matching (PSM) or propensity score weighting. Just don’t expect either one to fix differences driven by factors you didn’t observe.
PSM is a tool for improving balance on the covariates you observed. If you missed an important confounder that affects both treatment assignment and the outcome, bias can remain.
When the treatment and the outcome share a common cause
How to deal with endogeneity, and what repeated observations can and can’t do.
When treatment and outcome may affect each other, or omitted variables are a worry
Take years of schooling and wages: a person’s ability can affect both. Whenever an explanatory variable may be correlated with the error term, consider instrumental variables (IV). The catch is that the instrument has to meet strong conditions. Among other things, it must affect the treatment while having no direct effect on the outcome except through the treatment.
When the same people or regions are observed over time
Panel fixed effects help control for characteristics of a person or region that don’t change over time. They don’t handle confounders that do change over time, or reverse causation. And having panel data doesn’t, by itself, identify a causal effect.
| Situation | Method | Be sure to check |
|---|---|---|
| You have a candidate instrument that is plausibly exogenous | IV | Relevance, exclusion restriction, independence |
| Repeated observations of people, firms, or regions | Panel fixed effects | Which time-invariant confounders are controlled; time-varying confounders |
| Effects that change period by period around treatment | Event study | Pre-trends, dynamic effects, differences in treatment timing |
What should you check before running the analysis?
Check your identifying assumptions and the validity of your data before you worry about statistical significance.
| What to check | Why it matters |
|---|---|
| Treatment and comparison | Be clear about who got which treatment, when, and whom they’re being compared with. |
| Pre-treatment differences | For DID, look at pre-trends; for synthetic control, check the pre-policy fit. |
| Manipulation of the cutoff | For RDD, make sure people near the cutoff didn’t sort themselves onto one side of it. |
| Covariate balance and overlap | For PSM and similar methods, confirm there are enough comparable units and that key characteristics are balanced. |
| Instrument validity | An instrument needs evidence for both its strength and the exclusion restriction. |
| Changes happening at the same time | Consider whether another policy or event could explain the outcome. |
| Whom the effect applies to | Be clear whether your estimate is an average effect for everyone, an average effect on the treated, or an effect near the cutoff. |
Where you can, back up your results with checks suited to your design, such as placebo tests, sensitivity analyses, and pre-trend checks. Running one diagnostic checklist across every method doesn’t make sense.
All 7 causal inference methods at a glance
Organized by the design of the comparison, not by the type of data.
In causal inference, the comparison design comes before the method.
You need a convincing way to compare what happened with what would have happened without the treatment.
Start by pinning down when the policy took effect, how treatment was assigned, what your comparison units look like, and whether endogeneity is a concern. Then the choice follows from the design. With a comparison group observed before and after a policy, look at DID. If a cutoff decides who gets treated, look at RDD. If you need to combine several comparison regions, look at synthetic control. To shrink differences in observed characteristics, consider PSM, and if you have a source of valid exogenous variation, consider IV.
Whatever method you choose, state its assumptions clearly and keep straight what the data can confirm and what they can’t.