Difference-in-differences (DID)

Suppose the government introduced a youth employment support policy in certain regions. After the policy took effect, the youth employment rate in those regions rose from 60% to 68%. Can we then conclude that the policy raised the youth employment rate by 8 percentage points?

That is hard to say. During the same period, the economy may have recovered or companies may have increased hiring, so the youth employment rate could have risen regardless of the policy. So looking only at the change in the regions where the policy was implemented makes it hard to estimate the policy's effect accurately.

What if we could also observe regions where the policy was not implemented? For example, if the youth employment rate in regions without the policy also rose from 58% to 62% over the same period, part of the 8-point rise in the policy regions may be a change common to both kinds of regions.

In this way, the method that estimates a policy effect by subtracting the change in comparison regions from the change in policy regions is difference-in-differences (DID).

DID is a causal inference method widely used in policy evaluation, economics, and social science research. It can be used to analyze the impact of a policy or program when a randomized experiment is hard to carry out.

In this article, we cover difference-in-differences from its basic principle to how to calculate it, the regression model, a real policy evaluation example, the parallel trends assumption, event studies, and points to watch out for in the analysis.

The analysis generally needs two groups.

• Treatment group: the group where the policy or program was implemented
• Control (comparison) group: the group where the policy was not implemented during the same period

Outcomes are then observed before and after the policy takes effect. For example, if a youth employment support policy was introduced only in certain regions, those regions are the treatment group and the other regions without the policy are the comparison group. In DID, you first calculate each group's change from before to after the policy.

Then you subtract the comparison group's change from the treatment group's change. This helps remove differences between the groups that existed before the policy, as well as changes over time that affected both groups in common. However, it requires an important assumption: that without the policy, the two groups would have changed along similar trends.

For example, if the youth employment rate in the policy regions rose from 60% to 68%, the change is as follows.

But these 8 points may include not only the policy effect but also changes in the economy, seasonal factors, changes in the industrial environment, and so on.

To reduce this problem, DID also takes the comparison group's change into account.

DID can be more useful for policy evaluation than a simple before-after comparison, but adding a comparison group does not remove every confounder. In particular, if an event occurred that affected the two groups' trends differently, the DID estimate can also be biased.

Here, T denotes the treatment group and C the comparison group. Let's calculate it using the youth employment support policy example.

Youth employment rate before and after the policy

The change in the treatment group is 8 points. The change in the comparison group is 4 points. So the DID estimate is as follows.

In other words, the youth employment rate in the policy regions rose 4 points more than the change in the comparison regions. If the required causal inference assumptions, such as parallel trends, hold, these 4 points can be interpreted as the policy's average treatment effect.

The important point is that the DID estimate is not the 8-point total rise in the policy regions but the 4-point difference between the two groups' rises.

In the example above, the employment rate in the policy regions rose from 60% to 68%. The non-policy regions rose 4 points, from 58% to 62%. If we assume parallel trends, we can estimate that without the policy, the employment rate in the policy regions would also have risen from 60% to 64%.

Since the actual employment rate after the policy was 68%, the difference between these two values is 4 points.

Here, the 64% employment rate without the policy is not an actually observed value. It is a counterfactual outcome estimated using the comparison group's change. The causal interpretation of DID depends on how valid this counterfactual estimate is.

The policy was implemented only in some regions and not in other similar regions. The researcher surveyed the youth employment rate by region for 2 years before and after the policy.

Study design

To estimate the policy effect, you need to consider not just the difference in employment rates but also the timing of the policy and each group's change. It is also important to check how similar the comparison group is to the policy regions and what the employment rate trend looked like before the policy.

The basic two-group, two-period DID regression model is as follows.

Here, each term means the following.

The most important thing in this model is the coefficient of the Treatment × Post interaction term. This coefficient represents the difference in the two groups' before-after changes. In the example above, this value is 4, so it is interpreted as the employment rate in the policy regions rising 4 points more than in the comparison regions.

Note that this interpretation applies when the outcome variable is entered in percentage units (e.g., 60, 68). If the outcome variable is entered as a proportion between 0 and 1, the coefficient is expressed as 0.04.

DID regression results

The employment rate is assumed to be entered in percentage units.

The coefficient for policy region is 2.00. This means the difference in youth employment rates between the two groups before the policy was 2 points. The coefficient for after the policy is 4.00.

This means the comparison group's youth employment rate rose 4 points after the policy. The coefficient of the most important term, the interaction, is 4.00. This means the youth employment rate in the policy regions rose 4 points more than in the comparison group.

The p-value was .002, confirming a statistically significant positive DID estimate. But statistical significance alone does not prove the policy's causal effect. You need to check the validity of the study design, including the parallel trends assumption.

It is the assumption that, had the policy not been implemented, the outcome variable of the treatment and comparison groups would have followed the same trend on average. The important point here is that the two groups' employment rate levels do not have to be the same.

For example, even if the treatment group's employment rate before the policy was 60% and the comparison group's was 55%, the parallel trends assumption can hold if, without the policy, the two groups' employment rates would have changed by similar amounts. Conversely, even if the employment rates were the same before the policy, it is a problem if their subsequent trends are expected to differ.

Example of pre-policy trends

In this example, both groups rose 2 points each year before the policy. So the observed past trends look consistent with the parallel trends assumption. But similar past trends do not prove that, without the policy, the groups would necessarily have followed the same trend after the policy took effect.

This is because the parallel trends assumption is about counterfactual outcomes that cannot actually be observed.

The most basic way is to compare the outcome variable at several points before the policy in a graph. For example, you can look at the youth employment rate for 4–5 years before the policy to see whether the two groups changed in a similar direction and by a similar amount.

You can also use an event study regression to check how the difference between the groups changed at each point before the policy. However, the fact that the pre-policy coefficients are not statistically significant does not by itself let you conclude that parallel trends hold.

This is because, if power is low, a real difference in trends may not show up as significant. So it is good to consider the following as well.

• Visual similarity of pre-policy trends
• Economic and social characteristics of the treatment and comparison groups
• How the policy regions were selected
• Other factors causing change that appeared before the policy
• Possible anticipation responses or effects before the policy

In particular, if the policy was introduced only in regions where employment was deteriorating rapidly, the selection of policy targets itself can make the causal interpretation of DID difficult.

Whereas basic DID compares two broad periods, before and after the policy, an event study looks at the relationship at each point in time, such as 2 years before, 1 year before, the year of implementation, 1 year after, and 2 years after. For example, suppose we obtained the following results.

Estimated coefficients relative to policy start

In this example, 1 year before the policy was set as the reference point. The pre-policy coefficients were not very different from 0, and after the policy the coefficients gradually increased. These results show a pattern in which, after the policy, the youth employment rate in the treatment group became relatively higher compared with the comparison group.

However, insignificant pre-policy coefficients alone do not complete causal identification. It is best to think of the event study as a supplementary method for seeing how the policy effect changes over time and for examining the DID assumptions.

For example, it can be written as follows.

Here, each symbol means the following.

Region fixed effects control for characteristics of each region that do not change. Year fixed effects control for period-specific changes that act on all regions in common. In such a model, the policy effect is estimated using changes over time within the same region and changes in comparison regions.

However, including fixed effects does not automatically satisfy the DID parallel trends assumption. In particular, if different economic shocks hit different regions, region and year fixed effects alone may not control for them sufficiently.

But in real policies, implementation timing can differ by region. For example, suppose region A introduced a youth employment support policy in 2022, region B in 2023, and region C in 2024.

This situation is called staggered difference-in-differences.

When the policy effect differs by region or by time since implementation, the ordinary two-way fixed effects regression coefficient can be interpreted differently from a simple average policy effect. In particular, problems can arise in estimation when regions that have already implemented the policy are used as comparison groups for other regions.

So for data with different implementation timing, it is good to consider methods that estimate treatment effects by adoption cohort and time point. Well-known examples include the Callaway–Sant'Anna and Sun–Abraham approaches.

The important point is that just because they share the name DID does not mean the same regression model can be applied to every policy adoption structure.

Parallel trends assumption

The assumption that, without the policy, the treatment and comparison groups would have changed along similar trends on average is crucial.

Anticipation before the policy

If companies or individuals knew about the policy in advance and changed their behavior, policy effects could occur even before the official start date. In this case, treating the pre-policy period as unaffected may be inappropriate.

Other policies or shocks at the same time

If a large industrial park was built in the treatment regions at the same time as the youth employment support policy, it may be hard to separate out the cause of the rise in employment.

Spillovers between groups

The employment support policy in the treatment regions may also affect companies and young people in nearby comparison regions. In this case, the comparison group may actually have been affected by the policy.

Changes in group composition

If young people moved into the treatment regions after the policy, the change in regional employment rates may reflect not only the policy's direct employment effect but also changes in population composition.

Calculating standard errors

Observations from the same region may be related to each other across years. So you should consider standard errors clustered at the region level. In particular, if the policy was implemented in only a few regions, ordinary cluster-robust standard errors alone may not be enough, so inference methods suited to a small number of clusters should be considered.

You should also explain in particular how the pre-policy trends were checked and which fixed effects and control variables were included. Using the youth employment support policy example above, you could write it as follows.

In real studies, it is important to report the pre-policy trends and the rationale for choosing the comparison regions as well. Also, if the policy was adopted at different times in different regions, you should state clearly which DID estimation method was applied and how the comparison group was defined.

Even if the outcome in the policy regions increased by 8 points, if the comparison regions increased by 4 points over the same period, the DID estimate is 4 points. This approach helps remove fixed differences between groups and changes over time that occurred in both groups in common.

However, to interpret DID results as a causal effect, you need the assumption that, without the policy, the two groups would have followed parallel trends. So you should check the pre-policy trends and also examine whether other policies were implemented at the same time, spillovers between groups, how policy targets were selected, and so on.

Also, when the policy starts at different times for different groups or the policy effect changes over time, you may need an estimation method that reflects that structure rather than the basic DID model.