Triple Differences (DDD)

Suppose a local government introduced a policy restricting factory operations after 10 p.m. to reduce nighttime noise around an industrial park. After the policy took effect, the average sleep time of residents near the industrial park increased. Can we then conclude that sleep time increased because of the nighttime noise regulation?

During the same period, residents' work patterns may have changed, or living conditions may have improved in some regions. Residents living far from the industrial park may also have experienced similar changes. Difference-in-differences (DID) can partly reduce these problems by comparing the before-and-after changes in regions where the policy was implemented and regions where it was not.

But what if we could further separate people who are directly affected by the policy from those who are hardly affected? For example, we could compare residents living near the industrial park with those living far from it.

In this way, the method that estimates a policy effect by combining differences along three dimensions (whether the policy was implemented, before vs. after implementation, and whether people are affected by the policy) is the triple-difference method (Difference-in-Difference-in-Differences, DDD).

The triple-difference method is a causal inference method that extends difference-in-differences, and it is used in policy evaluations that must account for additional differences that exist between groups. In this article we look at the basic principle of triple differences, how it is calculated, a real-world example, the regression model, identifying assumptions, how it differs from DID, and how to interpret the results and what to watch out for.

Basic DID calculates two differences. First, the difference between before and after the policy. Second, the difference between the group that received the policy and the comparison group. DDD adds a third comparison criterion to these.

For example, in the noise regulation policy, the groups can be divided as follows.

Combining these three criteria produces 8 group-period combinations in total. DDD compares the differences in the outcome across these combinations. However, using three comparison criteria does not automatically remove every confounder.

An additional identifying assumption is needed about how the gaps between groups would have changed in the absence of the policy.

For example, we could compare changes in the sleep time of residents near the industrial park between the region where the noise regulation was implemented and the region where it was not. But if changes in the living conditions of near residents already differed between the two regions, the DID estimate can be biased. DDD uses far residents, who live a long way from the industrial park, as an additional comparison group.

This lets us compare how the difference between near and far residents within each region changed before and after the policy.

DDD is not always better than DID. If the third comparison group is inappropriate or is also affected by the policy, DDD can be biased too. So it is important to check what role the additional group actually plays.

A researcher wants to evaluate the effect of the regulation on residents' average sleep time. Residents of Region A, where the regulation was implemented, and Region B, where it was not, were surveyed. In both regions, residents living near the industrial park were also distinguished from those living relatively far from it.

Study design

In this example, assume that the environments around the industrial parks in Regions A and B are similar, but the regulation was introduced only in Region A. Also assume that far residents are not directly affected by the regulation and that there are no spillover effects from the policy. The research question is as follows.

The following data are from a survey of residents' average sleep time.

Average sleep time before and after the policy

First, calculate the difference in change between near and far residents in Region A.

In Region A, the sleep time of residents near the industrial park increased 0.7 hours more than that of far residents. Next, calculate the same difference in Region B.

In Region B, the sleep time of near residents increased 0.2 hours more than that of far residents. Finally, subtract the two regions' DID results once more.

So the DDD estimate is +0.5 hours, or 30 minutes. This means that in the regulated region, the increase in sleep time for residents near the industrial park was 30 minutes larger after also accounting for the changes in the additional comparison groups. If the appropriate identifying assumptions hold, this can be interpreted as the causal effect of the nighttime noise regulation.

But not all of this 1 hour may be the effect of the noise regulation. The sleep time of far residents in Region A also increased by 0.3 hours. So the additional change for near residents in Region A is 0.7 hours. If the analysis stopped here, the difference in change between near and far residents within Region A would be taken as the policy effect.

However, in Region B too, the sleep time of near residents increased 0.2 hours more than that of far residents. In other words, there may be different changes between residents near and far from the industrial park regardless of the noise regulation. DDD subtracts this additional change as well.

In this way, DDD additionally accounts for differences in change between specific groups that are not removed by a simple before-after comparison or by difference-in-differences. However, whether this subtraction actually removes confounding depends on how valid the assumptions about the third comparison group are.

Let us define three binary variables.

• Treat_i: 1 for the regulated region, 0 for the unregulated region
• Post_t: 1 after the policy took effect, 0 before
• Near_i: 1 for residents near the industrial park, 0 for far residents

The basic DDD model is as follows.

The most important part here is the triple interaction term of the three variables, Treat × Post × Near. The regression coefficient of this term, β₇, is the DDD estimate.

In a basic linear model that includes all the lower-order interaction terms and compares group means with the same weighting, β₇ corresponds to the third difference calculated above. However, if additional control variables are included or weights are applied, how the regression coefficient is calculated and interpreted can change.

Main regression results

The coefficients were set to match the group means above. The standard errors and p-values are illustrative numbers, not values estimated from a real sample.

The coefficient on the triple interaction term is 0.50. This means that, after subtracting the changes along the other comparison dimensions, the average sleep time of near residents in the regulated region is estimated to have increased by an additional 0.5 hours—an increase of about 30 minutes.

However, a statistically significant result does not mean the causal identifying assumptions are automatically met. A real study needs a research design that can support how the third difference would have changed in the absence of the policy.

This can be understood as a parallel trends assumption for triple differences. In the example above, it means that if the policy had not been implemented, the gap in sleep time between near and far residents should have changed in a similar way in Regions A and B.

The sleep time levels of all groups need not be the same here. Nor do the four groups each have to follow exactly the same trend. What matters is the condition that, without the regulation, the difference in change between near and far residents in Region A would have been the same as the corresponding difference in Region B.

For example, if near residents in Region A were on track to see their sleep time increase faster than other regions regardless of the policy, this assumption could be violated. In that case, the DDD estimate may include not only the policy effect but also changes that differed across groups.

But if you have data from several time points before the policy, you can examine its plausibility. For example, suppose the sleep time of the four groups was surveyed over the 3 years before the policy. For each year, you can calculate the following gap.

This value shows how different the gap between near and far residents is in Region A and Region B at a given time point. If G_t stayed broadly stable before the policy, that can be one piece of evidence supporting the DDD parallel trends assumption.

Conversely, if this gap kept growing or shrinking even before the policy, it may be hard to interpret the post-policy difference as an effect of the regulation. An event-study approach, which estimates the triple interaction effect at several pre-policy time points, can also be used.

However, pre-policy coefficients that are not statistically significant do not prove the assumption. You need to consider the influence of sample size and statistical power.

It is meaningful when the additional comparison group really plays an appropriate role. For example, suppose far residents in Region A are not directly affected by the noise regulation but are affected similarly by other changes in the region. In that situation, the third difference can separate the policy effect better.

In that case, treating far residents as an unaffected comparison group yields an estimate that differs from the total effect of the policy. Also, comparing along three dimensions splits the groups used in estimation more finely, so uncertainty can grow when the sample is small.

So when choosing DDD, what matters is not that there are more comparison dimensions but whether you can explain the causal role of the additional group.

For example, even if 2,000 residents were surveyed, the noise regulation is implemented at the regional level, not for individual residents. Sleep time of residents in the same region can be influenced by a common environment, so it may be hard to treat them as independent observations. So you should consider cluster-robust standard errors or similar methods that match the policy unit.

But caution is needed when, as in this example, Region A is the only region that implemented the policy and Region B is the only comparison region. Even with many residents, there are very few independent policy units. In this situation, the usual approximation for region-level cluster-robust standard errors can be unreliable.

Also, you must not assume that the regression p-values shown above could actually be computed stably using just these two regions. In a real study, you should obtain more policy and comparison regions, or consider randomization-based inference or inference methods that account for a small number of clusters, as suits the study design.

So when reporting DDD results, it is important to check not only the sample size but also the number of policy units.

If the third comparison group is affected by the policy, the meaning of the estimate can change.

Check the trends in change by group.

If the third difference was already changing before the policy, it may be hard to interpret the post-policy DDD estimate as a causal effect.

Check whether other policies were implemented at the same time.

If a new housing improvement project began near the industrial park in Region A at the same time as the noise regulation, it may be hard to tell which policy caused the increase in sleep time.

Consider changes in group composition.

If residents moved to areas with better sleeping conditions after the policy, the changes in each group's average sleep time may include changes in population composition.

Specify the three-way interaction correctly.

A DDD regression model should generally include the main terms of the three variables and all the necessary lower-order interaction terms. Simply adding the triple interaction term alone may fail to estimate the intended difference properly.

Check how far the estimates can be generalized.

There is no guarantee that the effect of noise regulation on residents near this industrial park will be the same in other regions or for other types of industrial parks.

You should also explain the rationale for choosing the additional comparison group and how you examined the parallel trends assumption. Using the nighttime noise regulation example from above, it could be written as follows.

In a real study, you should add standard errors and confidence intervals calculated with an appropriate inference method.

In particular, when there are few policy units, ordinary regression p-values may be hard to use as they are, so the inference method and its limitations should be reported clearly.

In addition to whether and when the policy was implemented, it also compares groups that are affected by the policy with groups that are not. For example, in the industrial park noise regulation case, we can consider regulated vs. unregulated regions, before vs. after the regulation, and whether residents live near the industrial park all together.

Through these comparisons, it tries to remove additional group-specific differences in change that are hard to explain with the simple change in the policy region or with difference-in-differences alone. But adding one more comparison dimension does not by itself make the causal conclusion more valid.

The assumption that the third difference would have stayed constant in the absence of the policy is important, and you need to examine the choice of comparison groups, other policies implemented at the same time, changes in group composition and so on together.