Synthetic Control Method

Suppose a city introduced a fare-free public transit policy. The goal of the policy is to increase residents' use of public transit and reduce car traffic downtown. If car traffic fell after the policy started, can you conclude that the fare-free policy worked?

It's hard to draw that conclusion. Car traffic might have fallen on its own over the same period because gas prices rose or more people worked from home. Then should you compare with another city that didn't adopt the policy?

That approach has problems too. Cities differ in population, transit networks, industrial structure, and car ownership rates, so it's hard to simply pick one other city as the comparison. The method to use in this situation is the synthetic control method (SCM).

The synthetic control method combines several cities that didn't adopt the policy in suitable proportions to build a hypothetical comparison city with characteristics similar to the city that did. This hypothetical city is called the synthetic control. The policy effect is then estimated by comparing the outcomes of the actual policy city with those of the synthetic control.

In this article we'll look at how the synthetic control method works, how the weights are calculated, a real policy evaluation example, counterfactual outcomes, estimating the policy effect, placebo tests, and what to watch out for in the analysis.

The counterfactual outcome is what would have happened had the policy not been implemented. For example, suppose City A introduced fare-free public transit starting in 2023. After the policy started, City A's car traffic can actually be observed.

But you can't directly observe how car traffic would have changed if City A had not adopted the fare-free policy. The synthetic control method estimates that value by combining Cities B, C, D, and others that didn't adopt the policy.

Rather than simply averaging several cities, it sets a weight for each city so that the combination closely reproduces City A's car traffic and related characteristics before the policy. After building a synthetic control with similar pre-policy changes, the difference between the two after the policy is used as the estimate of the policy effect.

But they differ in how the comparison group is built. In DID, the key assumption is that the treated and comparison groups would have followed parallel trends without the policy. The synthetic control method instead combines several comparison units so that their pre-policy outcomes and characteristics match, and uses that to construct the counterfactual outcome.

For example, if only one city adopted a new transportation policy, the synthetic control method can be useful. Using the synthetic control method doesn't, however, mean the policy effect is automatically estimated accurately.

There must be comparison units that can reproduce the pre-policy outcomes well, and you also need to check that changes other than the policy didn't distort the outcomes after the policy started.

A researcher wants to analyze how much car traffic fell after the policy started. The observation period runs from 2019 to 2025, and other cities that didn't adopt the policy were studied as well.

Study design

In this example we express car traffic as an index. The index sets City A's 2019 traffic to 100, and the other years are converted on the same basis. So an index of 90 means car traffic about 10% lower than in 2019.

The research question is:

This set is called the donor pool. For example, suppose Cities B, C, and D showed changes in car traffic and city characteristics similar to City A's. The synthetic control method combines these cities to create a hypothetical city similar to City A.

Example weights for the synthetic control

In this case, the synthetic control is built as follows.

Here Y_SC,t is the synthetic control's car traffic at time t. For example, if the traffic index is 110 for City B, 100 for City C, and 115 for City D, the synthetic control's value is:

So the synthetic control's traffic index is 108. In a real analysis, the researcher doesn't set the weights arbitrarily; they are determined through an optimization that reproduces the pre-policy data well. In the traditional synthetic control method, the weights are usually constrained to be nonnegative and to sum to 1.

For this, you can use not only pre-policy car traffic but also city characteristics that may be related to the policy outcome. For example, you might consider variables like these.

The weights depend on how closely they reproduce these variables and the pre-policy outcomes. For example, if City B has a population similar to City A's but a very different trend in car traffic, giving City B a high weight may not be appropriate.

Conversely, cities whose pre-policy traffic changes and main predictors are similar can help build the synthetic control even if their population is a bit different.

For example, suppose the results looked like this.

Pre-policy traffic index

Before the policy, City A's and the synthetic control's traffic indexes were very similar. A standard measure that summarizes these differences is the root mean squared prediction error (RMSPE). The formula is:

In this example it works out as follows.

So the pre-policy RMSPE is about 0.87 index points. This means the synthetic control reproduced City A's pre-policy traffic fairly closely. A good fit alone, however, doesn't guarantee that the post-policy counterfactual is accurate.

In particular, if the pre-policy observation period is short, or if the fit chases random fluctuations too closely, post-policy predictions can be unstable.

For example, suppose the results looked like this.

Post-policy traffic index

The policy effect at each time point in the synthetic control method can be calculated as follows.

This means that after the policy started, City A's car traffic index was 15 points lower than the estimate of what it would have been without the policy. It was 21 points lower in 2024 and 26 points lower in 2025. These results show that the gap between the actual City A and the synthetic control widened over time.

To interpret these differences as the causal effect of the policy, though, you need to assume that the synthetic control provides an appropriate counterfactual outcome.

2019–2022 is the pre-policy period and 2023–2025 the post-policy period.

Before the policy, traffic in the actual City A and in the synthetic control changed in similar ways. After the policy started in 2023, however, the two series clearly diverge. The traffic index fell in the actual City A, while it rose in the synthetic control.

So compared with how traffic would have changed without the policy, car traffic in the actual City A is estimated to have fallen relatively. A visualization like this is useful for checking pre-policy fit and the post-policy gap at the same time.

In the example above, the estimated effects for the three post-policy years were -15, -21, and -26. So the simple average post-policy effect is:

In other words, over 2023–2025, City A's car traffic index was on average about 20.7 points per year lower than the synthetic control's. But this value is a difference in index points. It should not be read as car traffic falling by exactly 20.7%.

To calculate a percentage decrease, compute the relative difference using the synthetic control's value at each time point as the base. For 2023, for example:

So in 2023, City A's traffic was about 13.9% lower than the estimated traffic without the policy. The index-point difference and the relative percentage difference are different measures, so it's best to report them separately.

Statistical inference needs particular care when the policy applies to only one city. A method often used to address this is the placebo test.

In-space placebo test

Apply the synthetic control method to other cities that didn't actually adopt the policy, as if they had.

For example, set each of Cities B, C, and D, other than City A, as a hypothetical treated city and calculate its difference from its synthetic control. Then check how large City A's actual post-policy difference is compared with the hypothetical effects in the other cities. If City A's difference is unusually large compared with the other cities, that can be evidence of a policy effect.

When comparing the size of placebo effects, however, you should also take pre-policy fit into account.

In-time placebo test

Set a point in time before the actual policy start as a hypothetical start date. For example, although the policy actually started in 2023, you can run the analysis as if it had started in 2021.

If a large effect appears even before the policy started, you should consider whether the model describing the policy effect is unstable or whether other factors were driving changes. A placebo test, however, does not fully prove that the identifying assumptions are valid.

So it's a good idea to check how sensitive the main results are to these choices.

Leave-one-out analysis

Drop the cities that carry high weights in the synthetic control one at a time and rerun the analysis. For example, if City B has a 50% weight, you can check how the results change when City B is excluded. If excluding City B reverses the direction of the policy effect, the stability of the results needs further review.

Changing the pre-policy period

Change the pre-policy period used to build the synthetic control. For example, you can see how much the results change when you use data from 2020–2022 instead of 2019–2022. But if the pre-policy period gets too short, the comparison group becomes less reliable.

Changing the set of predictors

Check how the synthetic control's weights and the policy effect change when you include or exclude predictors such as population size, car ownership rate, or public transit ridership. This kind of analysis helps you check whether the results depend too heavily on a particular choice of variables or on a single comparison city.

If the synthetic control doesn't reproduce the pre-policy outcomes well, it's hard to trust the post-policy gap. So always check the pre-policy graph, the RMSPE, and so on.

You need suitable comparison cities.

Not every city that didn't adopt the policy is a suitable comparison. Build the donor pool with city size, transportation environment, economic changes, and so on in mind.

Check for spillover effects of the policy.

If residents of nearby City B also started using City A's transit because of City A's fare-free policy, City B may also be affected by the policy. Including such a city in the comparison group can distort the estimate of the policy effect.

Consider other changes that happened at the same time as the policy.

If downtown vehicle restrictions or major road construction began at the same time as the fare-free policy, it may be hard to tell which policy caused the drop in car traffic.

You need enough pre-policy data.

In the synthetic control method, changes in the outcome before the policy provide important information. So if there are too few pre-policy time points, it may be hard to build a reliable synthetic control.

The synthetic control is not a real city.

It is a hypothetical comparison unit built as a weighted average of several cities, so you shouldn't interpret it as having the same environment as any particular real city. And the validity of the policy effect depends less on the weight calculation itself than on the assumption that, without the policy, the synthetic control would have properly reproduced the changes in the actual City A.

For the fare-free public transit example we've been looking at, you could write it up as follows.

In real research, it's best to present the graph of outcomes before and after the policy together with the results of placebo tests and sensitivity analyses. You should also clearly report the criteria for choosing comparison cities, the weight optimization method, the pre-policy fit, and so on.

For example, if one city introduced fare-free public transit, you can combine several cities that didn't adopt it into a hypothetical comparison city. The policy effect is then estimated by comparing car traffic in the actual city with traffic in the synthetic control.

A key feature of the method is that the comparison weights are chosen so that pre-policy outcomes and related characteristics are similar. But a good pre-policy fit doesn't necessarily mean the post-policy counterfactual is estimated accurately as well.

So you also need to check the validity of the comparison cities, spillover effects of the policy, whether other policies were introduced at the same time, and placebo tests and sensitivity analyses.