Regression Discontinuity Design (RDD)

Suppose a university gives scholarships to students whose GPA in the previous semester was 3.5 or higher. A student with a 3.49 GPA does not get a scholarship, but a student with a 3.50 can. The two students’ grades differ very little, but whether they receive a scholarship is different.

So by comparing students close to the cutoff like this, can we learn how the scholarship affected academic achievement in the next semester? A standard causal inference method used for questions like this is regression discontinuity design (RDD).

In situations where whether a policy applies depends on a threshold in a score, age, income or the like, RDD estimates the policy effect by comparing outcomes around the cutoff. Even without a randomized experiment, the causal effect of the policy can be estimated if the units just above and just below the cutoff are sufficiently similar.

In this article we’ll cover the basic principle of RDD, a worked calculation, the regression model, the difference between sharp and fuzzy RDD, bandwidth selection, the key assumptions and how to interpret the results.

For example, suppose a university gives scholarships to students whose previous-semester GPA was 3.5 or higher. Whether a scholarship is awarded is then determined as follows.

RDD checks whether the outcome changes discontinuously at this cutoff. The variable against which the cutoff is set is called the running variable (or assignment variable). In the scholarship example, previous-semester GPA is the running variable and 3.5 is the cutoff.

The core of the analysis is checking whether next-semester grades also change discontinuously when scholarship receipt switches at the cutoff. However, this change can be interpreted as the causal effect of the scholarship only if conditions are met, such as students just above and below the cutoff being continuously similar in their other characteristics.

Students who received scholarships may well have higher average grades. But that may be not an effect of the scholarship but simply because their grades were already high enough to earn one. RDD reduces this problem by comparing students around the cutoff.

For example, students with previous-semester GPAs of 3.49 and 3.50 are very similar academically. If whether they get a scholarship still differs because of the 3.50 threshold, we can analyze the discontinuous difference in outcomes that occurs just above and just below the cutoff.

So the strength of RDD is that, rather than comparing all students indiscriminately, it makes the comparison around the cutoff where treatment status changes.

A researcher wants to find out whether the scholarship affected students’ academic achievement in the next semester.

Study design

In this example, assume that every student with a previous-semester GPA of 3.50 or higher receives the scholarship and no one below 3.50 does. Also assume that no other program changes at 3.50 besides the scholarship rule. This structure is called a sharp regression discontinuity design (sharp RDD).

The research question is as follows.

Put previous-semester GPA on the horizontal axis and next-semester GPA on the vertical axis. You can then see how outcomes differ between students who did and did not receive a scholarship on either side of the 3.50 cutoff.

A conceptual diagram showing the discontinuous difference that appears when regression lines are estimated separately on each side of the cutoff.

In the graph, the line to the left of the cutoff shows the predicted grades of students who did not receive a scholarship. The line on the right shows the predicted grades of students who did. If the two lines meet the cutoff at different heights, there may be a discontinuity in the outcome.

RDD uses this discontinuous difference at the cutoff as the key estimate of the treatment effect.

The treatment effect in a sharp RDD can be written as follows.

Here c is the treatment cutoff and X is the running variable. For example, suppose the estimated next-semester GPAs at the cutoff are as follows.

The calculation is as follows.

So the effect of receiving a scholarship at the cutoff is estimated at 0.20 GPA points. This can be interpreted to mean that, for students with previous-semester GPAs near 3.50, receiving a scholarship raised next-semester GPA by about 0.20 points. This causal interpretation is possible only when the necessary identifying assumptions are met, however.

Also, 3.62 and 3.42 are not observations of two individual students; they are values estimated from the regression functions on each side of the cutoff.

A typical local linear regression model can be written as follows.

Here each term means the following.

This model allows different slopes below and above the cutoff. The important point is that, instead of forcing a single straight line onto the whole GPA range, it estimates the relationship using the observations around the cutoff. A real RDD analysis also considers the bandwidth, kernel weights, bias correction and an appropriate method for computing confidence intervals.

RDD estimation results

These are not results from analyzing real student data.

The scholarship effect was estimated at 0.20 points. This means that, for students near the 3.50 cutoff, receiving a scholarship raised next-semester GPA by about 0.20 points on average. The 95% confidence interval ran from 0.06 to 0.34 points and did not include 0.

So under the model and assumptions used, we can say a statistically significant positive effect was estimated.

This range is called the bandwidth. For example, if the cutoff is 3.50, it could be set as follows.

With a narrow bandwidth, you compare students very close to the cutoff. That makes the estimate less sensitive to differences in functional form far from the cutoff, but the sample used in the analysis is small, so the estimate can be unstable.

With a wide bandwidth, on the other hand, you can use more data, but students far from the cutoff are also included, so bias from how the functional form is specified can grow. Bandwidth selection therefore involves a trade-off between bias and variance.

In practice you can use data-driven bandwidth selection procedures, and it’s a good idea to also check how stable the results are across several bandwidths. You should not, however, report only the bandwidths that give statistically significant results.

Sharp RDD

The cutoff completely determines treatment. For example, every student with a previous-semester GPA of 3.50 or higher receives the scholarship, and no one below 3.50 does.

Fuzzy RDD

Crossing the cutoff makes the probability of treatment jump, but it does not completely determine actual treatment. For example, students with a GPA of 3.50 or higher may be eligible to apply for the scholarship, but some may not apply. Conversely, some students below 3.50 may receive a scholarship through a special selection track.

In a fuzzy RDD, the cutoff acts like an instrumental variable that affects treatment. So it uses both the change in the outcome at the cutoff and the change in the actual probability of treatment. To interpret a fuzzy RDD effect causally, however, additional assumptions such as monotonicity and the exclusion restriction may also be needed.

Continuity of potential outcomes

This is the assumption that, without the policy, the expected outcomes just above and just below the cutoff would have continued smoothly. In other words, without the treatment, there should be no reason for next-semester grades to change suddenly between students around 3.49 and 3.50.

There must be no precise manipulation of the running variable.

If students or the school can deliberately adjust GPAs to 3.50 to get the scholarship, the characteristics of students around the cutoff can change. For example, if there were a practice of administratively raising a 3.49 GPA to 3.50, the validity of the RDD would be weakened.

No other policy may change at the same cutoff at the same time.

If students with a GPA of 3.50 or higher received not only a scholarship but also a separate mentoring program, it would be hard to tell whether the jump in the outcome is due to the scholarship or to the mentoring program.

Relevant characteristics must be continuous around the cutoff.

If pre-treatment characteristics such as age, household income or prior academic ability change discontinuously at the cutoff, the validity of the analysis is open to doubt. So RDD requires a careful review of the data around the cutoff and of how the program was run.

For example, if an abnormally large number of students are bunched just above the 3.50 cutoff, you should check for GPA manipulation, rounding rules and the like. A standard way to check this is a density discontinuity test of the running variable. You can also check whether there are discontinuous differences in students’ household income or prior academic records at the cutoff.

However, the absence of statistically significant differences in these diagnostics does not prove that all the identifying assumptions hold. So it’s important to also review how the program actually operated and how the data were generated.

For example, if scholarship receipt is determined by a 3.50 GPA cutoff, RDD can be useful. If, on the other hand, a scholarship program was newly introduced in one region but not in others, you might consider DID.

If scholarship recipients were selected based on individual characteristics without a clear score cutoff, you could consider PSM. What matters is not deciding which method is generally better, but choosing the identification strategy that fits how the policy was actually implemented.

Presenting evidence on possible manipulation of the running variable, the continuity of covariates around the cutoff and the results of bandwidth sensitivity analyses also helps readers assess the validity of the analysis. Using the university scholarship example above, you could write it up like this.

In a real study, you should also report the chosen bandwidth and kernel function, the bias-correction and confidence-interval estimation methods, the number of observations around the cutoff and so on. You should also state clearly whether a sharp or a fuzzy RDD design was used.

For example, if scholarships are given to students with a previous-semester GPA of 3.50 or higher, you can compare students just above and just below the cutoff. If next-semester GPA changes discontinuously at the cutoff where scholarship receipt switches, then under suitable identifying assumptions that difference can be interpreted as the local causal effect of the scholarship.

RDD can make fairly convincing causal inference possible even with data from no randomized experiment, but you need to check for possible manipulation of the running variable, continuity around the cutoff and whether other policies were applied at the same time.

Also, RDD results are fundamentally effects for units near the cutoff, so they should not be generalized directly to the whole population.