Econometrics guide, Unit 4, step 16 of 20Previous step Next step

Step 16: Difference-in-Differences and Regression Discontinuity

By the EconTutorials team. 16-minute read with 6 practice questions.

SyllabusUndergraduate econometrics · Wooldridge, Introductory Econometrics, Ch. 13 (policy analysis) · applied microeconometrics

In 30 seconds

Every unit has two potential outcomes, one with the policy and one without, but we only ever see one of them. Policy evaluation is the art of finding a believable stand-in for the missing one.
A naive comparison of treated and untreated groups equals the true effect plus selection bias. A randomised trial makes selection bias zero; the other designs remove it under one named assumption each.
Difference-in-differences (DiD) compares the change over time in a treated group with the change in a control group. Its key assumption is parallel trends. In the book’s minimum-wage data it gives an effect of 2.75 full-time-equivalent (FTE) jobs per restaurant.
Regression discontinuity (RD) compares units just either side of an eligibility cut-off. Its key assumption is continuity with no manipulation. In the scholarship data the jump in GPA at the cut-off is 0.264.
Exam point: always name the identifying assumption precisely and give a concrete way to check it.

Video lesson coming soonA short animated lesson for this step will appear here.

What you will learn

This step pulls together the main tools of modern policy evaluation. By the end you should be able to:

define potential outcomes, the ATE and the ATT, and split a naive comparison into the ATT plus selection bias;
estimate a difference-in-differences effect from a 2×2 table and by regression, with clustered standard errors;
argue for or against the parallel-trends assumption, and list the main threats to a DiD design;
estimate a sharp regression discontinuity by local linear regression, check it across bandwidths and test for manipulation;

Potential outcomes: what “the effect” means

Take one student and an after-school tutoring programme. If she joins, she will score some mark in her final exam; call it $ Y(1) $. If she does not join, she will score a different mark, $ Y(0) $. Both numbers exist in principle before anyone decides anything. The effect of tutoring on her is the difference between them.

Key termFor each unit $ i $ and a treatment $ D_i $ (1 = treated, 0 = not), the potential outcomes are $ Y_i(1) $ and $ Y_i(0) $. The individual treatment effect is $ Y_i(1)-Y_i(0) $. The fundamental problem of causal inference is that at most one of the two is ever observed: the other is the counterfactual. So we estimate averages, such as the ATE, $ E[Y(1)-Y(0)] $, over everyone, or the ATT, $ E[Y(1)-Y(0) \mid D=1] $, over those actually treated.

Chapter 16 of Econometrics Made Simple covers this in full: potential outcomes, randomised trials, difference-in-differences and regression discontinuity, with Python, R and Stata code.Get the textbook · $15

The ATE answers a minister who asks what would happen if a programme were given to everyone; the ATT answers whether it helped the people who took part. They differ whenever effects vary and participants are not a random selection, which is the usual case.

Selection bias: why naive comparisons mislead

The book makes this concrete with a made-up table that no real dataset could contain, because it lists both potential outcomes for eight students. Students A to D are weaker and sign up for tutoring; E to H are stronger and do not.

Student	A	B	C	D	E	F	G	H
$ Y(0) $: score without tutoring	50	55	48	60	70	65	72	68
$ Y(1) $: score with tutoring	58	61	56	64	72	69	75	71
Joins the programme, $ D $	1	1	1	1	0	0	0	0
Individual effect	8	6	8	4	2	4	3	3
Observed score	58	61	56	64	70	65	72	68

Worked example

Using the table, find the ATE, the ATT, the naive difference in observed means and the selection bias, and check that they add up.

Step 1ATE: the average of all eight effects, $ 38/8 = 4.75 $ points.

Step 2ATT: the average for A to D only, $ (8+6+8+4)/4 = 6.50 $ points.

Step 3Naive difference: $ \dfrac{58+61+56+64}{4}-\dfrac{70+65+72+68}{4} = 59.75-68.75 = -9.00 $.

Step 4Selection bias: the tutored students would have averaged $ 53.25 $ without tutoring, against $ 68.75 $ for the others, so the bias is $ 53.25-68.75 = -15.50 $.

Step 5Check: $ -9.00 = 6.50 + (-15.50) $.

Tutoring raised every student’s score, yet the naive comparison says it lowered scores by 9 points. The true benefit is swamped by the fact that weaker students chose to take part.

\[ \underbrace{E[Y \mid D=1]-E[Y \mid D=0]}_{\text{naive difference}} = \underbrace{E[Y(1)-Y(0) \mid D=1]}_{\text{ATT}} + \underbrace{E[Y(0) \mid D=1]-E[Y(0) \mid D=0]}_{\text{selection bias}} \]

Selection bias is the omitted variable bias of Step 6 in a new form: prior attainment sits in the error term and is correlated with $ D $. It can go either way. If keener people sign up for a training course, they would have done well anyway, the bias is positive and a naive comparison overstates the effect. Our post on the direction of omitted variable bias works through the sign rules.

Common mistakeThinking that a large sample cures selection bias. The decomposition holds in the population. With a million students the naive comparison would estimate $ -9 $ more precisely, but it would still be $ -9 $. Bias is about who is compared, not how many.

Randomised trials remove selection bias

If a coin decides who is treated, $ D $ is unrelated to anything about the person, including both potential outcomes. Then $ E[Y(0) \mid D=1] = E[Y(0) \mid D=0] $, the selection bias is zero, and a simple difference in means is an unbiased estimate of the average effect. In regression form it is the slope in $ Y_i = \beta_0 + \beta_1 D_i + u_i $, and random assignment guarantees $ E(u \mid D) = 0 $.

The book’s simulated tutoring experiment has 600 students, with a true effect of +5.0 points. The first thing to report is a balance table of characteristics measured before the programme began.

Pre-treatment variable	Treated	Control	Difference	p-value
Baseline score	56.976	56.777	0.199	0.839
Share female	0.478	0.443	0.035	0.386
Share on free school meals	0.298	0.328	−0.030	0.436
Students	295	305

No difference is anywhere near significant: this is what successful randomisation looks like. Without controls the estimated effect is $ 62.726-56.762 = 5.964 $ points (robust standard error 0.980). Adding the three baseline variables moves it only to 5.621, but the standard error falls to 0.692, and the 95% interval narrows from [4.04, 7.88] to [4.26, 6.98]. The controls do not remove bias, because there is none to remove. They soak up variation in the outcome, which is worth about $ (0.980/0.692)^2 \approx 2.0 $ times as many students.

Exam tipControls in an experiment must be measured before treatment. A variable the treatment itself may change, such as attendance at revision sessions, is a “bad control”: conditioning on it brings back the selection that randomisation removed.

Real experiments are messier. With non-compliance, comparing by assignment gives the intention-to-treat (ITT) effect of the offer; if no controls are treated, the effect on participants is ITT divided by the take-up rate, the IV estimator of Step 13 with the coin as instrument. Attrition related to treatment, spillovers to the control group and limited external validity are the other main problems.

Difference-in-differences

Most policies are not randomised. A state raises its minimum wage and a neighbouring state does not. If you observe both before and after, you can use the change in the untouched state as the counterfactual change for the treated one. The book’s file minwage_did.csv (simulated, in the spirit of Card and Krueger’s 1994 study of New Jersey and Pennsylvania) records FTE employment at 400 fast-food restaurants, 220 in the treated state and 180 in the control state, each observed once before and once after the rise. The true effect built into the data is +2.6 FTE.

	Before	After	Change
Treated state	17.998	18.902	+0.905
Control state	19.540	17.694	−1.846
Difference (treated − control)	−1.542	+1.208	+2.751

Mean FTE employment per restaurant. Changes and the DiD are computed from unrounded means.

Worked example

Compute the difference-in-differences estimate from the 2×2 table in two ways, and find the treated state’s counterfactual end point.

Step 1Across the rows: the treated state rose by 0.905, the control state fell by 1.846, so the treated state did $ 0.905-(-1.846) = 2.751 $ FTE better.

Step 2Down the columns: before the rise the treated restaurants were 1.542 smaller; afterwards they were 1.208 larger, a swing of $ 1.208-(-1.542) = 2.750 $, the same up to rounding.

Step 3Counterfactual: under parallel trends the treated state would have fallen by the control state’s 1.846, ending at $ 17.998-1.846 = 16.152 $.

Step 4Effect: actual minus counterfactual, $ 18.902-16.152 = 2.750 $ FTE per restaurant.

The first difference removes any fixed gap between the states; the second removes anything that changed in the same way for both, such as a regional recession.

Difference-in-differences: mean employment per restaurant FTE

Treated stateControl stateTreated, if it had followed the control trend

Figure 1. The dashed line is the treated state’s path under parallel trends: it falls by the same 1.846 FTE as the control state, to 16.152. The treated state actually ended at 18.902, so the estimated effect is 18.902 − 16.152 = 2.75 FTE. Data: minwage_did.csv (simulated), from Econometrics Made Simple, Chapter 16.

The regression form

The same number comes from one regression on all 800 observations:

\[ y_{it} = \beta_0 + \beta_1\,\text{treated}_i + \beta_2\,\text{post}_t + \beta_3\,(\text{treated}_i \times \text{post}_t) + u_{it} \]

Plug in the four combinations of the dummies. Control before is $ \beta_0 $; control after is $ \beta_0 + \beta_2 $; treated before is $ \beta_0 + \beta_1 $; treated after is $ \beta_0 + \beta_1 + \beta_2 + \beta_3 $. So $ \beta_1 $ is the pre-existing gap, $ \beta_2 $ the common change and $ \beta_3 $ the DiD. The book’s output gives $ \hat\beta_0 = 19.540 $, $ \hat\beta_1 = -1.542 $, $ \hat\beta_2 = -1.846 $ and $ \hat\beta_3 = 2.751 $: exactly the table. If you need a refresher on interaction terms, see Step 9 and our post on dummy variables.

The regression has three advantages over the table: it gives a standard error, it allows controls, and it extends to many groups and periods. Adding chain dummies leaves the estimate at 2.751, because each restaurant belongs to the same chain in both periods and so chain cannot explain any part of the change. Store fixed effects, as in Step 15, give the same 2.751: with two periods they are the same as regressing each store’s change in employment on treated.

Clustered standard errors

The 800 observations are 400 restaurants observed twice, and a restaurant that is large before the rise is large after it, so its two errors are correlated. Standard errors should be clustered by store. Here the usual OLS standard error on $ \hat\beta_3 $ is 0.977 while the clustered one is 0.493. Clustering usually raises standard errors; here it halves them, because each restaurant’s persistent size cancels out of the DiD but the usual formula treats it as fresh noise. With $ t = 2.751/0.493 = 5.58 $, the effect is clearly significant, and the clustered 95% interval [1.78, 3.72] contains the true 2.6.

Formula: DiD regression$ \hat\beta_3 $ in $ y_{it} = \beta_0 + \beta_1\,\text{treated}_i + \beta_2\,\text{post}_t + \beta_3(\text{treated}_i \times \text{post}_t) + \text{controls} + u_{it} $ is the DiD estimate of the effect on the treated. Cluster standard errors by the unit observed repeatedly, or by the level at which the policy was assigned if that is coarser.

Parallel trends and threats to DiD

DiD uses the control group’s change as the treated group’s counterfactual change. The assumption is about a counterfactual, so it can never be tested directly, but you can and should make it plausible.

Key assumption: parallel trendsIn the absence of the policy, average outcomes in the treated and control groups would have changed by the same amount. The groups may differ in level; they must not differ in trend.

Show the pre-treatment trends. With several periods before the policy, plot both groups’ means. Years of parallel movement make it believable that they would have continued. The book’s data have only one “before” period, so this check is impossible: a real weakness of any two-period DiD.
Choose a similar control group. Neighbouring regions share weather, customers and the regional economy.
Run placebo tests. Pretend the policy happened a year earlier using only pre-policy data, or use an outcome the policy cannot affect. The “effect” should be zero.
Check that the policy bit. A minimum-wage rise should raise starting wages in the treated state.

The main threats are different underlying trends, other changes at the same time (a tax cut in the same year), Ashenfelter’s dip (people join training after a bad spell, so their earnings would have recovered anyway), anticipation and spillovers, and changing composition. With many periods, an event study replaces the single interaction with one interaction per period relative to the policy date. The coefficients before the policy, the “leads”, are a check on parallel trends: the policy cannot have had an effect before it happened. Our full post, Difference-in-Differences (DiD) Explained, covers event studies and staggered adoption in more depth.

Practise DiD tables, parallel-trends arguments and RD estimates until they are automatic. Over 1,150 practice questions, every one with a worked answer.Practice book · $9

Regression discontinuity

Many programmes are allocated by a rule with a threshold: a scholarship for an exam score of 70 or more, a pension at 65. Students who score 69.8 and 70.2 are, for practical purposes, the same kind of student, yet one gets the scholarship and the other does not. Comparing them just around the threshold estimates the effect of the scholarship.

Key termIn a sharp regression discontinuity, treatment is fully determined by whether a running variable $ x $ is above a known cut-off $ c $. If everything else that affects $ y $ changes smoothly with $ x $ (the continuity assumption), any jump in the average outcome at $ c $ is the effect of the treatment for units at the cut-off.

The book’s file scholarship_rd.csv (simulated) has 2,500 first-year students. Everyone scoring 70 or more on the entrance exam gets a scholarship; the outcome is first-year GPA, and the true jump at 70 is +0.25. A naive comparison of all 680 scholarship holders with the 1,820 others gives $ 3.098-2.027 = 1.071 $ GPA points: hopelessly biased, because stronger students earn higher grades with or without a scholarship.

Local linear estimation

Centre the running variable, $ \tilde x = \text{exam}-70 $, keep students within a bandwidth $ h $ of the cut-off, and fit a separate straight line on each side:

\[ \text{gpa}_i = \alpha + \tau D_i + \gamma_1 \tilde x_i + \gamma_2 (D_i \times \tilde x_i) + u_i, \qquad |\tilde x_i| \le h \]

Below the cut-off the fitted line is $ \alpha + \gamma_1 \tilde x $; above it, $ (\alpha + \tau) + (\gamma_1 + \gamma_2)\tilde x $. At $ \tilde x = 0 $ the lines differ by exactly $ \tau $. Centring is what makes $ \tau $ the jump at the cut-off. With $ h = 10 $ (1,225 students), $ \hat\tau = 0.264 $ with a robust standard error of 0.043, so $ t = 6.09 $, and the 95% interval [0.179, 0.349] contains the true 0.25.

Regression discontinuity at the scholarship cut-off GPA (0 to 4)

No scholarship (score below 70)Scholarship (70 or more)

Figure 2. Dots are average GPA in 2-point bins of the exam score. The lines are separate straight-line fits within 10 points of the cut-off (1,225 students). At 70 the fitted lines are 0.264 GPA points apart: the estimated effect of the scholarship for students near the cut-off. Data: scholarship_rd.csv (simulated), Chapter 16.

Bandwidth, manipulation and covariates

The bandwidth trades bias against variance: a narrow window uses comparable students but few of them; a wide one uses more, but a straight line far from 70 can miss the curve.

Bandwidth (exam points)	5	10	15	20	50 (all)
Students used	641	1,225	1,683	2,038	2,500
Estimated jump	0.258	0.264	0.235	0.220	0.214
Robust standard error	0.062	0.043	0.037	0.033	0.028

The standard error falls steadily as the window widens, and the estimate drifts down because the true relationship is curved. Every estimate is clearly positive, which is reassuring: a result that survives reasonable bandwidths is far more credible than one that appears at a single bandwidth. In practice, report a data-driven bandwidth (the rdrobust package computes one) alongside a table like this.

Continuity fails if people can manipulate the running variable, for instance by asking for a re-mark when they score 68. The tell-tale sign is bunching just above the cut-off. The book counts 131 students between 68 and 70 and 111 between 70 and 72; a binomial test of 111 out of 242 gives $ p = 0.222 $, so there is no sign of bunching. Pre-determined characteristics should not jump either: the estimated jumps are 0.329 ($000) in parents’ income ($ p = 0.849 $) and 0.066 in the share female ($ p = 0.255 $). This is the RD version of a balance table.

Common mistakeReading the RD estimate as the effect on all students. It is the effect for students near a score of 70. A scholarship may matter far more for a student scoring 50 who would otherwise work long hours, or far less for one scoring 95. RD gives no direct evidence about them.

Choosing a design

Two further designs complete the toolkit: matching on the propensity score, which relies on selection on observables, and synthetic control, which builds a weighted comparison unit for a single treated unit.

Design	Counterfactual comes from	Key assumption	How to support it
RCT	the randomly assigned control group	randomisation done properly; little attrition; no spillovers	balance table; attrition by group; randomise clusters
DiD	the change in a comparison group	parallel trends	pre-trend plots; event-study leads; placebo tests
RD	units just across the cut-off	continuity; no manipulation	density test; covariate jumps; several bandwidths
Matching	untreated units with the same observed $ x $	selection on observables	rich pre-treatment data; common support
Synthetic control	a weighted average of untreated units	pre-period weights still fit after the policy	close pre-period fit; placebo runs

A practical order of preference is: an experiment if you can run one; a natural experiment (DiD or RD) if the policy created one; matching or regression with controls only when nothing better exists, with an honest discussion of what might be unobserved.

Exam tipExaminers almost always ask “what is the identifying assumption, and how would you check it?” Name it precisely (“parallel trends”, not “the groups are similar”) and give a concrete check (“plot both groups’ mean outcomes for the five years before the policy”).

Practise: 6 questions

Try each one on paper first, then check the worked answer.

EasyQ1. Six workers have potential earnings (in $ per hour) without and with a training course: P 10 and 16, Q 12 and 17, R 14 and 18, S 15 and 17, T 18 and 20, U 21 and 22. Only P, Q and R take the course. Find the ATE and the ATT. Show worked answer

Individual effects: P 6, Q 5, R 4, S 2, T 2, U 1.
ATE $ = (6+5+4+2+2+1)/6 = 20/6 \approx 3.33 $ dollars an hour.
ATT $ = (6+5+4)/3 = 5.00 $ dollars an hour.
The ATT is larger because the workers who gain most are the ones who take the course.

EasyQ2. In a city that opened bus lanes, the average commute fell from 42 to 38 minutes. In a similar city without bus lanes it fell from 40 to 39 minutes. Find the DiD estimate and state the assumption behind it. Show worked answer

DiD $ = (38-42)-(39-40) = -4-(-1) = -3 $ minutes.
Assumption: parallel trends. Without the bus lanes, commutes in the first city would have fallen by the same 1 minute as in the second.

MediumQ3. A school randomly offers tutoring to half its students. Those offered average 64.2 in the final exam and the rest 61.8. Only 75% of those offered attend, and nobody in the control group finds a tutor. (a) Find the intention-to-treat effect. (b) Estimate the effect of tutoring on those who attend. (c) Why not simply compare attenders with non-attenders? Show worked answer

(a) ITT $ = 64.2-61.8 = 2.4 $ points: the effect of the offer.
(b) The whole ITT comes from the 75% who attend, so the effect on attenders is $ 2.4/0.75 = 3.2 $ points.
(c) Attending is a choice. Students who turn up are likely to be more motivated, so comparing them with non-attenders brings back selection bias. Comparing by random assignment keeps the groups comparable; dividing by take-up is the IV estimator with the offer as the instrument.

MediumQ4. A DiD regression gives $ \hat y = 30.0 + 4.0\,\text{treated}-2.5\,\text{post} + 3.2\,(\text{treated} \times \text{post}) $. Find the four group means, the treated group’s counterfactual mean after the policy, and the estimated effect. Show worked answer

Control before $ = 30.0 $; control after $ = 30.0-2.5 = 27.5 $.
Treated before $ = 30.0 + 4.0 = 34.0 $; treated after $ = 34.0-2.5 + 3.2 = 34.7 $.
Counterfactual: the treated group falls by the common 2.5, to $ 31.5 $.
Effect $ = 34.7-31.5 = 3.2 $, the interaction coefficient, as it must be.

MediumQ5. Students scoring below 40 on a test must attend summer school. With $ \tilde x = \text{score}-40 $, local linear fits of next year’s score give $ 52.0 + 0.80\tilde x $ below the cut-off and $ 49.5 + 0.75\tilde x $ at or above it. (a) Estimate the effect of summer school at the cut-off. (b) Predict next year’s score for a student who scored 38. (c) Name two checks you would run. Show worked answer

(a) The treated side is below the cut-off here, so the effect is the treated line minus the untreated line at $ \tilde x = 0 $: $ 52.0-49.5 = +2.5 $ points.
(b) $ \tilde x = -2 $: $ 52.0 + 0.80 \times (-2) = 50.4 $.
(c) A histogram or density test for bunching just below 40 (students deliberately scoring low, or teachers marking down), and a check that pre-determined characteristics such as prior attainment do not jump at 40. Also report the estimate at several bandwidths.

ExamQ6. In 2018 one region introduced free meals in all its primary schools; a neighbouring region did not. You have school-level attendance rates for 2014–2021. (a) Write a DiD regression and say which coefficient measures the effect. [2] (b) Mean attendance was 93.1% in 2017 and 94.6% in 2019 in the treated region, and 92.4% and 92.7% in the control region. Compute the DiD using these two years. [2] (c) State the identifying assumption and describe two ways to support it with these data. [3] (d) Explain why the standard errors should be clustered. [1] Show worked answer

(a) $ \text{att}_{st} = \beta_0 + \beta_1\,\text{treated}_s + \beta_2\,\text{post}_t + \beta_3(\text{treated}_s \times \text{post}_t) + u_{st} $, with post = 1 from 2018. [1] $ \beta_3 $ is the effect on the treated region. [1]
(b) $ (94.6-93.1)-(92.7-92.4) = 1.5-0.3 = 1.2 $ percentage points. [M1 A1]
(c) Parallel trends: without free meals, attendance in the two regions would have changed by the same amount. [1] Plot both regions’ mean attendance for 2014–2017 and check that they moved in parallel, or estimate an event study and check that the leads are close to zero. [1] Run a placebo test, pretending the policy began in 2016 and using only 2014–2017 data; the estimate should be about zero. [1]
(d) Each school is observed in many years, so its errors are correlated over time; cluster by school (or by region-level policy assignment if enough clusters exist). [1]
Total: 8 marks.

Check yourself

1. A naive comparison of treated and untreated groups equals

Naive difference $ = $ ATT $ + \; E[Y(0) \mid D=1]-E[Y(0) \mid D=0] $. Randomisation makes the second term zero.

2. The identifying assumption of difference-in-differences is that

Parallel trends allows a gap in levels (here 1.542 FTE before the rise) but requires the same trend.

3. A sharp RD estimate of 0.264 at an exam cut-off of 70 tells you the effect of the scholarship for

RD is local: continuity at the cut-off identifies the effect only for units near it.

Your score: 0 out of 3

The short version

Potential outcomes $ Y(1) $ and $ Y(0) $; only one is seen. We estimate averages: the ATE or the ATT.
Naive difference $ = $ ATT $ + $ selection bias. In the eight-student example, a true ATT of 6.50 appeared as $ -9.00 $ because of a selection bias of $ -15.50 $.
Randomisation sets selection bias to zero. Baseline controls improve precision (standard error 0.980 to 0.692), not bias.
DiD: $ (\bar y_{T,\text{after}}-\bar y_{T,\text{before}})-(\bar y_{C,\text{after}}-\bar y_{C,\text{before}}) $, or $ \hat\beta_3 $ on treated × post. Minimum-wage data: 2.751 FTE, clustered standard error 0.493. Assumption: parallel trends.
Sharp RD: local linear fits either side of the cut-off; the jump is the effect at the cut-off. Scholarship data: 0.264 (bandwidth 10), between 0.214 and 0.264 across bandwidths, no sign of manipulation ($ p = 0.222 $).
Chapter 16 of Econometrics Made Simple covers all of this in full, with Python, R and Stata code for every estimate. If you study more than one subject, the Complete Book Bundle includes every Economics, Statistics, Mathematics and Econometrics book.

Policy evaluation, panels, IV and time series in one place. Get the textbook and the practice book together for $19 (was $24).Both books · $19