Parallel Trends: How to Trust Difference-in-Differences Analysis in Policy Evaluation
"Uncover the pitfalls of relying on parallel trends assumptions with covariates in policy analysis and explore robust alternative strategies to ensure accurate results."
In the world of policy evaluation, the difference-in-differences (DiD) method stands out as a popular tool. It's used to measure the impact of a specific intervention or treatment by comparing changes in outcomes between a treated group and a control group over time. But what happens when we need to account for other factors that might influence these outcomes? This is where covariates come in. Covariates are additional variables that researchers include to ensure a more accurate and reliable analysis.
Imagine you're evaluating a new education policy aimed at improving student test scores. Simply comparing the change in scores before and after the policy in schools that adopted it versus those that didn't might not tell the whole story. Factors like the socioeconomic status of students, prior educational resources, or even regional differences could also play a significant role. To address this, researchers often include covariates in their DiD analysis to isolate the true effect of the policy.
However, including covariates isn't always straightforward. The underlying assumption in DiD analysis is that, without the intervention, the treated and control groups would have followed parallel trends. But what if these trends are only parallel after accounting for the covariates? In a recent research paper, economists Carolina Caetano and Brantly Callaway shed light on the challenges and potential pitfalls of DiD analysis when parallel trends are conditional on covariates, offering alternative strategies for more trustworthy policy evaluations.
The Hidden Weaknesses of TWFE Regressions
One of the most common ways to implement DiD with covariates is through two-way fixed effects (TWFE) regressions. TWFE is a statistical technique that allows researchers to control for time-invariant differences between groups (fixed effects) and time-specific shocks that affect all groups equally. It's widely used because it's relatively easy to implement and interpret.
- Linearity Assumptions: TWFE regressions assume a linear relationship between the covariates and the outcome variable. If this assumption is violated, the results can be biased.
- Time-Varying Covariates: When covariates change over time, TWFE regressions may not fully capture their impact. The transformation used to eliminate fixed effects can also eliminate important information about the level of the covariates.
- Time-Invariant Covariates: TWFE regressions typically absorb time-invariant covariates into the fixed effects, meaning their influence isn't directly estimated. This can be problematic if these time-invariant factors are important confounders.
Alternative Estimation Strategies
Given these challenges, Caetano and Callaway propose alternative estimation strategies that can circumvent the limitations of TWFE regressions. These strategies build on recent developments in the DiD literature, particularly augmented inverse propensity score weighting (AIPW) estimators. AIPW estimators combine outcome regression models with propensity score models, offering a more robust approach to causal inference. These models do not suffer from the hidden linearity bias.