Surreal illustration showing the transition from traditional A/B testing to innovative synthetic control designs.

Beyond A/B Testing: Revolutionizing Experimental Design with Synthetic Controls

Kai Mendoza in Tech & Innovation August 2025 • 4 min read.

"Discover how synthetic control designs are transforming experimental practices, offering a more robust approach than traditional A/B tests in settings with limited, aggregate data."

In the world of data-driven decision-making, A/B testing has long been the gold standard for evaluating the impact of changes. Whether it's a tweak to a website layout or a new marketing campaign, A/B tests provide a seemingly straightforward way to determine what works best. However, this approach often falls short when dealing with complex, real-world scenarios, particularly when experimental units are large and aggregate.

Imagine a ride-sharing company deciding between two different compensation plans for its drivers. Randomly assigning individual drivers within the same city to different pay structures can create equity issues and even skew the results due to drivers competing for the same riders. The limitations of A/B testing become even more apparent when you can only implement a change in a limited number of locations or markets. Randomization can lead to treated and control groups that are vastly different from each other, making it difficult to draw reliable conclusions.

That's where synthetic control designs come in. This innovative approach offers a compelling alternative to traditional A/B testing in situations where experimental units are aggregate entities, such as markets or cities, and the number of units that can be subjected to the treatment is limited. Synthetic controls provide a way to create a counterfactual scenario, allowing you to estimate the impact of an intervention with greater accuracy and less bias.

What are Synthetic Control Designs and How Do They Work?

Surreal illustration showing the transition from traditional A/B testing to innovative synthetic control designs.

Synthetic control designs are not about randomly assigning treatments; instead, they involve carefully selecting which units will be treated and constructing a "synthetic" control group from the remaining untreated units. This synthetic control is created by weighting the untreated units in such a way that they closely resemble the treated units in terms of pre-intervention characteristics. This approach leverages the strengths of the synthetic control method, initially developed for observational studies, and adapts them for experimental settings.

The core idea is to estimate average potential outcomes by calculating weighted averages. For potential outcomes with treatment, the outcomes of treated units are used to derive the average, applying specific weights. Similarly, potential outcomes without treatment use a weighted average of outcomes from control units. This careful weighting process is intended to mitigate selection bias, a common issue in experimental designs with aggregate units.

Selecting Treated Units: Identify the units that are most representative of the overall population you're interested in.
Creating the Synthetic Control: Construct a weighted combination of untreated units that closely matches the pre-intervention characteristics of the treated units.
Estimating Treatment Effects: Compare the post-intervention outcomes of the treated units to the projected outcomes based on the synthetic control.

The goal is to ensure that the treated units aren't so idiosyncratic that they cannot be approximated by untreated units. Average potential outcomes are then estimated using weighted averages, which allows for fairer comparisons. This enables a more reliable evaluation of the treatment's impact compared to traditional randomized experiments in aggregate settings.

The Future of Experimentation: Synthetic Controls and Beyond

While A/B testing will likely remain a valuable tool, synthetic control designs offer a powerful and versatile approach for experimental design. As data science continues to evolve, methodologies that adapt and refine experimental approaches to meet real-world complexities will be highly valuable. From corporate researchers to policy makers, synthetic control designs provide a crucial tool for drawing accurate conclusions and making informed decisions, even when sample sizes are limited.

About this Article -

This article was crafted using a human-AI hybrid and collaborative approach. AI assisted our team with initial drafting, research insights, identifying key questions, and image generation. Our human editors guided topic selection, defined the angle, structured the content, ensured factual accuracy and relevance, refined the tone, and conducted thorough editing to deliver helpful, high-quality information.See our About page for more information.

This article is based on research published under:

DOI-LINK: https://doi.org/10.48550/arXiv.2108.02196,

Title: Synthetic Controls For Experimental Design

Subject: stat.me econ.em

Authors: Alberto Abadie, Jinglong Zhao

Published: 04-08-2021

Everything You Need To Know

What are the core limitations of A/B testing in real-world scenarios, and why do Synthetic Control Designs offer a better alternative?

A/B testing, while seemingly straightforward, struggles with complex scenarios, especially when experimental units are large aggregates such as markets or cities. This is because A/B tests rely on random assignment, which can lead to equity issues and skewed results. In contrast, Synthetic Control Designs excel in these situations by creating a "synthetic" control group from untreated units. This synthetic group is carefully constructed to resemble the treated units based on pre-intervention characteristics, reducing bias and improving the reliability of the results when working with aggregate data.

How do Synthetic Control Designs work, and what are the key steps involved in their implementation?

Synthetic Control Designs don't rely on random treatment assignment. Instead, they select treated units representing the population and build a synthetic control group. The synthetic control is created by weighting the untreated units so they mirror the pre-intervention characteristics of the treated units. The main steps involve: (1) Selecting Treated Units: Choose the units most representative of your population. (2) Creating the Synthetic Control: Develop a weighted combination of untreated units to match the treated units' pre-intervention characteristics. (3) Estimating Treatment Effects: Compare post-intervention outcomes of treated units with the projected outcomes from the synthetic control. This method, by calculating weighted averages, mitigates selection bias.

What is the purpose of creating a 'synthetic' control group in Synthetic Control Designs, and how does it enhance experimental results?

The synthetic control group is built to mimic the treated units in terms of pre-intervention characteristics. This is achieved by assigning weights to untreated units, allowing for a comparison that is as fair as possible. By constructing a synthetic control, researchers can estimate what would have happened to the treated units if they had not received the intervention. This approach reduces biases that can arise from simple comparisons, providing a more reliable evaluation of the treatment's impact, especially with aggregate units.

In what specific contexts are Synthetic Control Designs particularly advantageous compared to A/B testing, and what kind of decisions can they help inform?

Synthetic Control Designs are most beneficial when dealing with aggregate experimental units, like markets or cities, or when the number of units that can be treated is limited. For example, a ride-sharing company testing different compensation plans across cities would find Synthetic Control Designs advantageous. They allow researchers and decision-makers, from corporate researchers to policy makers, to draw accurate conclusions and make informed decisions, even when sample sizes are restricted. Synthetic Control Designs offer a better way to measure impact where randomization of individuals isn't possible or desirable.

How does the concept of 'average potential outcomes' relate to the efficacy of Synthetic Control Designs, and why is it crucial for reliable experimental results?

In Synthetic Control Designs, estimating average potential outcomes is central to their methodology. This involves calculating weighted averages to determine the expected outcomes both with and without treatment. For potential outcomes with treatment, the outcomes of treated units are used to derive the average, applying specific weights, while for potential outcomes without treatment, a weighted average of outcomes from control units is applied. This technique helps to reduce selection bias. By using weighted averages and constructing a synthetic control, Synthetic Control Designs provide a more reliable evaluation of the treatment's impact compared to traditional randomized experiments in aggregate settings, leading to more accurate and dependable results.