Synthetic Control Methods for Evaluating Trading Strategies

This series has spent a long time on one route to a false edge: selection. The Deflated Sharpe Ratio prices the winner of a search. The Probability of Backtest Overfitting prices the search itself. The honest negative is what happens when you point both at a strategy you actually wanted to trade. That apparatus answers one question well: did I find this because I looked too many times?

It does not touch a second, entirely separate route: confounding. You changed one thing — deployed a new execution algorithm, switched a parameter, reacted to a regime — and performance improved. There was no search. There was no multiple-testing problem. And the improvement still may have nothing to do with you, because volatility doubled the week you shipped. No amount of deflation catches this, because deflation corrects for trials and you only ran one.

The Synthetic Control Method (SCM), developed by Abadie and Gardeazabal (2003) and refined by Abadie, Diamond, and Hainmueller (2010, 2015), attacks confounding directly. It constructs a counterfactual — a weighted combination of instruments you did not touch, chosen so that it tracked the treated instrument closely before the intervention. The post-intervention divergence between the two is your effect estimate. Crucially, SCM comes with a falsification criterion that tells you when it cannot produce a credible counterfactual, and a permutation-based p-value that does not require you to believe any asymptotic story.

Why the Obvious Alternatives Fail

Before-after comparison confounds the intervention with everything else that changed on that date. Comparison against a single benchmark instrument fails because no single asset tracks yours — this is the Single Instrument Trap from the other direction: if one instrument's idiosyncratic structure can fake an edge, one instrument's idiosyncratic structure can also fake a control. SCM's donor pool is the formal answer to exactly that objection: instead of picking one comparable asset, you let the pre-period data choose a weighted basket of them.

Difference-in-differences deserves more than a sentence, because it is the near-miss. DiD compares the change in the treated unit to the change in a control group, and it identifies the effect only under parallel trends: absent the intervention, treated and control would have moved together. In financial markets that assumption is close to indefensible — assets differ in beta, in volatility regime, in liquidity tier, and their relative paths wander. SCM replaces the assumption with a fitted object: rather than assuming a control that moves in parallel, it constructs one and then shows you how well it actually tracked. If it tracked badly, you are told to stop.

The Method

Setup and Notation

Consider $J + 1$ units (assets, strategies, or portfolios) observed over $T$ periods. Unit 1 is the treated unit — the asset or strategy where you intervened. Units $2, \ldots, J+1$ form the donor pool of comparable untreated units.

Let $Y_{1t}^I$ denote the outcome for the treated unit at time $t$ with the intervention, and $Y_{1t}^N$ the outcome without it. The treatment effect at time $t > T_0$ is:

$\tau_{1t} = Y_{1t}^I - Y_{1t}^N$

We observe $Y_{1t}^I$ directly. The whole problem is estimating $Y_{1t}^N$.

Constructing the Synthetic Control

SCM estimates $Y_{1t}^N$ as a weighted average of donor outcomes:

$\hat{Y}_{1t}^N = \sum_{j=2}^{J+1} w_j \, Y_{jt}$

where the weight vector $\mathbf{w} = (w_2, \ldots, w_{J+1})'$ satisfies $w_j \geq 0$ for all $j$ and $\sum_j w_j = 1$.

These two constraints do real work. They force the synthetic control to be a convex combination of donors, which prevents extrapolation outside the support of the data. An unconstrained regression can hand you a counterfactual built from a 3.4x long position in one coin against a 2.4x short in another — numerically a great pre-period fit, and a counterfactual that corresponds to no portfolio anyone could hold. The convexity constraint makes the counterfactual interpretable and the weights sparse: typically only a handful of donors get non-zero weight, and you can read them.

Optimal Weight Selection

Weights are chosen to minimize the distance between the treated unit and the synthetic control in the pre-intervention period:

$\min_{\mathbf{w}} \| \mathbf{X}_1 - \mathbf{X}_0 \mathbf{w} \|_V = \min_{\mathbf{w}} (\mathbf{X}_1 - \mathbf{X}_0 \mathbf{w})' \mathbf{V} (\mathbf{X}_1 - \mathbf{X}_0 \mathbf{w})$

where $\mathbf{X}_1$ is a $(k \times 1)$ vector of pre-intervention characteristics for the treated unit, $\mathbf{X}_0$ is the $(k \times J)$ matrix of the same characteristics for the donor pool, and $\mathbf{V}$ is a $(k \times k)$ positive semi-definite diagonal matrix assigning importance to each predictor.

$\mathbf{V}$ is chosen by cross-validation: pick $\mathbf{V}$ such that the resulting $\mathbf{w}^*(\mathbf{V})$ minimizes the mean squared prediction error (MSPE) of the outcome variable in the pre-intervention period. This is a nested optimization — an outer loop over $\mathbf{V}$, an inner constrained solve for $\mathbf{w}$.

Predictor Variables for Trading Applications

Where Abadie et al. used GDP per capita and industry composition, trading applications want:

• Return characteristics: mean return, volatility, skewness, kurtosis over the pre-period
• Liquidity measures: average spread, daily volume, Amihud illiquidity ratio
• Microstructure features: order flow imbalance, trade arrival rate, quote-to-trade ratio
• Factor exposures: beta to market, momentum, value, and volatility factors
• Lagged outcomes: cumulative PnL values at several pre-intervention dates

Lagged outcomes matter most. Abadie et al. (2010) show that if the synthetic control matches the treated unit on lagged outcomes over a long pre-intervention window, it implicitly controls for unobserved confounders under a linear factor model — which is the whole reason the method is worth the trouble. You are not required to name the confounder that drove your PnL. You are required to find donors that were exposed to it too.

Where This Applies

Scenario 1: Evaluating a New Execution Algorithm

You deploy a new TWAP schedule on one instrument and want its effect on implementation shortfall. Both the metric and the algorithm family are covered in depth elsewhere — see implementation shortfall and TCA for the cost decomposition and TWAP/VWAP/POV algorithms for the schedulers themselves. The SCM contribution is the evaluation layer on top: the donor pool is the same instruments still running the old scheduler, over the same window, so venue-wide changes in spread or volume land on treated and donors alike.

Pre-intervention period: 60 trading days before the switch. Post-intervention period: 30 trading days after. The effect estimate is the gap:

$\hat{\tau}_{1t} = Y_{1t}^{\text{actual}} - \hat{Y}_{1t}^{\text{synthetic}}, \quad t > T_0$

If the new scheduler genuinely reduces cost, $\hat{\tau}_{1t} < 0$ persistently after $T_0$ — and, more importantly, the treated gap sits outside the cloud of donor gaps.

Scenario 2: Impact of a Market Event

A fee-schedule change, tick-size pilot, transaction tax, or circuit-breaker rule change hits one venue. Treated unit: your strategy on the affected venue. Donor pool: the same pair on venues the change did not touch. Fee-structure effects on execution are treated separately in maker-taker fees and rebates; SCM is what turns "our fills got worse after the change" into an estimate with a null attached.

Scenario 3: Regime-Conditional Strategy Switching

You switch from mean-reversion to momentum on one asset when a regime-detection signal fires. Did the switch beat staying put? Donor pool: similar assets where the signal fired but you did not switch. Note the trap here — if the regime signal fires market-wide, your donors are treated too, and the design collapses. See the spillover rule below.

Placebo Tests and Inference

SCM's inference is permutation-based. The general case for building an empirical null by resampling rather than trusting a closed-form distribution is made in Monte Carlo and bootstrap methods for backtests; what follows is the SCM-specific construction.

In-Time Placebo

Shift the intervention date backwards into the pre-period — halfway, say — and re-run. If SCM finds a "treatment effect" at a date where nothing happened, it is picking up pre-existing divergence, not your intervention. A credible synthetic control shows no effect at false intervention dates.

In-Space Placebo (Permutation Test)

Apply the entire SCM procedure to every donor, pretending each in turn was the treated unit:

1. For each donor $j$, construct a synthetic control from the remaining donors plus the original treated unit.
2. Compute the post-intervention gap $\text{gap}_{jt} = Y_{jt} - \hat{Y}_{jt}^{\text{synth}}$.
3. Compare the treated unit's gap to the distribution of placebo gaps.

The p-value is a rank:

$p = \frac{\text{rank of treated unit's post/pre MSPE ratio}}{J + 1}$

where

$\text{MSPE ratio} = \frac{\text{MSPE}_{\text{post}}}{\text{MSPE}_{\text{pre}}} = \frac{\frac{1}{T - T_0} \sum_{t > T_0} (Y_{1t} - \hat{Y}_{1t}^N)^2}{\frac{1}{T_0} \sum_{t \leq T_0} (Y_{1t} - \hat{Y}_{1t}^N)^2}$

The ratio, rather than the raw post-period gap, is the right statistic: a donor that was never well fit will show a large post gap for reasons that have nothing to do with any intervention, and dividing by its pre-period MSPE penalizes exactly that.

This is the same inferential move the Deflated Sharpe Ratio makes — rank your candidate inside an empirically generated reference distribution, never against zero. DSR ranks your winner against a best-of-N null over trials; SCM ranks your treated unit against a null over units. Different estimator, same discipline, and the same warning applies: with $J = 15$ donors the smallest attainable p-value is $1/16 \approx 0.06$. A permutation test cannot resolve past its own granularity, so a small donor pool caps how much significance you are entitled to claim, no matter how large the effect looks.

Practical Filtering

Abadie et al. recommend excluding placebo units with poor pre-intervention fit — pre-MSPE more than $k$ times that of the treated unit, with $k$ typically 2 to 20. Poor-fit placebos have tiny denominators and produce enormous MSPE ratios, contaminating the reference distribution and making your treated unit look unremarkable. Report the ratio at several thresholds; if your p-value is only significant at one particular $k$, you have found a tuning parameter, not an effect.

Implementation

The code below uses only numpy and scipy so the mechanics are visible. Production alternatives — SparseSC (Microsoft Research), pysyncon, SyntheticControlMethods — handle edge cases and scale better.

Predictor Matrix Construction

import numpy as np
import pandas as pd
from scipy.optimize import minimize


def build_predictors(series: pd.Series, T0: int) -> np.ndarray:
    """
    Extract predictor vector from pre-intervention data.
    Features: mean return, volatility, skewness, kurtosis,
              plus lagged outcome values at regular intervals.
    """
    pre = series.iloc[:T0]
    daily_ret = pre.diff().dropna()

    stats = [
        daily_ret.mean(),
        daily_ret.std(),
        daily_ret.skew(),
        daily_ret.kurtosis(),
    ]

    lag_indices = np.linspace(0, T0 - 1, 5, dtype=int)
    lags = pre.iloc[lag_indices].values.tolist()

    return np.array(stats + lags)

df here is a panel of cumulative returns — treated unit in column 0, donors in the rest, one row per bar. Use cumulative returns, not price levels.

X1 = build_predictors(df["Treated"], T0).reshape(-1, 1)   # (k x 1)
X0 = np.column_stack([                                    # (k x J)
    build_predictors(df[col], T0) for col in df.columns[1:]
])

Weight Optimization

def solve_weights(
    X1: np.ndarray,
    X0: np.ndarray,
    Y1_pre: np.ndarray,
    Y0_pre: np.ndarray,
) -> np.ndarray:
    """
    Solve for optimal synthetic control weights.

    Nested optimization:
      Outer: optimize V (predictor importance)
      Inner: optimize w (donor weights) given V
    """
    k = X1.shape[0]
    J = X0.shape[1]

    def solve_w_given_V(V_diag: np.ndarray) -> np.ndarray:
        """Inner: find w minimizing weighted predictor distance."""
        V = np.diag(V_diag)

        def objective(w):
            gap = X1.flatten() - X0 @ w
            return gap @ V @ gap

        constraints = {"type": "eq", "fun": lambda w: np.sum(w) - 1.0}
        bounds = [(0, 1)] * J
        w0 = np.ones(J) / J

        result = minimize(objective, w0, method="SLSQP",
                         bounds=bounds, constraints=constraints,
                         options={"maxiter": 1000, "ftol": 1e-12})
        return result.x

    def outer_objective(V_diag: np.ndarray) -> float:
        """Outer: minimize pre-period MSPE given V."""
        w = solve_w_given_V(np.abs(V_diag))
        return np.mean((Y1_pre - Y0_pre @ w) ** 2)

    V0 = np.ones(k)
    result = minimize(outer_objective, V0, method="Nelder-Mead",
                     options={"maxiter": 5000, "xatol": 1e-8})

    return solve_w_given_V(np.abs(result.x))


Y1_pre = df["Treated"].iloc[:T0].values
Y0_pre = df.iloc[:T0, 1:].values
weights = solve_weights(X1, X0, Y1_pre, Y0_pre)

for col, w in zip(df.columns[1:], weights):
    if w > 0.01:
        print(f"  {col}: {w:.4f}")

The gap series is then df["Treated"].values - df.iloc[:, 1:].values @ weights, plotted against the intervention date alongside the donor gaps from the placebo run below.

Placebo Tests

def run_placebo_tests(df: pd.DataFrame, T0: int) -> pd.DataFrame:
    """In-space placebo: treat each unit in turn as if intervened upon."""
    results = []
    all_columns = df.columns.tolist()

    for placebo_col in all_columns:
        donor_cols = [c for c in all_columns if c != placebo_col]
        donor_df = df[donor_cols]

        X1_p = build_predictors(df[placebo_col], T0).reshape(-1, 1)
        X0_p = np.column_stack([
            build_predictors(donor_df[c], T0) for c in donor_cols
        ])

        w_p = solve_weights(
            X1_p, X0_p,
            df[placebo_col].iloc[:T0].values,
            donor_df.iloc[:T0].values,
        )
        gap_p = df[placebo_col].values - donor_df.values @ w_p

        mspe_pre = np.mean(gap_p[:T0] ** 2)
        mspe_post = np.mean(gap_p[T0:] ** 2)

        results.append({
            "unit": placebo_col,
            "mspe_pre": mspe_pre,
            "mspe_post": mspe_post,
            "mspe_ratio": mspe_post / mspe_pre if mspe_pre > 1e-10 else np.inf,
            "gap_series": gap_p,
            "is_treated": placebo_col == "Treated",
        })

    return pd.DataFrame(results)


placebo_results = run_placebo_tests(df, T0)

treated_pre = placebo_results.loc[placebo_results["is_treated"], "mspe_pre"].values[0]
filtered = placebo_results[placebo_results["mspe_pre"] <= 5 * treated_pre]

treated_ratio = filtered.loc[filtered["is_treated"], "mspe_ratio"].values[0]
p_value = (filtered["mspe_ratio"] >= treated_ratio).sum() / len(filtered)
print(f"MSPE ratio (treated): {treated_ratio:.2f}   p = {p_value:.3f}")

Calibrating the Estimator

There is a demonstration worth running and a demonstration worth refusing. Generating synthetic panel data, injecting a +0.3%/day effect, recovering roughly +0.3%/day, and presenting that as evidence proves only that SLSQP converges. It is circular, and this blog does not publish it.

What synthetic data is good for is calibration — the same standard applied in the DSR piece and PBO, where the known null is what makes the statistic trustworthy. Two checks, both run on data generated from a linear factor model with the donor count and pre-period length you actually have:

1. Power at a known effect size. Inject an effect of known magnitude. Does the estimated post-period gap recover it, and within what error? Sweep the effect size down until recovery fails — that floor is the smallest effect your panel geometry can detect, and any real result below it is noise regardless of its p-value.
2. Size under a true null. Inject nothing. Run the full placebo pipeline. The p-value should be approximately uniform on $[0, 1]$, so it should fall below 0.10 about 10% of the time across repeated draws. If it fires far more often than that, your predictor set or filter threshold is manufacturing significance and the pipeline is broken before it ever touches real data.

Check 2 is the one people skip and the one that matters. Run it first.

Practical Considerations

Choosing the Donor Pool

Donors must be plausibly comparable to the treated unit and unaffected by the intervention. For execution work: the same algorithm on similar instruments — same sector, market-cap decile, liquidity tier. The tier taxonomy in multi-symbol validation is directly reusable here; the properties that make instruments a fair robustness test also make them a fair donor pool. For venue events: the same instrument on venues the change did not reach.

Exclude units with spillover. If changing your algorithm on one instrument altered your quoting on correlated instruments, those instruments are partially treated. Leaving them in biases the synthetic control toward the treated unit's post-period path and shrinks your estimate toward zero — a design flaw that manufactures null results just as easily as a bad control manufactures positive ones.

Pre-Intervention Fit Is the Whole Argument

SCM's credibility rests entirely on pre-period fit. Report the pre-intervention MSPE and show the fit. A poor pre-fit invalidates everything after $T_0$.

Abadie et al. (2015) state the rule plainly: if no convex combination of donors reproduces the treated unit's pre-intervention trajectory, do not apply the method. This is an honest falsification criterion, and it is rarer than it should be — most estimators return a number whatever you feed them. SCM tells you when it cannot produce a credible counterfactual, and the correct response to that message is to publish the failure, not to widen the donor pool until the fit improves. Widening the pool to chase pre-fit is the same overfitting the rest of this series is about, relocated to the control-construction step.

Overfitting in Short Panels

The SCM-specific quantity to watch is the ratio of donors $J$ to pre-period observations $T_0$. With $J$ free weights fitted on $T_0$ points, perfect pre-fit becomes achievable by construction as $J$ approaches $T_0$ — and a perfect pre-fit obtained that way carries no information. Keep $T_0$ substantially larger than $J$; if a strategy ran for 30 days before the change and you have 20 candidate donors, you do not have an SCM problem, you have an interpolation. Mitigations:

1. Regularization: SparseSC adds $\ell_1$ and $\ell_2$ penalties to the weight optimization.
2. Restrict the donor pool: fewer, better-justified donors beat more donors — at the cost of a coarser permutation grid, since $J$ also sets your minimum p-value. That tension is real and has no clean resolution.
3. Augmented SCM: Ben-Michael, Feller, and Rothstein (2021) use an outcome model to bias-correct when perfect pre-fit is unattainable.

Non-Stationarity

Run SCM on cumulative returns, never on price levels — levels invite spurious fits. Confirm the pre-intervention gap is mean-zero and stationary before trusting the post-period divergence; the ADF and Engle-Granger machinery for that, including the corrected critical values you need when testing a fitted residual rather than an observed series, is covered in statistical arbitrage and pairs trading.

Multiple Treated Units

Deploying across several assets at once breaks the single-treated-unit design. Extensions: pooled SCM (average effects across treated units) and staggered adoption designs (Ben-Michael et al., 2022) for units treated at different times.

When Not to Use SCM

Well suited to: a discrete, well-defined intervention at a known date; a reasonable donor pool of comparable untreated units; interventions not anticipated in advance, since anticipation moves the treated unit before $T_0$ and violates identification.

Poorly suited to: continuous treatment, such as a parameter drifting over weeks; settings with no genuinely untreated comparables; high-frequency evaluation where the signal-to-noise ratio defeats weight estimation.

Conclusion

Selection and confounding are different failure modes and need different instruments. DSR and PBO price the search. SCM prices the counterfactual, and it is the only tool in this series that addresses the case where you ran exactly one experiment and still cannot trust the result.

The reason to prefer it over an ad-hoc benchmark is not that it produces larger effects — it frequently produces smaller ones — but that it can refuse. If no convex combination of donors tracks your instrument before the intervention, SCM says so and you stop. If the treated gap sits inside the donor cloud, the placebo p-value says so and you report a null. Those two exits are the value of the method, and an SCM write-up without a reported pre-period MSPE and a placebo p-value has skipped the only parts that make it more than a chart of two lines diverging.

Further Reading

• Abadie, A., Diamond, A., & Hainmueller, J. (2010). "Synthetic Control Methods for Comparative Case Studies." Journal of the American Statistical Association, 105(490), 493-505.
• Abadie, A. (2021). "Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects." Journal of Economic Literature, 59(2), 391-425.
• Ben-Michael, E., Feller, A., & Rothstein, J. (2021). "The Augmented Synthetic Control Method." Journal of the American Statistical Association, 116(536), 1789-1803.
• Cunningham, S. (2021). Causal Inference: The Mixtape. Chapter 10: Synthetic Control. Available at mixtape.scunning.com.
• Microsoft Research. SparseSC: Sparse Synthetic Controls. github.com/microsoft/SparseSC.