The Other Way a Regression Lies: Endogeneity, 2SLS, and the Gamma Calibration Problem

There are two structurally different ways a regression can lie to you, and this blog has so far only attacked one of them.

The first is selection bias in the search: you ran ten thousand backtests, kept the best, and the number you are looking at is the maximum of a noise distribution rather than an effect. That is the subject of the deflated Sharpe ratio and its capstone honest negative. The fix is to price the search.

The second is bias in the coefficient itself, on a single regression you never searched over at all. If the regressor is correlated with the error term, OLS is biased and inconsistent — and unlike sampling noise, this does not shrink with sample size. More data makes the wrong number tighter. No amount of bootstrapping, walk-forward, or deflation touches it, because none of those methods question the identification, only the selection.

This article is about the second failure, and it is not an abstract concern here. The Almgren-Chriss calibration article fits permanent impact $\gamma$ by regressing 5-minute mid-price changes on net taker flow, gets an $R^2$ of a few percent and a coefficient that moves 2-5x between days, and names endogeneity as the first structural reason: "Flow responds to price as much as price responds to flow. Momentum traders buy because the price rose; a naive OLS of returns on flow picks up their reaction and attributes it to impact." It also names the clean fix — your own fills, exogenous by construction.

That is an instrumental-variables problem with a named instrument and data already in-house. Below is the estimator, the diagnostics, and the experiment.

Endogeneity in one paragraph

OLS estimates the causal $\beta_1$ in $Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i$ only under $\mathbb{E}[\varepsilon_i \mid X_i] = 0$. Three things break it. Omitted-variable bias: if $W$ drives both $X$ and $Y$ and is unobserved, it lives in the error, $\varepsilon_i = \gamma W_i + u_i$, and $\text{Cov}(X_i, W_i) \neq 0$ forces $\text{Cov}(X_i, \varepsilon_i) \neq 0$ — in microstructure, unobserved information arrival drives volume and volatility together. Measurement error: with a noisy proxy $X_i = X_i^* + \nu_i$, $\text{plim } \hat{\beta}_1^{\text{OLS}} = \beta_1 \cdot \sigma_{X^*}^2 / (\sigma_{X^*}^2 + \sigma_\nu^2)$, always attenuated toward zero — netted aggTrades flow is a proxy for the true signed order flow across a fragmented market, and the part you never see is measurement error. Simultaneity: already stated in sharper and more concrete form in the Almgren-Chriss impact calibration, which is the motivating problem for everything below.

The IV solution

An instrument $Z$ must satisfy two conditions:

1. Relevance: $\text{Cov}(Z_i, X_i) \neq 0$. Testable, and you must test it.
2. Exclusion: $\text{Cov}(Z_i, \varepsilon_i) = 0$ — $Z$ affects $Y$ only through $X$. Not testable. Ever. It is an economic argument, not a statistic.

For one instrument and one endogenous regressor the estimator is a ratio of two reduced-form effects (the Wald estimator):

With controls and multiple instruments you use 2SLS. Stage 1: regress the endogenous variable on instruments and controls, $X_i = \pi_0 + \boldsymbol{\pi}_Z' \mathbf{Z}_i + \boldsymbol{\pi}_C' \mathbf{C}_i + v_i$, and keep the fitted values $\hat{X}_i$ — by construction they contain only instrument-driven, exogenous variation. Stage 2: run $Y_i = \beta_0 + \beta_1 \hat{X}_i + \boldsymbol{\beta}_C' \mathbf{C}_i + \varepsilon_i$. In matrix form, with $\mathbf{P}_Z = \mathbf{Z}(\mathbf{Z}'\mathbf{Z})^{-1}\mathbf{Z}'$:

The trap that catches everyone who hand-rolls this: if you literally run two lstsq calls, your stage-2 standard errors are wrong. $\hat{X}_i$ is a generated regressor, and the residual variance must be computed from the original $X$, not the fitted one. Naive two-step standard errors are too small, so your $t$-stats are too big and you will over-reject. Use a real IV implementation:

from linearmodels.iv import IV2SLS
import pandas as pd

def iv_regression(df, dep_var, endog_var, instruments, controls=None):
    """2SLS with correct generated-regressor standard errors."""
    y = df[dep_var]
    endog = df[[endog_var]]
    instr = df[instruments]

    const = pd.Series(1, index=df.index, name="const")
    exog = pd.concat([const, df[controls]], axis=1) if controls else const

    res = IV2SLS(dependent=y, exog=exog, endog=endog,
                 instruments=instr).fit(cov_type="robust")
    print(res.summary)
    return res

The experiment: OLS gamma versus 2SLS gamma

The regression under test is the permanent-impact fit from the Almgren-Chriss article:

$Q_{\text{net}}$ is endogenous by the simultaneity argument above. The instrument is your own fills in the window: your maker fills are triggered by your own quoting schedule and inventory policy, not by the market's price path, so they are a source of order-flow variation that is plausibly uncorrelated with the momentum-reaction component of the error. The exclusion restriction — that your fills move the mid only through their contribution to net flow — is the assumption to argue about, and it is weakest exactly when your quoting is itself signal-driven. If your quotes condition on short-horizon price prediction, the instrument is contaminated and this design fails. Fills from a pure inventory-management or spread-capture policy are the clean case.

The fill data comes from the TCA record described in implementation shortfall and DIY TCA — the same log that supplies the fitting sample for slippage curves from your own fills. Controls: lagged $\Delta S$ and realized volatility in the window.

ols  = IV2SLS(dependent=df.dS, exog=df[["const", "dS_lag", "rv"]],
              endog=None, instruments=None).fit(cov_type="robust")
tsls = iv_regression(df, "dS", "q_net", ["q_own"], ["dS_lag", "rv"])

Four numbers decide whether this was worth doing: the OLS $\gamma$, the 2SLS $\gamma$, the first-stage $F$, and the gap between the two coefficients — which is the share of the OLS estimate that was momentum reaction rather than impact.

If the first-stage $F$ comes back below 10, that is the result: own fills are too small a fraction of net flow at 5-minute resolution to identify $\gamma$, and the honest write-up is a negative in the style of the honest negative. A weak-instrument negative is a real finding — it tells you the endogeneity in the gamma fit is not fixable at this sample size and horizon, and that pre-trade cost estimates built on the OLS $\gamma$ carry an unremovable upward bias.

Diagnostics

First-stage F (weak instruments). Staiger-Stock rule of thumb: $F > 10$. Stock-Yogo critical values for one endogenous regressor are stricter — for 10% maximal IV size, $F > 16.38$ with one instrument, 19.93 with two, 22.30 with three; for 15% size, 8.96 / 11.59 / 12.83; for 20%, 6.66 / 8.75 / 9.54.

Durbin-Wu-Hausman (is IV even needed). If OLS is consistent it is also more efficient, so test before you switch. The residual-augmentation form is the practical one: run stage 1, keep $\hat{v}_i = X_i - \hat{X}_i$, then run the OLS structural regression with $\hat{v}_i$ added as a regressor. A significant coefficient on $\hat{v}_i$ is evidence of endogeneity.

def durbin_wu_hausman(df, dep, endog, instruments, controls):
    """Augmented-regression form of the DWH endogeneity test."""
    import statsmodels.api as sm
    X1 = sm.add_constant(df[instruments + controls])
    v_hat = df[endog] - sm.OLS(df[endog], X1).fit().fittedvalues
    X2 = sm.add_constant(df[[endog] + controls].assign(v_hat=v_hat))
    res = sm.OLS(df[dep], X2).fit(cov_type="HC1")
    return res.tvalues["v_hat"], res.pvalues["v_hat"]

Sargan-Hansen (overidentification). With $q > p$ instruments you can test joint validity: $J = n \cdot R^2_{\hat{\varepsilon} \sim \mathbf{Z}} \sim \chi^2(q - p)$, where the $R^2$ comes from regressing the 2SLS residuals on all instruments and exogenous controls. The caveat matters more than the statistic: Sargan only detects instruments that are invalid in different ways. If every instrument is correlated with the same omitted variable, the test passes cleanly and is telling you nothing.

Instruments that exist in crypto microstructure

Finding a valid instrument is the whole difficulty, and the equity-corporate-finance standbys (analyst coverage, tax shields, windfall cash flow) are inert here. Two classes actually apply:

Pitfalls

Weak instruments do not fail loudly. With a weak first stage, 2SLS is biased toward OLS in finite samples, roughly

At $F = 5$ you retain 25% of the OLS bias while reporting a "causal" estimate. The remedies are LIML, which has better finite-sample behaviour, or the Anderson-Rubin test, which is valid regardless of instrument strength.

Many instruments bias. $\text{Bias}_{2\text{SLS}} \approx (q/n) \cdot \text{Bias}_{\text{OLS}}$. Throwing lags of everything into $\mathbf{Z}$ to boost the first-stage $F$ walks the estimate straight back to OLS. If $q/n$ is not negligible, use LIML, jackknife IV (JIVE), or a regularized first stage.

The exclusion restriction cannot be tested. Not by Sargan, not by anything. It is defended with domain reasoning, and the honest form of that defense is a sensitivity analysis: how large a violation would be needed to overturn the conclusion?

Takeaways

• Deflation fixes bias from the search. It does nothing for bias in the coefficient. They are different failures and need different tools.
• Endogeneity bias does not vanish with sample size. A tight, well-bootstrapped, walk-forward-validated coefficient can still be systematically wrong.
• The gamma regression in the Almgren-Chriss calibration is a live, in-house instance, with a named instrument (own fills) and the data already logged by the TCA pipeline.
• Report the first-stage $F$ before the 2SLS coefficient. Below 10, the coefficient is not evidence, and saying so is the publishable result.