The Fidelity Gate: Coarse-to-Fine Backtesting Fools You Faster Unless the Cheap Proxy Ranks Like the Expensive One

Part of the "Backtests Without Illusions" series.

Every serious parameter search hits the same wall: the space is enormous, and each honest evaluation is expensive. A multi-timeframe strategy with a dozen thresholds and three indicator periods, evaluated over a multi-fold walk-forward split, can cost seconds per configuration. Ten thousand configurations is hours. So you look for a shortcut, and the shortcut is always the same idea: evaluate cheaply, promote only the survivors to the expensive test. Screen ten thousand configs on one fold, keep the top few hundred, run those on all folds. Screen on hourly candles, drill down to 1-second only for the finalists.

This is coarse-to-fine search — drill-down, multi-fidelity, successive halving, Hyperband, ASHA — and it is one of the genuinely good ideas in optimization. It can turn a day of compute into an hour. We use it. But it hides a trap so quiet that most people who deploy it never check for it, and the trap is this: the whole method assumes your cheap proxy ranks configurations the same way the expensive evaluation does. If it doesn't, you are not searching faster. You are throwing away your future winners at the first gate and promoting noise — you are just fooling yourself faster.

This article is about the one measurement that tells you whether your drill-down is real or a delusion. We ran it on our own multi-timeframe search, and the answer was uncomfortable: at the cheapest fidelity, our proxy ranked configurations almost at random (Spearman ρ ≈ 0.03). If we had pruned aggressively at that fidelity — as every off-the-shelf ASHA default invites you to — we would have discarded the eventual winners in the first round. The fix is a mandatory gate that runs before the search: measure the rank correlation, and refuse to prune below the fidelity where it is trustworthy.

What multi-fidelity search actually is

The family of methods has one shape. You define a fidelity (also called a resource or budget): a cheap-to-expensive dial that, at its maximum, gives the true objective. Then you run configurations at low fidelity, rank them, keep the best fraction, and re-run only those survivors at higher fidelity — repeating until a small number of finalists have been evaluated at full fidelity.

Successive halving (Jamieson & Talwalkar, 2016) is the core: start N configs at the minimum resource, keep the top 1/η, multiply the resource by η, repeat. Hyperband (Li et al., 2018) wraps successive halving in an outer loop that hedges across different starting resources, so you don't have to guess how aggressively to prune. ASHA (Li et al., 2020) is the asynchronous, parallel-friendly version. The elimination rate η is the one knob they share; the canonical default is η = 3 — keep the top third at each rung, triple the budget.

The fidelity itself can be almost anything cheap-to-expensive that converges to the truth:

• Number of walk-forward folds. Evaluate on 1 fold, then 2, then 3, up to K. This is the fidelity we study below, because it is the natural one for a backtest with a rolling out-of-sample split.
• Candle resolution. Screen on 1-hour bars, promote to 1-minute, drill down to 1-second or raw trades only for finalists. This is the adaptive drill-down idea applied to search rather than to fill simulation.
• Length of history, epochs, dataset subsample — the classic ML fidelities.

In a backtest, folds and resolution are the two that matter, and — this is the point of the whole article — they are not equally safe. One of them has a bias that quietly selects the wrong parameters no matter how carefully you gate it.

The one assumption everything rests on

Write down what successive halving actually needs to be correct. It does not need the cheap fidelity to give the right objective value — nobody cares that one fold reports a different Sharpe than six folds. It needs something weaker but far more specific: the cheap fidelity must rank configurations the same way the expensive one does. If config A beats config B at full fidelity, A should tend to beat B at low fidelity too. That is all. That is also everything.

Formally, the quantity that governs whether drill-down works is the rank correlation between the low-fidelity and high-fidelity objective, across the configuration space. If that correlation is high, the survivors of the cheap round are the same configs that would have won the expensive round, and you have saved compute for free. If it is low, the cheap round is a random filter: it discards good configs and promotes bad ones, and you have spent your budget accelerating your way to a worse answer. There is no middle setting of η that rescues you — a bad proxy pruned gently still leaks winners; pruned aggressively it hemorrhages them.

The failure is invisible in the usual way. The search completes, reports a champion, the champion looks fine in-sample. Nothing throws an error. You only discover the proxy was lying when the champion falls apart out-of-sample — and by then you have attributed the failure to the strategy, not to the search that selected it. So the discipline is the same one this series keeps arriving at: measure the thing the method assumes, before you trust the method. For drill-down, the thing to measure is the rank correlation, and the measurement is cheap.

Measuring it: the fidelity gate

The gate is a small experiment you run once, before the search proper, on the same space you are about to search. The procedure:

1. Draw a few hundred configurations at random from the parameter space (we use ~200 — enough for a stable Spearman estimate, cheap enough to afford).
2. Evaluate each one at every fidelity rung: at 1 fold, 2 folds, … up to the full K folds. For a fold-count fidelity this is nearly free, because the folds you compute for the cheap rungs are reused in the expensive one — a cumulative average.
3. For each rung r, compute the Spearman rank correlation between the r-fold ranking and the full K-fold ranking, across those ~200 configs.
4. The first rung where ρ crosses a threshold (we use ρ ≥ 0.5) is the shallowest fidelity you are allowed to prune at. Below it, the ranking is too noisy to trust; you must not eliminate configs there.

The whole thing is a couple of dozen lines. The heart of it:

def fidelity_check(cache, n_probe, seed=7):
    """Spearman ρ: cumulative mean over the first r folds (in FOLD_ORDER)
    vs the full K-fold objective, on n_probe random configs."""
    rng = np.random.default_rng(seed)
    k = len(FOLDS)
    per_fold = np.empty((k, n_probe))
    order_win = np.array([list(FOLDS[fi]) for fi in FOLD_ORDER], np.int64)
    for j in range(n_probe):
        p = _unit_to_params(rng.random(len(PNAMES)))       # random config
        scores, *_ = eval_group(cache, p, _sp_of(p)[None, :], order_win, False)
        per_fold[:, j] = scores[0]                          # score on each fold
    cums = np.cumsum(per_fold, axis=0) / np.arange(1, k + 1)[:, None]  # r-fold mean
    rhos = []
    for r in range(1, k):
        rho = spearmanr(cums[r - 1], cums[-1]).statistic    # r folds vs all K
        rhos.append(0.0 if math.isnan(rho) else rho)
    return rhos

Note one deliberate detail: the folds are taken in FOLD_ORDER, an interleaved order that alternates early and late slices of the calendar (fold 0, then a middle fold, then fold 1, then a later one, …). This matters enormously and is the subject of a later section: it means "1 fold" is one slice spanning the middle of the history, and "2 folds" is one early plus one late slice — never a contiguous recent window. The cheap fidelity gets cheaper by using fewer folds, not more recent ones.

Results: one fold ranks almost randomly

Here is what the gate reported, run on two of our multi-timeframe setups. The number is Spearman ρ between the r-fold ranking and the full K-fold ranking across ~200 random configs — how faithfully cheap-fidelity rank predicts full-fidelity rank.

Read the harder-regime column first, because it is the alarming one. At one fold, ρ = 0.03. That is not a weak correlation; it is no correlation — the one-fold ranking of two hundred configurations is statistically indistinguishable from shuffling them. A successive-halving run configured (as most are, by default) to start pruning at min_resource = 1 would, in this regime, do its first and most aggressive cut on a coin flip. The configs that would eventually win at full fidelity have a 50/50 chance of surviving the very first round. By the time you reach the fidelity where ranking means something (ρ = 0.78 at three folds), most of them are already gone.

The multi-TF column is milder but makes the same point differently. Even there, one fold gives ρ = 0.43 — below our 0.5 gate. It looks like a decent correlation, and that is exactly the danger: 0.43 is high enough to lull you and low enough to leak your best configs. Only at two folds (ρ = 0.67) does the ranking become trustworthy.

Two things generalize from this. First, ρ at one fold is regime-dependent and unreliable — we measured anywhere from 0.03 to 0.43 across setups, and in neither case did a single fold clear the bar. Second, ρ rises monotonically and fast: by three folds both setups are at 0.78, and by five they converge to 0.91. The signal is there; you just have to spend enough fidelity to see it. The gate's job is to find the exact rung where "enough" begins — and to forbid pruning below it.

Why a single fold is so noisy

The noise is not a bug in our fold construction; it is intrinsic, and understanding why keeps you from "fixing" it the wrong way. A single walk-forward fold is a short window — weeks of one market regime. A strategy's score on that window is dominated by how well its parameters happen to fit that regime, which is only loosely related to how well they generalize. Two configs that are genuinely different in quality can trade places on a single fold just because one of them caught a trend that fold happened to contain. The objective on one fold is a high-variance estimator of the objective you actually care about, and rank correlation is exactly what high variance destroys first — you can preserve the mean of an estimator while its ranking is pure noise.

Adding folds averages down that variance. Each fold is a partly independent draw of market conditions; the r-fold mean is a lower-variance estimator, and its ranking converges toward the full-fidelity ranking. That is precisely the ρ = 0.03 → 0.43 → 0.78 → 0.91 climb: not the objective changing, but its rank-estimate stabilizing as regime-specific luck averages out. The lesson is that fidelity, for a backtest, is fundamentally about how many independent regimes you have sampled — and one is almost never enough to rank on.

This also explains why the harder regime starts at 0.03 while the multi-TF run starts at 0.43. In the harder regime the single fold is more regime-idiosyncratic — the configs' one-fold scores are driven more by luck and less by durable edge, so they rank closer to random. The gate reads that difference automatically and responds by demanding more folds before it will prune. You do not have to know in advance which regime you are in; you measure.

The gate in code: auto-raise the minimum fidelity

The gate's output is a single integer: min_resource, the shallowest fidelity ASHA is permitted to prune at. The rule is mechanical — walk the rungs, take the first one whose ρ clears the threshold:

RHO_GATE = 0.5
min_res = len(FOLDS)                      # default: pruning OFF (full fidelity)
rhos = fidelity_check(cache, n_probe=200) # [ρ@1, ρ@2, …, ρ@(K-1)]
passing = [r for r, rho in enumerate(rhos, 1) if rho >= RHO_GATE]
if passing:
    min_res = passing[0]                  # first rung that clears the gate

pruner = SuccessiveHalvingPruner(min_resource=min_res, reduction_factor=3)

Trace it through the two runs. In the multi-TF setup, ρ@1 = 0.43 fails the gate but ρ@2 = 0.67 clears it, so min_resource is auto-raised to 2: ASHA runs every config on at least two folds before it is allowed to eliminate anything, then prunes normally from there. In the harder regime, ρ@1 = 0.03 and ρ@2 = 0.43 both fail; ρ@3 = 0.78 is the first to clear, so min_resource becomes 3. And the crucial fallback: if no rung reaches 0.5, passing is empty and min_resource stays at K — pruning is switched off entirely, and the search degrades gracefully to a plain full-fidelity search rather than a fast wrong one. A drill-down that cannot prove its proxy simply refuses to prune.

This is the whole philosophy in one line of control flow. The default of every successive-halving library is "prune from min_resource = 1, trust the cheap rung." The gate replaces that with "prune from the first rung the data says is trustworthy, and if none is, don't prune." It costs one ~200-config probe up front and it converts drill-down from an act of faith into a measured decision. The elimination rate η = 3 stays as-is; the gate does not touch how hard you prune, only how early you are allowed to start.

One honesty note visible in the code above: raising min_resource eats into the speedup. Pruning at 3 folds instead of 1 means every config — including the ones you will discard — pays for three folds. That is the price of correctness, and it is the right trade: a 2× smaller speedup on a search that finds real winners beats a 6× speedup on a search that discards them. The gate makes that trade explicitly instead of hiding it.

The wrong cheap axis: shorter history is a trap

There is a cheap fidelity that is tempting, obvious, and biased, and it is worth calling out by name because everyone reaches for it first: shorter history. Screen configs on the last month, promote the survivors to the full two years. It is trivially cheap and it feels like the same idea as fewer folds. It is not.

Fewer folds and shorter history differ in one decisive way. A fold-count fidelity, done right, still spans the entire calendar — it just samples it more coarsely. A shorter-history fidelity samples a sub-interval of the calendar densely. And a sub-interval of market history is a specific regime. When you rank configs on the last month, you are not getting a noisy-but-unbiased estimate of their full-period rank; you are getting a biased estimate that systematically favors configs tuned to last month's regime. Raise the sample size all you want — average over more configs, more trials — and the bias does not shrink, because it is not variance. You will promote the configs that best fit the screening window, and those are exactly the ones most likely to be overfit to a transient regime.

This is why our fidelity walks folds in an interleaved order rather than chronologically. With K folds spread across the whole history, "1 fold" is a single slice near the middle, "2 folds" is one early and one late slice, "3 folds" spreads early/middle/late — every fidelity level, even the cheapest, samples across the full calendar. The cheap proxy is a coarse view of the whole period, never a sharp view of one slice of it. That interleaving is what makes the fold-count fidelity merely noisy (curable by the gate) instead of biased (which no gate can fix — a high ρ against a biased target just certifies that you will reliably pick the regime-fit config). If your only way to make a fidelity cheaper is to make it more recent, you do not have a valid fidelity. Make it coarser instead.

Resolution as the honest cheap axis

The other cheap axis avoids the bias entirely, and it is the natural partner to the adaptive drill-down we described for fill simulation: candle resolution. Screen the whole space on 1-hour bars, promote survivors to 1-minute, and drill down to 1-second or raw trades only for the handful of finalists. Coarser candles over the same full history are cheaper to evaluate — fewer bars, faster indicator passes, faster simulation — and, unlike a shorter window, a coarse view of the whole calendar is unbiased: it sees every regime, just with less intrabar detail.

Resolution and folds are complementary fidelity axes, and the fidelity gate applies to both. Before you trust a 1-hour screen, run the same probe: take ~200 configs, score them at 1h and at 1m, and measure the rank correlation. If ρ(1h, 1m) is high, screening on hours is safe and you have bought a large speedup honestly. If it is low — which happens when the strategy's edge lives in intrabar structure that hourly candles smear away — then hourly screening is a random filter for that strategy, and the gate tells you to start finer. The rule never changes: you are allowed to prune at a fidelity only after you have measured that its ranking agrees with the truth.

The two axes also fail in opposite ways, which is useful. Coarse resolution loses intrabar information; fewer folds lose cross-regime information. A momentum strategy on daily horizons may rank perfectly on hourly candles but need many folds to average out regime luck; a scalping strategy may rank fine on few folds but fall apart above 1-second resolution. The gate, run per-axis, tells you which fidelity you can afford to be cheap on for this strategy — rather than assuming an answer that happens to be convenient.

Where the savings actually come from

A caveat that keeps drill-down honest: a fidelity only saves compute where the cost actually lives. In our multi-timeframe engine the expensive part is precomputing indicators — the multi-timeframe HMA and separation signals — which is paid once per configuration, before any fold runs. The per-fold simulation on the cached signals is comparatively cheap. So pruning on fold count saves only the simulation cost, not the indicator cost that dominates; the fold-count fidelity is real but its ceiling is lower than the raw fold ratio suggests.

The resolution axis, by contrast, attacks the dominant cost directly: coarser candles mean fewer bars to compute indicators on, so both the expensive precompute and the cheap simulation shrink together. This is not a detail — it decides which drill-down is worth building. Before you invest in a multi-fidelity search, ask where your seconds go. If 90% is indicator precompute shared across folds, a fold-count fidelity buys you little and a resolution fidelity buys you a lot. Profile first; the right cheap axis is the one that removes the cost you actually have, and it must still clear the gate.

Where this connects

The fidelity gate sits at a specific point in this series' chain of backtest hygiene:

• It is upstream of overfitting control. A drill-down that prunes on a bad proxy is a new way to overfit — you are letting a noisy early rung select your finalists. The champion still has to survive Deflated Sharpe and PBO, and the count of trials feeding those gates must include every pruned trial, not just the survivors — pruning does not make a trial free from the multiple-testing ledger.
• It shares its enemy with plateau analysis: a config that wins one fold or one screening window and nowhere else is the same regime-fit artifact both tools exist to reject. The gate refuses to select on such a fold; plateau analysis refuses to trust a champion that stands on one.
• It presumes the honest walk-forward split underneath — folds spanning the full calendar, out-of-sample held aside — and it is the search-time complement to the drill-down we built for fill simulation: same coarse-to-fine principle, applied to which configs to evaluate rather than how precisely to fill them.
• And it depends absolutely on the fidelity being leak-free at every rung. If the cheap rung has look-ahead bias the expensive one lacks, ρ measures agreement with a contaminated target and the gate certifies a leak. Measure rank correlation, yes — but on an honest objective.

The unifying idea, again, is the one this series will not stop repeating: a backtest is a statistical experiment, and every shortcut in it is a hypothesis you are obligated to test. Drill-down's hypothesis is "cheap rank ≈ expensive rank." It is testable in ~200 configs. Test it.

Takeaways

1. Coarse-to-fine search rests on one assumption: the cheap proxy ranks configs like the expensive one. Not the same value — the same ranking. If rank correlation is low, aggressive pruning discards your future winners and promotes noise. You search faster toward a worse answer.
2. Measure it before you trust it. Draw ~200 random configs, score them at every fidelity rung, and compute Spearman ρ between each cheap rung and the full-fidelity ranking. It is a couple dozen lines and one cheap probe.
3. One fold ranks almost randomly. We measured ρ@1 from 0.03 (a coin flip) to 0.43 (still below trust) depending on regime; it climbs to 0.67, 0.78, 0.82, and 0.91 as folds accumulate. The default min_resource = 1 of every off-the-shelf ASHA is, for a backtest, usually wrong.
4. Auto-raise the minimum fidelity to the first rung where ρ ≥ 0.5, and if none clears, don't prune at all. The gate turned into min_resource = 2 for one setup and 3 for a harder one; the fallback degrades gracefully to full-fidelity search. Correctness costs some speedup — pay it.
5. Choose the cheap axis by whether it is biased, not by whether it is cheap. Shorter history is biased — it selects regime-fit params and no sample size fixes that. Use fewer folds spanning the full calendar (interleaved, not contiguous) or coarser resolution over the full period. And spend the cheap axis where your cost actually lives.

Drill-down is one of the best speedups in backtesting, and one of the easiest to turn into a faster way of fooling yourself. The difference between the two is a single number you can measure before the search starts. If your proxy can't prove it ranks like the truth, it isn't a proxy — it's a random number generator with a plausible cost profile.