Random vs Smart Search: The Crossover Is Eval Cost, Not the Algorithm

Part of the "Backtests Without Illusions" series.

There is a piece of received wisdom in hyperparameter optimization: random search is a baseline you outgrow. Bergstra & Bengio's classic result (2012) established that random beats grid; then Bayesian optimization, TPE, CMA-ES, and multi-fidelity methods like Hyperband/ASHA were supposed to beat random in turn. So when we sat down to benchmark parameter-search methods for our own trading engine, we expected the usual ladder: random at the bottom, a smart sampler on top.

We got the opposite — and then we got the textbook answer too. Same strategy, same parameter space, same objective, same machine. The only thing we changed was how expensive a single backtest was, and the ranking of the search methods inverted. When each evaluation was cheap, a dumb scrambled Sobol sequence crushed every "smart" sampler. When we made each evaluation expensive, the smart methods pulled ahead and found the only configuration that survived out-of-sample.

The lesson is not "random is underrated" or "Bayesian is overrated." It is that the crossover between random and smart search is governed by evaluation cost, not by the cleverness of the algorithm. Pick your optimizer by how much one backtest costs, not by its reputation. This article measures exactly where the crossover sits, why it sits there, and one precondition — fidelity — that decides whether the expensive-regime tricks (early stopping, multi-fidelity pruning) are safe to use at all.

Everything below is from two scripts in our backtester: bench_search.py (v4, commit ee092f1) for the cheap single-timeframe regime, and bench_search_multitf.py (commit 102853c) for the expensive multi-timeframe regime. Both are leak-free — decision on the close of bar i, fill on open[i+1] — and both score every method on a multi-fold rolling walk-forward objective with a held-out test window the search never sees.

The question: throughput or sample-efficiency?

Every search method spends its wall-clock budget on two things: deciding where to sample next (the sampler's own compute) and evaluating the sample (running the backtest). Call the first the ask/tell cost and the second the eval cost. A method's effective search power over a fixed wall-clock budget is, roughly:

$\text{points explored} \;\approx\; \frac{\text{wall-clock budget}}{\text{ask/tell cost} + \text{eval cost}}$

and its final quality is that count times how well-placed each point is. Two knobs, pulling against each other:

• Throughput — points per second. Dumb samplers (random, scrambled Sobol/QMC) have near-zero ask/tell cost: they emit a low-discrepancy point and move on. They maximize the count.
• Sample-efficiency — quality per point. Smart samplers (TPE, CMA-ES, ASHA) spend real compute modeling the objective to place each point better. They maximize the placement, at the cost of throughput.

Which knob wins depends entirely on the denominator. When eval cost is tiny, the ask/tell cost dominates the denominator, so anything that inflates it — a surrogate model, a kernel density estimate, a covariance update — directly shrinks the number of points you explore. When eval cost is large, the ask/tell cost is a rounding error, so smartness is effectively free and you should buy all of it you can.

That is the whole thesis in one sentence: the ask/tell tax is fixed, but its significance is set by the eval cost you divide it against. Now let us watch it happen.

The cheap regime: dumb Sobol wins on throughput

Our single-timeframe strategy is a leak-free HMA/HMA3 separation rule over a 7-parameter space, evaluated by an in-process numba eval_batch that runs prange over configurations with no inter-process overhead. On this engine, one backtest is almost free — the raw kernel evaluates configs at roughly 3–4k cfg/s. This is the cheap regime, and it is where trading differs sharply from the deep-learning setting most HPO folklore comes from: our "objective function" is not a 6-hour GPU training run, it is a 0.3-millisecond array pass.

We gave every method the same budget — 1,500 evaluations — and recorded the wall-clock each one needed to spend those evals, plus the held-out test objective it found. Because the eval budget is fixed, the wall-clock column is a direct read of each sampler's overhead:

TEST is the walk-forward objective (annualized PnL-per-active-time × trade-count confidence) on a held-out window the search never touched; higher is better.

Two facts jump out. First, look at the throughput column. Scrambled Sobol runs at ~2,830 cfg/s — near the engine's ceiling. TPE runs at ~154 cfg/s and ASHA at ~95. That is an 18–30x slowdown to do the identical number of evaluations. The smart samplers are not evaluating anything extra; they are spending that time inside their own ask/tell machinery.

Second — and this is the part that keeps the story honest — no method found a positive out-of-sample result. Every TEST value is negative. In the single-timeframe regime our strategy simply has no durable OOS edge, so "which method wins" is not a question about final profit; it is a question about search efficiency. And on search efficiency at a fixed eval budget, the dumb methods win outright: Sobol and random reach the same or better held-out numbers as the smart samplers while spending 1/20th of the wall-clock.

Now flip the comparison the way a practitioner actually experiences it — fix the wall-clock, not the eval count. If you give every method the 15.8 seconds ASHA needed for its 1,500 evals, Sobol does not stop at 1,500. It keeps going, to roughly 45,000 configurations. In the cheap regime the question is never "which sampler places 1,500 points best" — it is "would you rather have 1,500 cleverly-placed points or 45,000 scrambled ones, when each point is nearly free?" With a near-free eval, breadth wins. Thirty times more coverage of a 7-dimensional space beats a better model of it.

The ask/tell tax

Where does the 20x go? Not into the backtest — that is identical across methods. It goes into the sampler's per-point bookkeeping, run in Python, in the loop:

• TPE fits a pair of kernel-density estimates (good vs bad trials) on every ask, and the cost grows with the trial history. Multivariate TPE fits them jointly across dimensions — more modeling, more Python.
• CMA-ES updates and samples from a covariance matrix each generation. Cheaper than TPE here (it ran at ~850 cfg/s), but still an order of magnitude of overhead above emitting a Sobol point.
• ASHA pays the pruner's promotion/rung bookkeeping and, in our folds-as-fidelity design, pays the fixed indicator precompute before it can prune anything — so its "saved" evaluations save less than the accounting suggests.

None of this is a knock on the algorithms. It is the point: the ask/tell cost is a roughly fixed number of milliseconds per point, and when the eval it wraps is also a few milliseconds, that fixed cost is suddenly 90% of your budget. The smart sampler spends nine-tenths of its wall-clock thinking about where to look and one-tenth actually looking. A scrambled Sobol sequence spends all of it looking. When looking is cheap, looking wins.

We deliberately did not benchmark a full Gaussian-process Bayesian optimizer here, and for the same reason: a GP surrogate is $O(n^3)$ in the trial count. Against an eval that costs milliseconds, fitting the surrogate would consume the entire search budget before it evaluated a meaningful fraction of the space. In the cheap regime, GP-BO is disqualified by arithmetic.

The expensive regime: the crossover flips

Now we make one backtest expensive. The multi-timeframe strategy stacks a high, a mid, and a low timeframe (triple-TF), each contributing an indicator pass and its own thresholds, all scored across the same multi-fold walk-forward. A single evaluation now costs on the order of 0.1–0.5 seconds instead of 0.3 milliseconds — a three-orders-of-magnitude jump. The eval cost has moved from the rounding-error term of our denominator to the dominant term. By the thesis, the ask/tell tax should stop mattering and smartness should start paying. It does.

We ran every method under a fixed ~150-second wall-clock budget on the triple-TF problem (an 18-parameter space), let each spend that budget however its sampler dictates, and evaluated the single best config it returned on a held-out test window:

TEST is the same walk-forward objective as before. Only one method crossed zero.

The dumb Sobol baseline that dominated the cheap regime is now dead last, at −673. Blind low-discrepancy sampling of an 18-dimensional space, with only 349 evaluations to spend because each one is expensive, never localizes anything. The smart method, sobol→cmaes — 30% of the budget on Sobol to seed a basin, then CMA-ES refinement from the best seed — is the only method that produced a positive out-of-sample result at all. On the final untouched holdout the champion returned +2.62% (19 trades, ~6.6% capital exposure) on top of a test window that returned +16.35% (46 trades, ~15.7% exposure). Every competitor's champion lost money out-of-sample.

That is the crossover, measured on the same strategy family, the same objective, the same machine: change nothing but the cost of one evaluation, and the ranking of search methods inverts. In the cheap regime Sobol wins and the smart samplers are a 20x waste; in the expensive regime the smart sampler is the only thing that works and Sobol is the waste.

Why "smart" wins here — and it isn't only sample-efficiency

The tidy version of this story is "expensive evals reward sample-efficiency, so the method that places fewer, better points wins." That is half true, and the data forces the honest, more interesting other half.

Look at the eval counts again. sobol→cmaes did not win by evaluating fewer points than blind Sobol — it evaluated 15,239 to Sobol's 349, forty times more, in the same 150 seconds. How? Because our multi-TF eval cost is structured, not uniform. There are two axes: an expensive indicator axis (the timeframe periods and HMA lengths, 30–500 ms each to compute, because they force an indicator recompute) and a cheap threshold axis (the entry/exit separation levels, ~1–2 ms each on cached signals). The gap between them is 30–100x.

Blind Sobol ignores this structure. Every point it draws jitters the expensive indicator axis, forcing a fresh recompute — so it pays full price on all 349 evaluations. sobol→cmaes, once CMA-ES has localized a promising region, tends to hold the coarse indicator structure roughly fixed and perturb the continuous thresholds, which land on cached signals and cost almost nothing. The smart method converts the same wall-clock into both better-placed points and far more of them, because being adaptive here means being cost-aware: staying on the cheap axis after the expensive axis is pinned down. Our explicit cascade(sobol²×64) exploit does this most aggressively — 20,864 evals by batching cheap thresholds on cached signals — and while it lost the triple-TF test, in the two-timeframe variant it won the test window outright at +20.2% (before failing its own holdout — more on that below).

So the sharper statement of the crossover: in the expensive regime, the ask/tell tax becomes negligible, which frees you to be smart — and "smart" means adapting to the objective's cost structure, not just its shape. Blind sampling can do neither. This is precisely the two-axis structure our adaptive-resolution drill-down engine is built to exploit, and it is why multi-fidelity methods belong to the expensive regime — provided one condition holds.

Fidelity: the hidden precondition for pruning

Multi-fidelity methods — Hyperband, ASHA, any early-stopping pruner — rest on one assumption: that a cheap, partial evaluation ranks configurations the same way the expensive, full evaluation would. If a config that looks good on one walk-forward fold tends to look good on all of them, you can kill the losers early and spend the survivors' budget on the winners. If the cheap fidelity ranks randomly, early stopping is just throwing away good configs by coin-flip.

So before trusting any pruner, we measure the assumption directly. Our fidelity is the number of walk-forward folds (evaluate on r folds cheaply, or all K folds at full cost), and the fidelity gate computes the Spearman rank correlation ρ between the cheap r-fold objective and the full objective across a sample of random configs. ρ@1 is the correlation when you judge on a single fold — the most aggressive, cheapest fidelity. Here is what that gate reported across the two regimes:

In the single-timeframe regime, one fold ranks configs with a correlation of ~0.03 to the truth — statistically indistinguishable from random. This is not a coincidence; it is the same fact as "no method found an OOS edge." When a strategy has no durable signal, its per-fold performance is mostly luck, so any single fold is a near-random draw and low-fidelity pruning would kill good configs and promote lucky ones. Multi-fidelity is unsafe in the cheap regime here — not because the method is bad, but because the cheap signal is noise. (Our gate flags this and refuses to prune aggressively.)

In the multi-timeframe regime, a real edge exists, and the fidelity picture transforms: ρ@1 rises to 0.43, and by five folds ρ climbs to 0.91. Now one fold carries real ranking information and five folds carry almost all of it. Early stopping becomes safe — a config losing on the first couple of folds is genuinely likely to be a loser. This is the second reason multi-fidelity methods belong to the expensive regime: not just that expensive evals make pruning worth it, but that the expensive regime is where the cheap fidelity finally ranks like the expensive one.

The rule this hands you is blunt and cheap to run: measure ρ before you prune. Fidelity correlation is a two-line computation on a few hundred random configs, and it is the difference between multi-fidelity search accelerating you and multi-fidelity search quietly sabotaging you.

Winning the search is not surviving it

One more honesty note, because this series is about backtests that lie. Our triple-TF champion, sobol→cmaes, was the only method to post a positive holdout — +2.62%, on top of +16.35% in the test window. That is the good news. Here is the caveat: it did not survive statistical deflation.

The champion is the best of tens of thousands of configurations tried across all methods. Under that much multiple testing, a +2.62% holdout is not automatically real. We ran the overfitting gates the whole series leans on — the Deflated Sharpe Ratio with effective-N corrected for the correlation among trials, and PBO via combinatorially-symmetric cross-validation. The champion passed PBO (0.12, comfortably under the 0.2 threshold — its rank is stable across CSCV splits) but its deflated Sharpe collapsed to zero (gate demands ≥ 0.95). Verdict: does not survive.

Read that carefully, because it is the point of the whole exercise. The crossover result is real: smart search won the search in the expensive regime, decisively, and Sobol lost it. But winning the search is a statement about the optimizer, not about the strategy. The best config a good optimizer can find out of an edgeless space is still edgeless — deflation is what tells you which one you are holding. Choosing the right search method gets you the best available answer efficiently; it does not manufacture an edge that was never there. The optimizer and the overfitting gates are different instruments measuring different things, and you need both.

A decision rule you can actually apply

You do not need to re-run our benchmark to pick an optimizer. You need one number: how long does one backtest take? Time a single evaluation of your objective — one full walk-forward pass, all folds — and read off the regime.

• Cheap eval (≲ ~10 ms/backtest): buy throughput. Use scrambled Sobol/QMC or random. The ask/tell tax on TPE/CMA-ES/ASHA will cost you 10–30x your point count for placement that a near-free eval does not reward. Do not bother with multi-fidelity pruning — and if you are tempted, check ρ@1 first; in a low-edge cheap regime it is likely near zero, which means pruning is coin-flipping. Spend the saved engineering time widening the search, not narrowing it.
• Expensive eval (≳ ~100 ms–seconds/backtest): buy sample-efficiency. Use CMA-ES, TPE, or a Sobol-seeded CMA-ES hybrid; the sampler overhead is now a rounding error against the eval. If your eval cost is structured (a slow indicator axis and a fast threshold axis, as in multi-TF), prefer methods that exploit that structure — cascades, drill-downs, anything cost-aware — over methods that treat every dimension as equally expensive.
• In between, or unsure: a Sobol-seeded hybrid (sobol→cmaes) is the robust default. It behaves like breadth-first Sobol early (cheap, no model to fit) and like a smart refiner late, so it degrades gracefully in whichever regime you turn out to be in — which is exactly why it was our expensive-regime champion.
• Before any pruner, measure fidelity. Compute Spearman ρ between the cheap fidelity and the full objective on a few hundred random configs. If ρ@1 is low, do not prune on one fold; raise the minimum resource until ρ clears ~0.5. This costs two lines of code and prevents your accelerator from silently discarding your best configs.
• Whichever wins the search, run the deflation gates. The optimizer's winner is the most overfit-prone object you will produce all week. DSR and PBO, not the optimizer's score, decide whether it is tradable.

Where this connects

This result sits at the center of a few threads this series has been pulling:

• It presumes the engine underneath is honest. The entire cheap-regime advantage exists because our in-process numba engine hits thousands of configs per second with no IPC — the speed ladder is what puts you in the throughput-wins regime in the first place. A slow, framework-taxed engine would put every problem in the expensive regime by default, and you would never see the crossover.
• The expensive-regime exploit is the two-axis cost structure our adaptive-resolution drill-down engine is designed around: localize on the coarse, expensive axis, then exploit the fine, cheap one.
• Every method here is only trustworthy because the engine is leak-free. A search over tens of thousands of trials is the most efficient possible machine for finding and exploiting a look-ahead bug — the "winner" would be whichever config leaned hardest on the leak. Fix the clock first, then search.
• And the champion's fate — winning the search, failing deflation — is the whole argument for treating parameter search and overfitting control as separate stages with separate instruments.

The academic backdrop is the same one the field keeps returning to: Bergstra & Bengio (2012) on why random beats grid; Li et al.'s Hyperband (2018) and the ASHA follow-up (2020) on multi-fidelity; and Bailey & López de Prado (2014) on why the winner of any large search must be deflated before it is believed. None of them prescribe a single best optimizer, because there isn't one — there is a regime, and a cost that selects it.

Takeaways

1. The crossover between random and smart search is set by evaluation cost, not by the algorithm. We inverted the ranking of every method by changing nothing but how expensive one backtest was.
2. Cheap eval → dumb Sobol wins on throughput. On our single-TF engine (~3–4k cfg/s), TPE and ASHA ran 18–30x slower for the same eval count — ~95–154 cfg/s versus ~2,830 for Sobol. At equal wall-clock, breadth beats a better model of the space when each point is nearly free.
3. Expensive eval → smart methods win on efficiency. On the triple-TF problem, sobol→cmaes was the only method to find a positive out-of-sample result (+16.35% test, +2.62% holdout); blind Sobol finished dead last.
4. In the expensive regime, "smart" means cost-aware. The winner exploited the 30–100x gap between the expensive indicator axis and the cheap threshold axis — it did more evals and placed them better, by staying on the cheap axis once the expensive one was pinned.
5. Fidelity is the precondition for pruning. Single-fold rank correlation rose from ρ@1 ≈ 0.03 (single-TF, essentially random) to 0.43 (multi-TF), reaching 0.91 by five folds. Multi-fidelity/ASHA only pays off once the cheap fidelity ranks like the expensive one — so measure ρ before you prune.
6. Winning the search is not surviving it. The champion passed PBO but failed the Deflated Sharpe gate. Pick the optimizer by eval cost; decide tradability with the deflation gates. They are different instruments, and you need both.

Pick your optimizer by what one backtest costs. Then remember that the best answer an optimizer can find in an edgeless space is still edgeless — and let the gates, not the search, tell you which one you're holding.