The Honest Negative: Tens of Thousands of Backtests, Five Majors, No Robust Edge

Part of the "Backtests Without Illusions" series.

The result we did not want

This series has spent several articles building instruments to catch a lie: look-ahead bias that manufactures a Sharpe of 15 from a one-bar leak, the Deflated Sharpe Ratio that prices the winner of a search, the Probability of Backtest Overfitting that prices the search itself. Every one of those pieces was, in a sense, a rehearsal. This one is the performance: we point the whole apparatus at a real strategy family we actually wanted to trade, and we let it deliver the verdict it was built to deliver — even when the verdict is no.

Here is the honest ending up front. We ran tens of thousands of backtests across five major coins, in dual- and triple-timeframe configurations, searching for a robust edge. We did not find one. Not "we found a small edge and shrank the position." We found nothing that survives contact with the machinery — no configuration that is simultaneously profitable across instruments and defensible under multiple-testing correction. That is not a failure of the experiment. That is the experiment succeeding.

The seductive part — the part that would have gotten a worse-instrumented team to allocate capital — is that a naive read looked genuinely good:

Read that top row the way we first read it: a moving-average-crossover strategy, tuned on ETHUSDT across a dual-timeframe grid, printing +16.35% on data it never saw during the search, and holding a positive +2.62% on a second window we had walled off entirely. If you stop there — and most published backtests stop there — you ship it. The rest of this article is the machinery that told us not to, and why it was right.

Act 1 — The tempting winner

The strategy family is deliberately ordinary: a Hull-moving-average crossover, evaluated on closed bars, with an honest execution model (decide on the close of bar i, fill on the open of i+1 — the one-bar discipline that this series will not compromise on). "Dual-timeframe" means the signal is gated by a slower timeframe's trend; "triple" adds a third, slower still. Each timeframe adds free parameters, and free parameters are exactly what a search converts into apparent performance.

The single-symbol study ran on ETHUSDT. The protocol was already the good kind: a rolling walk-forward split (a warmup window, several in-sample folds, an out-of-sample test window), plus a final holdout window that the search was forbidden to touch until the very end. A Sobol/QMC search explored the parameter space; the survivor was the configuration with the best walk-forward score, and it was carried once — exactly once — onto the holdout.

The survivor looked like an edge:

• +16.35% on the out-of-sample test window — data used only to score configurations, never to fit them.
• +2.62% on the untouched holdout — a second wall, cleared.

This is the moment that decides whether a research process is honest or theatrical. The out-of-sample profit is real in the narrow sense that the numbers are not fabricated and there is no look-ahead leak — we checked. But "real numbers, no leak" is a much lower bar than "real edge." Between them sits the thing this entire series is about: selection. We did not evaluate one strategy and find it made 16%. We evaluated an enormous number of strategies and reported the best one's 16%. The out-of-sample window was clean of look-ahead, but it was not clean of selection — because we chose the winner partly by how it did there. The only instrument that can tell those two stories apart is one that knows how many times we looked.

Act 2 — The deflation: ~37,000 trials, DSR = 0.00

Count the looks. Across the folds, the timeframe combinations, and the parameter grid, the dual-timeframe search evaluated on the order of 37,000 distinct configurations. Every one of them is a draw from the strategy space, and the search kept the maximum. The Deflated Sharpe Ratio article has the full derivation, but the one fact you need here is the False Strategy Theorem (Bailey & López de Prado): the expected maximum Sharpe of N strategies with zero true edge grows with N. At N ≈ 30,000, the best of pure noise sits about four standard deviations above zero purely by selection. Four sigma looks like a discovery. It is the shadow of the search.

So the right question is not "is the winner's Sharpe positive?" — of course it is, you picked the maximum. The right question is "is the winner's Sharpe beyond what the luckiest of 37,000 coin-flippers would post?" That is exactly what the DSR computes: it moves the benchmark from zero up to the noise ceiling implied by the trial count, and reports the probability the true Sharpe clears that.

The ETHUSDT winner's out-of-sample track corresponds to a daily Sharpe of about 0.19. On its own, a daily SR of 0.19 over a long window is a perfectly respectable number. Deflated against ~37,000 trials, it evaporates:

$\text{DSR}\big(\hat{SR}=0.19\,\text{daily},\; N\approx 37{,}000\big) = \mathbf{0.00}$

Zero. Not "marginal," not "0.4, keep an eye on it." The DSR says: given how hard we searched, a daily Sharpe of 0.19 is indistinguishable from the best draw of pure noise. The +16.35% out-of-sample and the +2.62% holdout are consistent, to the precision this test can resolve, with a strategy that has no edge at all and simply won a lottery with 37,000 tickets.

A subtlety worth flagging, because we do not want to overstate the deflation: neighboring points on a parameter grid are near-duplicates, so the raw trial count over-counts the independent looks. Our gate uses the effective number of trials — trials clustered by return correlation via ONC (López de Prado & Lewis) before deflating — precisely so we do not reject a real edge for a bookkeeping reason. Even with that correction folded in, the ETHUSDT winner does not survive. When a result reads DSR 0.00, the effective-N nuance is not going to rescue it; it is deep inside the noise.

That could have been the end. One symbol, one search, deflated to nothing. But DSR failing on a single symbol leaves a loophole a determined optimizer will always try to squeeze through: maybe ETHUSDT is just a hard symbol, and the config is real elsewhere. To close that loophole you have to change the axis of the test.

Act 3 — The decisive test: robustness is across instruments

A single-symbol search has a structural weakness even when it is otherwise perfect: its only out-of-sample axis is time. It can tell you the config held up on a later window of ETHUSDT — but it cannot tell you whether the config learned something about markets or something about ETHUSDT specifically. Overfitting to one instrument is invisible to a test that never leaves that instrument.

So we changed the objective. Instead of "best on ETHUSDT out-of-sample," the cross-instrument hunt asks for generalists: configurations that are good at once across many symbols. The protocol:

• Five liquid majors: ETHUSDT, BTCUSDT, SOLUSDT, BNBUSDT, XRPUSDT — roughly 1.18 million 1-minute bars each, one shared calendar window, one shared set of splits (warmup → K in-sample folds → test → an untouched holdout).
• A robust objective: score each configuration on every symbol's walk-forward result, then rank by the median across symbols. The median is the point — a config that is spectacular on one coin and terrible on four cannot buy its way in with a single outlier. To be selected, it has to be at least middling on most of them.
• A portfolio return matrix for the gates: per-trial daily returns are an equal-weight portfolio across the five symbols (1/S of capital each), giving the T×N performance matrix that the DSR and PBO-CSCV gates consume.
• The holdout is touched once, by the robust champion of each mode only.

This is a strictly harder test than the single-symbol one, and deliberately so. A config can win the ETHUSDT search by exploiting one coin's idiosyncrasy; it cannot win the median-across-five search that way. If a robust edge exists in this strategy family, this is the setup that finds it. If it does not, this is the setup that says so without flinching.

Act 4 — The verdict: both timeframes fail the gates

We ran the cross-instrument hunt in both configurations and gated each robust champion. The gates are the standard two: DSR ≥ 0.95 (deflated against the effective number of trials) and PBO ≤ 0.2 (from CSCV over the performance matrix). Here is the whole verdict, honestly:

Both fail, both gates, both modes. Read each number with the calibration the earlier articles established, because the two gates are saying different things and they agree:

• DSR 0.24 (dual), 0.14 (triple). DSR is the probability the true Sharpe exceeds the noise ceiling implied by the search. We need 0.95. We got 0.24 and 0.14 — barely a one-in-four and one-in-seven chance the edge is even positive once you account for how many configurations were tried. Adding the third timeframe made it worse, not better: more parameters, more ways to fit the sample, less that generalizes. That inversion is itself a fingerprint of overfitting.

• PBO 0.264 (dual), 0.327 (triple). Recall the one fact everyone misreads about PBO (full treatment here): its null is 0.5, not 1. PBO is the probability that the in-sample winner lands in the bottom half out of sample. A trustworthy selection sits near 0; a pure coin flip sits at 0.5. Our 0.264 and 0.327 are below 0.5 — the selection is not literally a coin flip, there is a faint whisper of signal — but both are well above the 0.2 we require to call the selection reliable. And again the triple (0.327) is closer to the coin-flip line than the dual (0.264): more complexity, less generalization.

The two instruments are orthogonal — DSR is parametric and prices the winner, PBO is non-parametric and prices the procedure — and they converge on the same answer from opposite directions. There is no reading of this table on which either strategy clears the bar. The +16.35% that started this whole hunt does not have a robust cross-instrument cousin. It was a property of one coin and one search.

Act 5 — Follow the champion, symbol by symbol

Aggregate gates tell you that a strategy failed; the per-symbol breakdown tells you how, and the how is the most instructive part of the whole study. Take the triple-timeframe champion — the configuration the median-across-five objective actually crowned — and look at what it did on each symbol's out-of-sample test window:

There is the entire illusion, laid bare in five rows. The champion is profitable on exactly one of the five symbols — SOLUSDT, at a gaudy +14.74% — and negative on the other four. It is not a generalist that happens to be weak. It is a SOL specialist wearing a portfolio's clothes. The one big positive is doing all the work; the median objective demoted it below the raw ETHUSDT winner precisely because the median refuses to be fooled by a single outlier, but even the median-selected champion turns out to lean almost entirely on one coin once you unpack it.

The holdout — the window nobody was allowed to optimize against — tells the same story from the cleanest possible vantage: across the five majors, the champion's holdout return is positive on only 1 of 5 symbols. If this were a real edge in the strategy family, it would show up, at least faintly, on more than one instrument's untouched data. It shows up on one. That is the signature of a config that learned a symbol, not a market.

This is why the cross-instrument axis was the decisive test and not merely a nice-to-have. The single-symbol DSR already deflated ETHUSDT to zero. But it took the median-across-five design to diagnose the failure — to show that the apparent edge was never distributed across instruments in the first place, that it was a property of whichever coin the search happened to over-fit that run. On ETHUSDT it was ETHUSDT's; on the median hunt it migrated to SOLUSDT's. The edge moved. Real edges do not move like that.

Why a negative result is the correct result

It is worth being explicit about what we are and are not claiming, because "we found no edge" is easy to misread as either false modesty or a confession of incompetence. It is neither.

We are not claiming HMA crossovers can never work, or that these five coins are unpredictable, or that no dual/triple-timeframe strategy exists. We are claiming something narrower and much stronger: within this strategy family, over this data, at this search intensity, there is no configuration whose apparent performance survives correction for the number of things we tried. Everything that looked like an edge is inside the confidence band of the best of noise. That is a precise, falsifiable, defensible statement — and it is the correct one to publish.

The temptation the machinery defeats is enormous, and it has a name in every other field: the file-drawer problem. Negative results get buried; positive results get written up. In trading the incentive is sharper still, because the positive result you failed to deflate is not just a bad paper — it is capital deployed against noise, real money paying real fees to trade a lottery ticket you mistook for a signal. The look-ahead article showed a leak fabricating a Sharpe of 15; the DSR article showed a search fabricating a Sharpe of 1.63 from pure noise with no leak at all. This article is what it looks like when those instruments are pointed at your own favorite idea and asked to be honest. The apparatus — DSR, PBO/CSCV, effective-N clustering, cross-instrument selection — does not exist to bless your strategies. It exists to stop you from shipping the best of noise as alpha, and the only proof it works is that sometimes it says no.

A team without this apparatus would have shipped the +16.35%. They would have had a clean-looking walk-forward, a positive holdout, no detectable look-ahead leak, and a plausible story. They would have been wrong, and they would not have known it until the live P&L diverged — the backtest-live gap that a negative-but-honest result never has to explain, because it never went live. The value of a rigorous no is measured in the drawdowns you never took.

Provenance

Every number in this article traces to code, not narrative. The cross-instrument edge hunt — the five-symbol load, the shared splits, the median-across-symbols objective, the equal-weight portfolio matrix fed to the gates — lives in scripts/edge_hunt_multitf.py (commit acd84e8) in the backtester repository. The statistical gates it calls — probabilistic and deflated Sharpe, minimum track-record length, effective-N via ONC clustering, and PBO through CSCV, all implemented from scratch on NumPy/SciPy against the primary sources rather than a black-box library — are in scripts/overfit_gates.py (commit 7b966e1), which ships with a self-test that plants a known edge in pure noise and confirms the gates pass it and reject the noise. The single-symbol ETHUSDT study that produced the tempting +16.35% came from the earlier bench_search_multitf harness the hunt imports read-only. Nothing here is a hand-computed figure; the gates are the same code path whether the answer is yes or no.

Takeaways

1. We ran tens of thousands of backtests across five majors, dual- and triple-timeframe, and found no robust edge — and that is the result the machinery is built to produce. A negative result, rigorously established, is a finding, not a failure.
2. A clean-looking out-of-sample number is not an edge. The ETHUSDT winner posted +16.35% out-of-sample and +2.62% on an untouched holdout, with no look-ahead leak — and deflated to DSR 0.00 once you counted the ~37,000 trials behind it. Out-of-sample clears look-ahead; only deflation clears selection.
3. The False Strategy Theorem is the reason. At ~30,000 trials, the best of pure noise sits about four sigma above zero by selection alone. A daily Sharpe of 0.19 is exactly what that lottery pays. You must compare your winner to the noise ceiling, never to zero.
4. Robustness lives across instruments, not just across time. Selecting by the median across five symbols turned a single-coin illusion into a diagnosable one: dual DSR 0.24 / PBO 0.264, triple DSR 0.14 / PBO 0.327 — both fail both gates, and the triple (more parameters) is worse than the dual on every metric.
5. Unpack the champion before you trust it. The triple-timeframe "portfolio" champion was profitable on 1 of 5 symbols (SOL +14.74%; ETH −0.39%, BTC −0.38%, BNB −8.58%, XRP −4.13%) and positive on the holdout for only 1 of 5. An edge that lives on one instrument and moves when you re-search is not an edge — it is an overfit wearing a portfolio's clothes.
6. Publish the negative. The anti-overfit apparatus — DSR, PBO/CSCV, effective-N, cross-instrument selection — is worth having precisely because it sometimes tells you no, and the discipline is to listen when it does.

The strategy we most wanted to work did not work. The instruments we built to catch that told us so, in four independent ways, before a single dollar was at risk. That is the whole point of the series, and this is the article where the point cashes out: the machinery earns its keep the day it stops you — not the day it flatters you.