Contents
On 7 September 2026 a gate in my research loop promoted a candidate with the reason PROMOTE_CANDIDATE: OOS 0.85 >= 0.8, and a validator GO expected. The strategy was a volatility-targeting rule on a crypto perpetual. An independent validation agent ran its own audit scripts that morning and returned the opposite: NO-GO, promote step not approved. It found no look-ahead in the signal and no cheat in the code. It said the number itself was worth nothing, and it showed its work.
The number was an out-of-sample Sharpe ratio of 0.843. The correction that killed it was two lines of arithmetic I will spell out below, because this failure does not need a quant to understand once you have seen it.
What the search actually contained
The ledger behind the decision held 84 rows: seven rules over three crypto perps, from a run block appended three times over. Deduplicated by (strategy, asset, Sharpe), counted with a Python script over the ledger JSON, the 84 rows are 42 distinct candidates. Nineteen finished above zero out of sample, 21 landed at exactly 0.000, 2 went negative, the worst at −0.327. The walk-forward window was 2 years of training against 3 months of test.
Data
| OOS Sharpe | Candidates |
|---|---|
| Above 0.000 | 19 |
| At 0.000 | 21 |
| Below 0.000 | 2 |
The promoted row claimed an OOS Sharpe of 0.847, a CAGR of 0.2891, a maximum drawdown of −0.5268. The benchmark’s top five was that same row three times plus a second candidate twice. The gate’s own ranking was inflated before anyone asked whether the winner was real.
What the validator recomputed
The validator is a separate agent role: it re-runs the backtest through its own scripts, in its own workspace, and answers only GO or NO-GO. Its independent recompute at the frozen cost of 15 bps per side, over t_eff=2070 effective observations, gave Sharpe 0.843, CAGR 0.2872, max drawdown −0.5273, bit-identical across three consecutive runs.
The claimed signature 0.847 / 0.2891 / −0.5268 matched, to four decimals, what the engine produces at a cost of 10 bps. The frozen contract says 15 bps is the only valid cost. The claim that reached the gate had been minted with the cheaper number. The verdict’s cost sweep shows the strategy barely cares which cost you use, but the contract is the contract, and an audit that trusts the claim’s own arithmetic cannot see this.
Data
| Bps per side | Sharpe |
|---|---|
| 5 | 0.850 |
| 10 | 0.847 |
| 15 | 0.843 |
| 20 | 0.839 |
| 30 | 0.832 |
Then came the correction that decided the case.
The selection correction, in plain numbers
The Sharpe ratio of one strategy on one window is a sample mean divided by a sample standard deviation: a Summary statistics quantity with a sampling error attached. The maximum of many such draws is biased upward even when every candidate is pure noise. Under independent normal returns, the expected best Sharpe out of N trials grows like sqrt(2 ln N), which is why “best of a big search” is not evidence of anything. The Overfitting here is not in the parameters of one rule. It is in the act of reporting the winner.
The validator used the deflated Sharpe ratio of Harvey and Liu: DSR is the probability that the observed Sharpe beats the best Sharpe expected from pure luck at that trial count and observation length. The promote contract in the same repository demands DSR >= 0.95. The validator recomputed the values, and cross-checked each against a second, hand-transcribed implementation that never touched the repo’s code:
- DSR at N=7 (the smallest defensible trial count) and t_eff=2070: 0.0
- DSR at N=21, the rows of one run block: 0.0
- DSR at N=84, the advertised search size: 0.0
At either end of the range the deflated Sharpe is zero, and the threshold sits at 0.95. For a DSR of 0.95 at t_eff=2070 you would need a Sharpe of 2.01 with N=7, or 3.04 with N=84. The candidate stood at 0.843.
Data
| Item | Sharpe |
|---|---|
| As promoted | 0.85 |
| Recomputed | 0.843 |
| Needed, N=7 | 2.01 |
| Needed, N=84 | 3.04 |
The report’s own words: the OOS 0.85 is statistically indistinguishable from the selection noise of an 84-row candidate scan. The gate that promoted it had never read a DSR, because the ledger record it read had no DSR block and the gate code had no line that reads one. Nothing in the loop that wrote deflated Sharpes read them either.
This is what the architecture review had warned about one day earlier, in a review of the same lab: no multiple-testing correction anywhere, the trial-count field cosmetic, and a best-of-84 search whose chance-level winner the review put at roughly 1.6 to 1.9 Sharpe. The promotion bar of “OOS >= 0.8” was set as if you could not search your way to it. You can. I just did, with seven rules.
Three things the NO-GO was not about
First, the signal itself looked clean. The validator’s independent future-perturbation test, ten seeds, noise injected after the test window: the signal responded to none of them. A second probe perturbing the test-window prices by 30% lognormal noise left the output unchanged. The verdict records this as a positive finding: no look-ahead found, determinism clean, three runs bit-identical.
Second, the cost realism was not the problem: 0.002 of Sharpe per 5 bps, because the rule holds positions for about 30 days. Third, it was not a sign flip: the verdict shows no reversal of direction across halves. The regime split, though, reads 0.489 for the first half of the OOS window against 0.161 for the second. One observation of one possibility, and a weak second half.
What the validator could not tell me
The honest half of this story is what the audit left open.
It could not tell me the search was really 42 or 84 wide. Those counts come from ledger rows of one run family. The window choice, the parameter grid inside each rule family, and every earlier experiment that informed these seven rules are uncounted trials, and the verdict says so: the real multiple-effect space must also count the window and parameter grid. N=84 is a floor, not the search size. With an uncounted search, no deflation number can be exact; the correction is an honest lower bound on the penalty.
It could not tell me the record was written by the code that produced it. The claim rows were timestamped 06:53:57Z while the script that generated them carries a modification time of 07:33:31Z, after the audit started. The ledger rows have no git revision, no input hash, no schema version. The price file itself changed its md5 between audit rounds while keeping a constant mtime, before stabilising at the sha256 the final runs used. During the audit, the lab’s own ledger module was truncated to 481 lines, deleting the deflated-Sharpe function, then restored. The validator cross-checked its DSR numbers in the windows where the formula lived. I do not know what caused the churn; the cause was not isolated. What I know is that the evidence cabinet had no lock.
It showed that the leak gate is not fail-closed, and that the probe behind it cannot be trusted either way. The mandatory leakage probe in the backtest path rejects this candidate with “signal_fn reacts to FUTURE bars” — and is itself broken: it poisons the last historical row, which the rule’s rolling window legitimately reads, and manufactures a false positive (output size drops from 0.610 to 0.0128 when the poisoned row moves). Production sidesteps the whole probe by passing run_probe=False, so no candidate has ever passed the check its own spec makes mandatory. The agent-side probe does the perturbation correctly. Two probes disagree and only one can be right.
And it could not certify its own ground. The “independent” recomputes run on the same machine, over the same price file and the same backtest engine, differing by scripts and by not trusting the claim. Independent in method, not in data.
What the validator asked for
The gate is dead to me as written. A decision record that cannot see the field it is supposed to gate on is a fail-open gate wearing a checklist. The validator’s correction list says the loop has to fail closed: no promote decision may be minted for a record without a DSR block, the trial count comes from a pre-run manifest rather than a field the code sets itself (Pre-registration, same logic as in my agent-team runs), and the promotion report answers “how many of N passed Holm and DSR”, never “here are the best five”.
The Sharpe of a winner is a Repeated sampling artifact until someone pays for the trials. 0.843 was not a failed strategy. It was a passing grade on a test where the whole class took the exam and you only looked at the top paper.