Contents
Data
| Item | Years |
|---|---|
| Needed for one test | 12.6 |
| The old loop held | 13.7 |
The previous version of a quant research system I run held about 5,000 daily observations of one asset that trades every day, roughly 13.7 years of its daily closes (my division by 365). Not a bug, not a missing feature: that was the whole evidence base. Then a power calculation told me what a single clean test of the only question the system existed to answer would cost. Telling a true Sharpe ratio of 0.7 from zero takes roughly 12.6 years of data. That is almost everything the loop had, before it paid for any of the trials it ran. With one asset and 13.7 years of history, the loop could print a Sharpe all day and never earn the right to believe it.
The system ran as software and never found a profitable strategy. When I worked out why, the answer was not code quality. It was the problem definition, and the fix was architectural.
The arithmetic, in one paragraph
The standard error of a Sharpe estimate falls with the square root of the observation count, so separating a true Sharpe of 0.7 from zero at a one-sided 5% level (z = 1.645, the lower end of a two-sided 90% interval) with 80% power (z = 0.842) needs ((z_a + z_b) / SR)^2 observations, which is ((1.645 + 0.842) / 0.7)^2 ≈ 12.62 in annual units, about 4,600 daily bars on a market that trades every day, for one test and before any correction for the search itself. I computed that in a Python script from those stated assumptions, and the ~12.6-year figure in my design notes agrees to the decimal. The z-values are my labels for those conventions; the notes carry only the rounded result and the 80% power rule.
Then multiply, because one Sharpe was never the claim. The earlier loop ran 80 trials before I stopped it, and every one of them got to look at the same 5,000 rows, about 13.7 years of them. With about 50 effective trials, the best of them is expected to show a Sharpe near 0.70 on pure noise. So 0.7 was not the threshold for “this works”. It was the expected score of the luckiest coin flip I had already paid for. The Statistical power problem was structural: more parameters and better code could not manufacture the missing independent observations.
What the rebuild changed
The old loop and the new one differ on five decisions, each traceable to that one calculation.
A cross-section instead of one asset. The rebuilt system predicts relative value across many assets on every bar, not direction on one asset per day. This is the direct answer to the power problem: each hourly bar of each asset is a fresh bet, so observations and independent bets grow together rather than one at a time. The question changed shape, from “is this one asset going up” to “which assets are rich relative to their peers, and by how much”, and the old long-or-flat timing question left with it: a pre-registered decision round in the earlier system had already ruled that a timing system was not worth building. My paired piece on a Repeated sampling failure, the post where a validator deflated a 0.843 Sharpe to zero, is the sibling lesson: one winner from a search proves nothing, so widen the search’s denominator instead of narrowing its question. The rebuild is mid-frequency: positions held from roughly four hours to ten days, no order-book racing, no latency game.
Data
| Loop | Decisions per day |
|---|---|
| Old daily loop | 1 |
| Rebuilt hourly loop | 24 |
Hourly bars instead of daily, and their bill. Twenty-four hourly bars replace one daily bar on the same calendar, and each is a decision the loop can take: the decision cadence rises from one per day to one per hour per asset, and the hourly figure in the chart above is per asset before the cross-section multiplies it again. The design lists a cost model as part of the core, so the evaluation does not assume frictionless fills.
One read path, and it is an as-of reader. Every row in the store carries two times: when the event happened, and the earliest moment the value could be known. The as-of reader returns, for each key, the latest row whose available time is at or before the query time, and it is the only read path research code has.
There is no function that gives you “today’s panel” without a clock. Data leakage stops being a discipline you maintain with checklists and becomes a property of the type system: you cannot accidentally read the future if the only door hands you the past. The same reader enforces a sealed window at the front of the data that research cannot see, so research code cannot peek at the exam; only I can open it.
One time engine for backtest and live. The earlier loop had drift risk in exactly the place drift is silent: backtest and live paths drifting apart. In the rebuild, the batch path that computes a feature over history and the streaming path that advances the same feature on the newest observation are the same code, and the equality between the two paths is a test. A single clock in the core answers “what is known now” for both the simulator and the paper driver, which advances the same strategy object against live data. Backtest and live are the same code with two drivers.
Pre-registration and a ledger of every trial. Decision criteria are written before results, and “no edge, do not build” is a registered outcome with the same standing as “promote”. The ledger is an append-only, hash-chained JSONL file: the evaluation API writes the trial record before it returns the metric, so an unregistered result cannot exist. Each record carries the code commit, a data snapshot id, the config, the hypothesis family and whether the hypothesis came from a human or a model. The trial count used to deflate every reported Sharpe comes from that file, and it is budgeted globally at N_eff ≤ 40, with the 80 trials of the old system carried as a named legacy family rather than wiped, because deleting the count deletes the correction.
The honest boundary: the five decisions above are the design in the system’s own documents. The as-of reader is its first milestone and the ledger the next one; the full cross-section, the model training line and the live paper driver come later, and the defaults of the design are still open for my approval. I describe the rebuild as planned, not as run. What I have actually run is the old single-asset loop: it worked as software, produced its 80 trials, and earned this rebuild.
What the number still does not settle
12.6 years is a floor, not a verdict. It assumes 0.7 is the truth being tested; the cross-section and hourly bars raise the observation count, and every added asset and horizon adds trials that the ledger has to charge for, because a bigger denominator only helps if the trial count rises with it. Whether the rebuilt loop can actually separate a real edge from noise at its planned cadence is the question the power analysis in the evaluation layer exists to answer, and I have not run that study yet.
The lesson I keep is transferable beyond trading. Before optimising the analysis, ask what sample would let the analysis win. My first system spent its effort on rules for a market that could not convict or acquit. If one clean test needs 12.6 years of data and you hold 13.7, you can afford exactly one test. Every further trial on the same rows makes the best result less believable, not more, and that is the part a research loop never prints.