Where this started, and why it collapsed
The project changed direction entirely, from systematic trading to fundamental research.
Claim tested
That a large automated backtest sweep can identify trading strategies with a real edge.
Result
The design cannot distinguish a real survivor from a lucky one, and the data it runs on rewrites itself.
Consequence
Not a fix, but a switch: away from mining price data for patterns, toward using AI to rebuild individual companies' fundamentals from primary filings.
The starting point
The project began with a video in which a former investment banker describes running roughly nine thousand backtests across thirty assets and fifteen years, using an AI assistant to build the testing system. Forty-seven strategies from six families, a few hundred parameter configurations each, put through a six-filter validation funnel, with mean reversion emerging as the only family that survived on average.
The arithmetic checks out. At roughly 300 configurations across 29 tickers you land at about 8,700 backtests — "about nine thousand" is honest rounding rather than inflation. The prompts specify transaction costs, a one-day signal lag, walk-forward windows and bootstrap counts up front, which is more than most published attempts manage.
So the question was not whether the system could be built. It was whether its output would mean anything.
Where it breaks
No parameter fitting happens in-sample. But the six filters then choose winners out of nine thousand candidates using those same out-of-sample numbers — the out-of-sample set has quietly become the selection set. With nine thousand draws you would expect a few hundred to clear a Sharpe threshold of 0.5 by chance alone.
One filter rejects out-of-sample Sharpe above 2.5 on the grounds that the asset did the work. That is precisely what a buy-and-hold Sharpe per asset would tell you, and it is never computed. A null arm of random signals with matched trade frequency would let survival be read against chance instead of against zero.
Requiring that out-of-sample Sharpe not exceed in-sample by more than about 30% caps the upside only. In-sample 2.0 against out-of-sample 0.6 passes comfortably, and a 70% degradation is the textbook overfitting signature.
Reshuffling daily returns destroys autocorrelation and volatility clustering. For a trend strategy, serial dependence is the edge. A block or stationary bootstrap is the correct tool.
Backward-adjusted price series rewrite every historical level whenever a dividend or split occurs, so the series an indicator sees for June 2015 is not the series that existed in June 2015. The ticker universe is also survivorship-selected by construction: chosen in 2026, run from 2010, with no delisted names in it.
Turtle, Parabolic SAR, Supertrend, Donchian breakout and chandelier exits are stop-based systems whose behaviour depends on price touching a level intraday. A daily-close engine understates drawdown control and overstates whipsaw. Those strategies still get scored. The numbers are about a system nobody trades.
What the professional literature already has
The uncomfortable part is that the funnel is groping toward machinery that already exists in closed form. The Deflated Sharpe Ratio adjusts a Sharpe for the number of trials and the variance across them. Probability of Backtest Overfitting, via combinatorially symmetric cross-validation, gives the fraction of the time an in-sample-best configuration lands below median out-of-sample. White's Reality Check and Hansen's Superior Predictive Ability test are bootstrap tests for exactly the "best of N strategies" selection problem. None of it is in the design, and none of it would be added unprompted.
The methods named above, in the order they appear.
| Method | Source |
|---|---|
| Deflated Sharpe Ratio (DSR) | Bailey & López de Prado (2014), Journal of Portfolio Management, 40(5), 94–107 |
| Probability of Backtest Overfitting (PBO), via CSCV | Bailey, Borwein, López de Prado & Zhu (2017), Journal of Computational Finance, 20(4) |
| Same result, in plain English | Bailey, Borwein, López de Prado & Zhu (2014), "Pseudo-Mathematics and Financial Charlatanism," Notices of the AMS, 61(5), 458–471 |
| Reality Check for data snooping | White (2000), Econometrica, 68(5), 1097–1126 |
| Superior Predictive Ability (SPA) test | Hansen (2005), Journal of Business & Economic Statistics, 23(4), 365–380 |
Why I did not simply fix it
I could have. A null arm, a benchmark column, a symmetric degradation filter and a genuinely untouched final window is one more prompt's worth of work.
I stopped for a different reason. Even fixed, the exercise offers no structural advantage. Everyone with a Python environment and a data feed can run the same sweep on the same liquid assets. A better-validated answer to a question thousands of people are asking simultaneously is not an edge; it is a slightly less wrong version of the crowd. The constraint that actually binds is the data, and free daily bars on thirty of the most-watched instruments in the world is not a dataset anyone is neglecting.
That thought is what produced the rest of this project.