Backtest Overfitting
If you found a strategy that produces a 40% annual return when run over the past 10 years of data, will you get rich? Unfortunately, that dazzling report card is likely an "answer key fitted only to the past," not to the "future."
What Is Backtest Overfitting
A backtest applies an investment strategy to past prices and calculates "how much would I have made if I had done this back then." Up to here, it is a good learning tool.
The problem arises when you change variations of a strategy hundreds or thousands of times against the same past data and pick out only the "best-performing combination." A strategy chosen this way isn't skilled at predicting the future—it has merely molded itself to the chance bumps of that specific past segment. This is backtest overfitting.
In a school-exam analogy, it isn't that you built skill but that you "memorized only the answers to last year's exam." You ace last year's exam (past data), but you collapse on this year's exam (the real world), where the questions change.
In statistics this is called "selection bias under multiple testing." If you try many times and pick only the best, that performance is inflated by luck as much as by skill.
Why It's So Easy to Be Fooled: The Trap in Numbers
The key is the "number of tries." The more times you change the strategy, the greater the chance of finding a combination that looks good purely by luck.
According to analyses by researchers such as Bailey and López de Prado, even a meaningless strategy whose true skill (long-term expected return) is effectively 0 can, if you run about 100 backtests, produce a "jackpot-looking" strategy with a Sharpe ratio of 2.5 by chance. Even though the real value is 0.
A paper published in 2014 in the Notices of the AMS mathematically showed that "even trying a relatively small number of variations easily manufactures a plausible simulation result, and the more tries, the greater the probability of overfitting." The bigger problem is that most strategy write-ups do not disclose "how many times they tried before choosing this combination." So the reader cannot even gauge the degree of overfitting.
The Sharpe ratio is a metric that represents excess return per unit of risk taken. Here, understanding it as a "report card that looks better the higher it is" is enough.
How to Filter Out Overfitting
There is no perfect vaccine, but researchers use representative defenses.
1) In-sample/out-of-sample separation — split the data in two, build the strategy using only the front part (in-sample), and verify it on the back part (out-of-sample) that you never looked at. If performance drops sharply on the back part, it is a sign of overfitting.
2) Walk-forward analysis — a method proposed by Robert Pardo in 1992, in which you shift the window bit by bit and repeat "optimize on the front segment → verify on the next unseen segment," stitching them together. You mimic the real world many times so as not to be fooled by one lucky segment.
3) Counting the number of tries honestly — there are also statistical techniques, like the Deflated Sharpe Ratio (DSR), that reflect how many times you tried and "discount" the performance to see whether it is genuinely significant.
For beginners, the practical principles are simpler. Use few parameters (conditions), always reflect trading costs and taxes, and be suspicious of any "too-perfect report card."
Why "The Past As It Is" Is the Answer
The opposite of overfitting is "not touching the strategy and looking at what actually happened as it is." This site, 《Returns of Almost Everything》, stands exactly at that point.
We do not boast that "some secret combination made 40% in the past." Instead, we show, without hiding, how much you would have now if you had bought good assets steadily over a long time at actual prices—including the maximum drawdown and drawdown period. Showing not a dazzling optimization curve, but the -50% time you had to endure alongside it, is an honest backtest.
For example, comparing lump-sum and recurring-investment results with real data, or checking how much you fell and when you recovered during a financial-crisis segment, carries almost no risk of overfitting. It's a record of "what happened," not a strategy that was "picked out."
One reason the simple rules of long-term and steady investing are strong is exactly this. The simpler the rule, the less room there is to secretly fit it to the past.
Frequently Asked Questions
Q. If the backtest results are good, won't it be good in the future too?
Not necessarily. A backtest is only a record that "it was like this in the past"—it does not guarantee the future. Especially if the strategy was chosen by changing conditions countless times against the same data and picking only the best score, the good result is more likely luck than skill. What matters far more is whether the performance holds up even over a period you never looked at.
Q. How can I notice whether something is overfit?
There are a few warning lights. Be suspicious if the report card is unrealistically smooth with almost no drawdown. Also beware of material with excessively many conditions (parameters), or that doesn't disclose "how many tries it took to choose this combination." If performance deteriorates sharply on the out-of-sample segment, the probability of overfitting is high.
Q. So should I not trust backtests at all?
No. A backtest is an excellent tool for learning and understanding. It is safest when used not for "finding an optimal secret formula" but for understanding "what actually happened, and what the maximum drawdown and recovery period were." The more you look at the past as it is without fitting a strategy to it, the more the risk of overfitting falls.
Related pages
📋 Results are based on historical data; past returns do not guarantee future returns.
📋 This service is provided for educational purposes to help you understand investing, not as investment advice.