The Honest Truth About Backtesting: What It Can't Tell You
Backtesting is essential. It's also not omniscient. It tells you what would have happened, not what will happen. Here are the five biggest limitations — and how to use backtesting despite them.
1. Slippage
Backtests assume you fill at the exact price the bar closed at. In reality, market orders slip — especially during volatility. A strategy that shows 2% profit in backtesting might make 1% after slippage in live trading.
The impact is worst on: fast-moving markets (NATGAS, crude oil), news events, and tight stop-losses that get triggered and filled at worse prices than expected.
2. Spread Widening
Backtests typically use historical close prices with fixed spreads. During news events or low-liquidity periods, spreads can blow out to 10-50x normal. Your stop-loss gets triggered at a much worse price than the backtest shows.
Example: EURUSD normally has a 0.5 pip spread. During NFP release, it can spike to 15 pips. If your stop-loss is 10 pips, you get filled 5 pips beyond your stop — on every trade that triggers during that window.
3. Liquidity Gaps
Price can gap over your stop-loss entirely. A weekend gap or overnight gap can blow through your SL and execute at the next available price — which could be significantly worse.
This is especially dangerous on: indices (weekend gaps), commodities (supply shock news), and forex (central bank surprises). No backtest using daily closes captures this.
4. Survivorship Bias
If you backtest on instruments that still exist today, you're only testing the survivors. Delisted instruments — bankrupt companies, cancelled contracts, discontinued products — aren't in the data.
A strategy tested on "all current S&P 500 stocks" excludes every company that went bankrupt during the test period. The backtest looks better than reality because the worst outcomes are missing.
5. Overfitting
The more parameters you tune, the more likely you've fitted noise instead of finding an edge. A strategy optimized to make 500% on historical data may make 0% going forward.
Symptoms of overfitting: too many indicators, parameters tuned to exact decimal values, exceptional backtest results that don't make logical sense, dramatically different results when the test period shifts by even a month.
How to Use Backtesting Despite These Limitations
Focus on robustness over raw return. The metrics that matter:
- ✓ Monte Carlo survival rate (high = robust to trade ordering)
- ✓ Consistent performance across multiple instruments (not just one lucky pair)
- ✓ Reasonable, not extreme, parameters (no "magic" settings)
- ✓ Moderate return with low drawdown beats extreme return with high drawdown
The Strategy Lab is designed around this principle: 1,200 backtests with 500 Monte Carlo simulations each — surfacing robust setups, not overfit ones. Test any strategy on /backtest and judge it by these criteria, not just the P/L number.