Backtesting a Strategy
Testing ideas on historical data without fooling yourself · 18 min
Builds on CAPM/Beta (Lesson 1 of this track) and comfort writing basic Python loops from the Playground.
What Is Backtesting?
A backtest is a simulation: what would have happened if you'd run this strategy on historical data instead of just having the idea? It's the main tool quants use to vet a systematic, rules-based strategy before putting real money behind it.
The question a backtest is really answering is "given these exact rules, what return and risk would I have gotten over this historical stretch?" Simple enough in theory. In practice there are a startling number of ways to accidentally fool yourself, and a huge chunk of a professional quant's job is just staying paranoid about all of them.
A Simple Backtest Framework
Every backtest needs to nail down the same handful of things:
- Universe: which stocks or assets is the strategy even allowed to trade?
- Signal generation: what triggers a buy or sell — say, the 50-day moving average crossing above the 200-day?
- Position sizing: how much capital goes into each position?
- Execution assumptions: what price do you assume you actually get? Next-day open is realistic; assuming you traded at the same-day close is a classic way to quietly cheat.
- Transaction costs: commissions, the bid-ask spread, market impact — all of it eats into returns.
- Performance metrics: total return, Sharpe ratio, max drawdown, and so on.
The Holy Trinity of Backtest Metrics
Total Return: the headline number — how much did the strategy make? Useful, but close to meaningless without knowing the risk taken to get there.
Sharpe Ratio: return adjusted for that risk. Sharpe = (Strategy Return − Risk-Free Rate) / Strategy Volatility. A strategy worth taking seriously usually targets a Sharpe above 1.0 before costs.
Maximum Drawdown: the worst peak-to-trough decline the portfolio suffered during the test. A strategy that compounds at 20% a year but occasionally craters 60% is far harder to actually stick with than the headline return suggests — most people would panic-sell at exactly the wrong moment.
Max Drawdown = max(Peak − Trough) / Peak
Professionals track the relationship between the two closely — the Calmar ratio (Annual Return / Max Drawdown) captures it in one number. Anything above 1 is considered respectable.
A strategy that quietly compounds at 15%/year but once dropped 55% from peak to trough is a much harder hold than the return alone suggests. Play with the numbers and see how fast drawdown scales.
The Deadly Sins of Backtesting
1. Look-ahead bias: letting your signal see data it wouldn't actually have had yet. Generating today's trading signal off today's closing price only works if you own a time machine. Every signal has to use only what was genuinely available before the trade fired.
2. Survivorship bias: testing only on companies that are still around today. Most historical databases quietly drop names that went bankrupt or got delisted, which skews everything upward — you're only ever grading the survivors, never the failures.
3. Overfitting (data mining): running thousands of parameter combinations and reporting whichever one happened to win. Test enough variations and something will look brilliant purely by chance. A real edge holds up across a range of nearby parameters — it shouldn't collapse the instant you nudge one number.
4. Transaction cost underestimation: a high-frequency strategy can look wildly profitable on paper and then get entirely eaten alive by bid-ask spreads and market impact once it meets reality.
5. Regime dependence: a strategy tuned to 2010–2020 conditions can quietly stop working once interest rates or market structure shift, as many did in 2022–2023. Always test across more than one market regime before trusting the result.
Walk-Forward Testing
The standard defense against overfitting: split your data into an in-sample period, where you build and tune the strategy, and an out-of-sample period you don't touch until you're already satisfied with what you built. Peek at the out-of-sample data even once while tuning, and you've unconsciously started fitting to it too — the whole point of the split quietly evaporates.
Walk-forward optimization takes this further: train on a rolling window of history, trade forward one period with those parameters, then slide the window ahead and repeat. It's closer to how live trading actually works, which makes it the more honest estimate of real performance.
When a Backtest Is "Too Good"
A Sharpe ratio above 3 in a backtest is almost never real — it's usually overfitting, look-ahead bias, or costs that got left out. Even elite hedge funds are typically targeting a Sharpe of 1–2 after real trading costs in live markets. Treat any backtest claiming 5+ with real suspicion until it's survived contact with the real world.