What is a Backtest and What it is Not

Last year NSE put out a circular saying that algo platforms should not make claims about returns that their systems have generated: neither past returns nor promises of future returns. What was not expected was that many people responsible for implementation confused these for backtests, and started asking for backtest results to be removed.

This misunderstanding stems from the beliefs that (1) backtesting shows how a strategy would have performed in the past, and that (2) the same behaviour can be expected in the future. Both notions are wrong because (1) no backtest can mimic market depth, (2) order book dynamics, (3) slippages, (4) order fills, (5) market impact and perhaps dozens of other details that have a material impact on strategy performance.

Backtesting will always have more pitfalls than benefits, so it will never reveal the true picture of real world behaviour of an algo. Many veterans will vouch that their main beginning struggles were that algos that gave phenomenal backtest results made horrible losses in real world.

"all models are wrong, but some are useful"

 – George E. P. Box

("one of the great statistical minds of the 20th century")

That said, there’s great utility in backtesting if done and analysed properly. Next article in this series will cover the latter; in this article we will cover the analysis. What CAN backtests tell us?

1. Proof of Concept – for starters, a backtest can tell how closely the coded strategy realizes the intended algorithm, the hypothesis on market behaviour. For beginners, this should be the first and most important concern: there are no accidental winners, but the world is strewn with accidental losers. A backtest log should be vetted trade-by-trade, candle-by-candle, and if possible then tick-by-tick to check the logic trader had in mind when they started.

2. Favourable Regimes – all strategies have favourable and unfavourable operating conditions. Trend Following strategies will invariably bleed in sideways markets, and Mean Reverting strategies will invariably bleed in trending markets. Market Making algos will struggle in low liquidity conditions, and certain Arbitrage strategies will struggle in highly liquid conditions. Backtests may not reveal all of the above, but traders should attempt to derive as much as possible from the output.

3. Risk Measures – once unfavourable regimes are identified, trader is in a position to select the risk management techniques required to weather said regimes. Trader should look for the longest losing streaks, size of losses, and maximum drawdowns in order to fix stoploss, take-profit, even an automated kill switch. It can be considered to switch to paper trading mode when a losing streak extends, and to restart real trading when it seems that the conditions have alleviated.

4. Volatility Estimation – large volatility in P&L, even if it seems that an algo makes bank at the end of the day, should be avoided like the plague. If backtests themselves show a wild ride, the algo may go out of control in real markets. Trader can choose whether to change the logic and tweak the calculations (like in case of forecasting models) or tighten the risk parameters (like in case of trend following logic).

5. Trade Frequency – algos which work on smaller timeframes i.e. smaller candle sizes, have a higher probability of over-trading, to the extent that even if they have high hit rate, they can be net losers on account of commissions (brokerage), taxes, and other overheads. While backtests will not reveal the hit-rate accurately, trader can estimate trade frequency and adjust time frames to prevent over-trading. Some strategies may stop working altogether on larger time frames, so trader should be ready to dispose of a hypothesis that overtrades on smaller candles and isn’t profitable on larger ones.

6. Position Sizing – with all of the above ascertained, and assuming everything is in the trader’s favour, position sizing and capital allocation can be estimated, given backtest results. There may be a considerable deviation, but it is possible to improve the sizing and allocation by paper trading, which must be used to fine tune all of the above as well.

7. Relative Comparisons – lastly, trader can compare multiple strategies on the above parameters, given that backtest parameters are kept constant. Strategies that work on vastly different time frames should not be compared, even if the time range is the same.