Backtesting and Simulation

Before a quantitative strategy gets real money, it is tested on the past and stress-tested on futures that have not happened. This module covers how to design and run a rolling-window backtest, how to read its output, the biases that make backtests look better than reality (survivorship, look-ahead and data snooping), and how historical scenario analysis, historical simulation and Monte Carlo simulation, combined with sensitivity analysis, show how robust a strategy really is. Level II item sets typically pair a performance calculation with the diagnosis of a flawed backtest design.

Start this lesson free Practice questions

What you will learn

1. Describe the purpose of backtesting and the steps of a rolling-window backtest, and calculate the main performance measures from its output.

A backtest is a hypothetical simulation of rules on past data, used to test and refine an idea, not a track record.
Steps: strategy design, historical investment simulation (rolling window), analysis of output.
Rolling window: estimate using data up to date t only, hold the portfolio over t to t+1, roll forward.
Annualise: mean x 12, standard deviation x square root of 12, Sharpe x square root of 12 (from monthly data).
Report drawdowns and downside measures (Sortino, CVaR) alongside the Sharpe ratio.

2. Interpret backtest output and identify common problems, including survivorship bias, look-ahead bias, data snooping and unrealistic cost assumptions.

Survivorship bias: using today's survivors inflates returns; use point-in-time universes including dead companies.
Look-ahead bias: using data not yet published (period-end dates, restatements); use point-in-time data and reporting lags.
Data snooping: testing many variants and reporting the best; P(at least one false positive) = 1 - (1 - alpha)^N.
Remedies for snooping: higher t-stat hurdles (about 3.0), holdout samples, cross-validation, economic rationale.
Include costs and shorting constraints; check skewness, fat tails and drawdowns, not just averages.

3. Evaluate a strategy with historical scenario analysis across regimes, and contrast historical simulation with Monte Carlo simulation.

Historical scenario analysis splits backtest results by regime to reveal regime dependence and structural breaks.
A rolling-window backtest yields one realised path; simulation yields a distribution of outcomes.
Historical simulation resamples past periods with replacement; keeps observed correlations and fat tails, but no unseen extremes and no serial dependence.
Monte Carlo draws from a specified, calibrated distribution; can create unseen outcomes but carries model risk.
Use Monte Carlo to test the effect of fat tails and stressed correlations.

4. Explain the inputs and modelling decisions in a simulation and use sensitivity analysis to test how robust a strategy's results are.

Key decisions: variables, horizon, distribution, parameters, number of trials, portfolio rules, outputs.
Simulation sampling error falls with the square root of the number of trials.
Normal distributions understate tails; fat-tailed or skewed alternatives and tail-dependent copulas are common upgrades.
Sensitivity analysis reruns the simulation with changed inputs (distribution, returns, correlations, costs) and compares outputs.
Normal 5% CVaR is about 2.063 sigma - mu; a simulated CVaR far above it signals fat tails.

Module at a glance

A backtest is a hypothetical test of rules on past data: design, rolling-window simulation, output analysis.
Rolling window: estimate with data up to t, hold over t to t+1, roll forward; annualise Sharpe ratios with the square root of 12.
Survivorship bias: today's survivors only; look-ahead bias: data not yet published; data snooping: best of many variants.
With N independent tests at level alpha, P(at least one false discovery) = 1 - (1 - alpha)^N; use higher hurdles and holdout samples.
Historical scenario analysis splits results by regime and exposes regime dependence.
Historical simulation resamples the past; Monte Carlo draws from a chosen distribution and can create unseen extremes.
Sensitivity analysis changes inputs (distribution, returns, correlations, costs); robust strategies keep their conclusions.

This free module has 4 lessons, 16 practice questions with full explanations.

Start the lesson