Selection bias occurs when the sample used to test a strategy is not the same kind of sample the strategy will face in actual trading.

Selection bias occurs when the sample used to test a strategy is not the same kind of sample the strategy will face in actual trading. In systematic trading, that mismatch can enter through instrument selection, date filtering, survivorship effects, or any rule that changes which observations are included in the test. The practical result is straightforward: measured backtest quality can look better than the evidence justifies because the test set has been made easier, cleaner, or more favorable than the future trading universe.

Sonar’s strategy validation research frames this as part of a broader validation problem: a backtest is only informative when the testing process reflects the conditions under which the strategy will be deployed. If the historical sample is selectively constructed, the estimated statistics no longer describe the distribution that matters for future use. In that setting, a high in-sample Sharpe ratio may reflect the sample construction as much as the strategy itself, and any comparison with out-of-sample or live behavior should be treated as a check on whether the original estimate was overstated rather than as confirmation that the in-sample estimate was reliable in the first place.

The mechanism is easiest to see in three common cases:

1. Data filtering. If a researcher removes difficult periods, excludes illiquid names, or otherwise screens the dataset using information tied to the observed results, the tested sample becomes more favorable than the one the live process will encounter. 2. Survivorship. If the sample contains only instruments that remain available through the end of the test window, the historical universe is no longer representative of the contemporaneous universe that existed at each point in time. 3. Look-ahead choices. If universe membership, features, or eligibility rules are defined using information that would not have been known at the decision time, the composition of the tested sample is contaminated by future information.

Sonar’s strategy validation material emphasizes that reliable validation requires separating model development from honest evaluation and using procedures that detect whether reported performance is likely to be inflated by the research process. That is where an overfitting audit becomes useful. Sonar’s backtest overfitting audit is designed to assess whether apparent backtest strength is consistent with genuine signal or whether it is more plausibly explained by multiple testing and selection effects during research. In this context, selection bias matters because it changes the effective population from which the strategy was judged, so the audit helps quantify how much confidence should remain after accounting for the research path that produced the final specification.

A related tool is the deflated Sharpe ratio. Sonar’s glossary describes the deflated Sharpe ratio as an adjustment to the observed Sharpe ratio that accounts for factors such as non-normal returns and multiple testing. That matters here because selection bias often appears together with repeated specification search: researchers try alternative universes, filters, and date ranges, then retain the version with the strongest historical statistic. The raw Sharpe ratio from that process is not a neutral estimate. The deflated Sharpe ratio is intended to judge whether the observed Sharpe is still statistically meaningful after adjusting for these distortions.

Selection bias arises when the historical sample used for evaluation is not representative of the future trading universe, and that mismatch can inflate reported performance statistics. To address it, Sonar’s validation framework points toward strict separation of development and evaluation, explicit auditing for backtest overfitting, and interpreting Sharpe ratios through a deflated lens when the research process involved multiple trials or other bias-inducing choices.

Covered in depth in the Strategy validation & overfitting pillar hub.