Alpha is the part of a return not explained by exposure to a benchmark.

Alpha is the part of a return not explained by exposure to a benchmark. In research practice, that residual can look larger than it really is when a strategy is tuned too closely to historical data.

Backtest overfitting happens when repeated specification changes, parameter searches, or selection across many variants capture noise rather than a stable effect. A backtest can then report strong in sample results even when the same idea weakens materially out of sample. This is one of the easiest ways to manufacture apparent alpha by accident.

Sonar’s backtest overfitting audit is built around this exact problem. The tool evaluates how much performance survives when a strategy is tested outside the segment used for fitting and highlights the gap between in sample and out of sample results. The point of the audit is not that any positive backtest is invalid. The point is that a large in sample alpha can be consistent with a weak or unstable out of sample process once multiple testing and parameter search are taken into account. The mechanism is straightforward. The more variants a researcher tries, the greater the chance that one variant will fit random structure in the sample and present that fit as alpha.

Sonar’s fundamentals research makes the same point with risk adjusted statistics. The deflated Sharpe ratio is designed to adjust a reported Sharpe ratio for non normal returns, limited sample length, and the number of trials or strategy variants considered. That adjustment matters because an apparently impressive Sharpe ratio can arise from luck when many tests are run. After deflation, a meaningful share of apparent alpha often decays because the original estimate did not fully account for selection effects. In that sense, alpha decay is not only a market effect. It is also a measurement effect caused by optimistic inference from the research process.

The practical distinction is between synthetic alpha and persistent alpha. Synthetic alpha is created by excessive tuning to historical data, feature selection on the same sample, or choosing the best result from a large family of related models without a proper correction. Persistent alpha is the smaller subset of effects that continues to appear in forward or otherwise independent testing. Sonar’s research materials frame this as a validation problem. If performance depends on narrow parameter choices or disappears once the fitting period ends, the reported alpha was at least partly an artifact of the backtest procedure.

For systematic strategy evaluation, the implication is simple. Alpha should be treated as a hypothesis about residual skill or signal, not as a direct reading from the best historical run. The relevant question is how much of that residual remains after accounting for multiple testing, parameter freedom, and out of sample degradation. In many backtests, a large share of the reported alpha is better understood as overfitting than as true out of sample predictive power.

Covered in depth in the Strategy research fundamentals pillar hub.