How to detect regime dependence in results

Segmenting a backtest by calendar year and by volatility regime is a practical robustness check. Stable strategies should be examined across time slices and market conditions, and concentration of results in a narrow subset of periods is a warning sign for regime dependence and possible overfitting.

A backtest can look strong in aggregate while depending heavily on one favorable period. A practical way to test for that dependence is to segment results by calendar year and by market regime, then compare whether the same basic characteristics persist across those slices. In Sonar’s validation framework, this kind of segmentation is part of checking whether a result is stable rather than an artifact of a particular sample period or market condition [1].

The core idea is simple: if a strategy’s apparent edge is general, its behavior should not be confined to a single year or a single volatility environment. If instead most of the aggregate result is concentrated in one segment, that is evidence that the backtest may be regime-dependent. Sonar’s strategy-validation material explicitly frames robust research around testing stability across time and conditions, rather than relying on a single full-period summary [1].

A useful first cut is a calendar-year breakdown. For each year in the sample, compute the same metrics used in the full backtest, such as return and Sharpe ratio, and inspect how widely they vary from one year to the next. The point is not that every year must look identical, but that the strategy should show some continuity of behavior rather than a profile dominated by one exceptional interval. Sonar’s validation guidance emphasizes examining subsamples and out-of-sample behavior to detect whether a result survives beyond the period in which it was discovered [1].

A second cut is to segment the sample by volatility state. An example of low, medium, and high volatility states is consistent with the broader validation principle of testing a strategy under different market conditions, but no specific volatility‑state definition or thresholding method is prescribed [1]. Within those user-defined states, the same metrics can be compared to see whether the strategy behaves consistently or whether its aggregate result is mostly an export from one volatility regime.

This matters because aggregate Sharpe ratios can be misleading when they pool together very different environments. Sonar’s glossary on the Deflated Sharpe Ratio explains that observed Sharpe ratios are vulnerable to inflation from selection effects and repeated testing, and that stronger inference requires adjusting for the fact that many trials may have been attempted [3]. Regime concentration adds another layer of caution: even before accounting for multiple testing, a high aggregate metric may be less informative if it is driven by a narrow slice of time or volatility conditions.

That is where an overfitting audit becomes useful. Sonar’s backtest-overfitting audit is designed to examine whether reported results are credible after accounting for the research process and the possibility of selection bias [2]. A strategy that looks attractive in the full sample but weak or unstable when broken down by year and regime is exactly the kind of pattern that should trigger further scrutiny. The audit perspective is not that every variation across segments invalidates the strategy, but that concentration of results in a small subset of conditions is evidence against robustness [2].

The methodological conclusion is that breaking backtest results down by calendar year and by volatility regime is a valid way to learn whether an apparent edge is broad-based or concentrated in one favorable period [1][2].

A disciplined workflow would therefore look like this:

1. Start with the full-sample backtest summary. 2. Recompute the same metrics on calendar-year slices to inspect temporal stability [1]. 3. Define volatility states ex ante, then recompute the same metrics inside each state. 4. Compare whether the strategy’s behavior is persistent across slices or concentrated in a small number of them [1]. 5. Evaluate the result in the context of overfitting risk and selection effects, using tools such as an overfitting audit and metrics such as the Deflated Sharpe Ratio where appropriate [2][3].

If the edge appears in many years and across multiple volatility states, that is more consistent with robustness. If it appears mainly in one standout year or one volatility bucket, the safer interpretation is that the strategy may be regime-dependent and needs more validation before any broader conclusion is drawn [1][2].

Drafted with AI assistance from cited sources. Reviewed and approved by Sonar Sciences Quant & Research Team.