Why you must count every backtest you run

Multiple testing means a strategy’s significance depends on the full number of variants explored, not only the final survivor. Sonar’s strategy-validation research, Backtest Overfitting Audit, and deflated Sharpe ratio materials all indicate that more backtests increase the penalty required to judge a result as credible. If discarded tests are not counted, overfitting risk is understated and significance estimates become unreliable because the search process itself has been mismeasured.

A strategy does not earn statistical credibility only from the test that survives. It also inherits the burden of every variant tried and discarded along the way.

That is the core reason total backtest count matters in strategy validation: each additional test increases the multiple-testing penalty that a surviving result must overcome. If those attempts are not recorded, the significance of the final strategy cannot be computed reliably.

Sonar’s strategy-validation research frames this as a multiple-hypothesis problem. When many variants are tested, some will look strong by chance alone, even if no true edge exists. Standard correction frameworks such as Bonferroni and false discovery rate control are designed for exactly this setting: the more hypotheses examined, the stricter the evidentiary bar for any single survivor. Sonar’s discussion of strategy validation places this problem in the context of backtest overfitting, where repeated search across rules, parameters, filters, and assets increases the likelihood of selecting noise instead of signal [1].

The same logic appears directly in the deflated Sharpe ratio. Sonar’s glossary describes the deflated Sharpe ratio as an adjustment to the observed Sharpe ratio that accounts for non-normal returns, sample length, and importantly, the number of trials or backtests considered. In other words, the metric is not evaluating a Sharpe ratio in isolation; it is evaluating that Sharpe ratio after recognizing that many alternative specifications may have been tested. As the number of backtests rises, the adjustment becomes more severe, because the probability that an apparently attractive Sharpe emerged from search alone also rises [3].

This point is not cosmetic. It means total search breadth directly changes the threshold the selected strategy must clear. If ten variants were tested, the correction is smaller than if one thousand variants were tested. If the true number of attempts is unknown because discarded experiments were never counted, then the required deflation cannot be computed correctly. The result is not just a slightly optimistic estimate; it is an estimate built on missing information [3].

Sonar’s Backtest Overfitting Audit tool makes this dependence explicit. The tool asks for the total number of backtests and uses that input to quantify overfitting risk. Its documentation emphasizes that overfitting assessment depends on how many trials were run, not merely on the final chosen result. That structure reflects the statistical reality: a strategy selected from a larger search space needs stronger evidence than one selected from a smaller search space. If the total count excludes abandoned branches of the research process, the audit will understate overfitting risk because its penalty is being applied to an incomplete search history [2].

This is why discarded variants matter. A failed parameter sweep, an unused filter, a rejected execution rule, or a model family that was explored and dropped all still consumed degrees of freedom in the research process. They affected the probability that at least one specification would eventually look compelling in-sample. From the perspective of multiple testing, these were real trials, even if none became the final strategy [1][2].

Sonar’s materials also connect this issue to overstated risk-adjusted performance. The deflated Sharpe ratio exists because the raw Sharpe ratio can exaggerate evidence when it is taken from a large mining process without accounting for that search. If discarded tests are ignored, the number of effective trials is understated, the deflation is too small, and the resulting assessment of significance is too favorable [3]. Sonar’s strategy-validation framework treats this as a central reason naive backtest results are unreliable after extensive iteration [1].

The practical implication is simple: research logs are not administrative overhead; they are part of the statistical model. To estimate whether a strategy’s observed performance is genuinely unusual, you need to know how many chances you gave yourself to find something that looked unusual. Without that denominator, significance becomes uncomputable in any disciplined sense.

So counting every backtest is not about punishing exploration. Exploration is normal. The requirement is to measure it honestly. Once every trial is counted, tools such as deflated Sharpe-based validation and Sonar’s Backtest Overfitting Audit can apply the appropriate penalty for search. When the count is incomplete, the apparent evidence for the surviving strategy is mechanically overstated, because the survivor is being judged as though it emerged from a narrower search than it actually did [2][3].

The governing principle is straightforward: the survivor must clear the bar set by the full research process, not just by the final chart. Every discarded variant raises that bar. If you do not count those variants, you cannot know whether the survivor truly cleared it [1][2][3].

Drafted with AI assistance from cited sources. Reviewed and approved by Sonar Sciences Quant & Research Team.