A confidence interval is a range of values that remains statistically consistent with the data at a stated confidence level.
A confidence interval is a range of values that remains statistically consistent with the data at a stated confidence level. In strategy validation, that range is often more informative than a single reported metric because a backtest is only one sample of what a strategy might experience. A point estimate such as a Sharpe ratio, return, or hit rate compresses that uncertainty into one number. A confidence interval keeps the uncertainty visible and makes the estimate more honest about sampling variation and model risk.[1][2]
The basic mechanism is straightforward. First, a statistic is chosen from the backtest, such as mean return or Sharpe ratio. Then a method is used to quantify how much that estimate would vary if the underlying sample had been different. The result is an interval rather than a point. At a stated confidence level, the interval represents the set of values that are statistically compatible with the observed data under the assumptions of the method used to construct it.[1][2]
This matters in trading research because backtests are noisy and strategy selection is vulnerable to overfitting. Sonar’s strategy-validation research emphasizes that validation should focus on uncertainty, generalization, and the difference between in-sample fit and out-of-sample behavior, rather than treating a backtest summary statistic as exact.[1] The backtest overfitting audit tools likewise frame performance evaluation in terms of robustness checks, resampling, and statistical adjustments, which is the setting in which interval-based reporting is useful.[2]
A confidence interval is wider when the evidence is weaker and narrower when the evidence is stronger. Short samples, volatile returns, and unstable strategy behavior tend to produce wider intervals. Larger and more stable samples tend to produce narrower intervals. That width is not a flaw. It is the visible expression of how much precision the data can actually support. For strategy developers, a wide interval can be more useful than a sharp point estimate because it shows the degree of uncertainty before any live deployment decision is considered.[1][2]
The interval also depends on the construction method. Sonar’s materials on validation and overfitting audits discuss methodology choices and the role of assumptions in statistical evaluation.[1][2] In practice, confidence intervals can be built with parametric methods, which rely on distributional assumptions, or with resampling methods such as bootstrapping, which estimate variability by repeatedly recomputing the statistic on resampled data. The interpretation of the interval is only as credible as the assumptions behind that method. If returns are dependent, nonstationary, or heavily selected through repeated testing, a naive interval can look more precise than the research process justifies.[1][2][3]
This is where interval-based reporting connects to overfitting controls. Sonar’s glossary entry on the Deflated Sharpe Ratio explains that a headline Sharpe ratio can be misleading when many trials, non-normality, or short samples are involved, and that statistical adjustment is needed to account for selection effects and multiple testing.[3] That same logic supports confidence intervals: the reported uncertainty should reflect the real research process, not just the final chosen backtest. An interval built without accounting for repeated specification search may understate uncertainty, while a more careful audit can reveal that the plausible range of performance is broader.[2][3]
Used properly, a confidence interval changes the research question. Instead of asking whether the backtest produced one attractive number, the researcher asks what range of values the data can support and whether that range remains acceptable under realistic validation assumptions. That shift is aligned with Sonar’s strategy-validation framing, which stresses robustness and out-of-sample discipline over reliance on isolated summary statistics.[1]
The concept should also be stated carefully. A confidence interval does not guarantee that the true future performance will fall inside the interval in any single case. It is a statistical statement tied to the procedure used to generate the interval and the assumptions of that procedure.
Interval-based assessment is more robust and more transparent than relying on a single point estimate, especially in the presence of overfitting and multiple testing.[1][2][3]
Covered in depth in the Strategy validation & overfitting pillar hub.