P-hacking in backtesting is the practice of repeatedly changing tests, samples, filters, or parameters until a result looks statistically convincing.
P-hacking in backtesting is the practice of repeatedly changing tests, samples, filters, or parameters until a result looks statistically convincing. In strategy research, that often means searching until a Sharpe ratio appears significant. The core problem is selection: once a researcher keeps the best result from many trials, the observed Sharpe ratio is no longer an unbiased reading of skill. It is the maximum of a search process, and maxima are mechanically pushed upward by chance alone. Sonar’s strategy-validation research frames this as backtest overfitting: the more variants, windows, and rules that are tried, the easier it becomes to mistake noise for evidence of edge.【https://sonar-sci.com/research/strategy-validation/】
Sonar’s glossary entry on the deflated Sharpe ratio explains why a raw Sharpe ratio is not enough when many tests were run. A conventional Sharpe ratio can look impressive even if it emerged from broad experimentation, because it does not account for multiple testing, non-normality, and selection bias. The deflated Sharpe ratio is designed to adjust the apparent significance of a strategy after recognizing that the reported result may be the best among many trials rather than a single pre-specified hypothesis. In that sense, the statistic is not just evaluating returns relative to volatility; it is evaluating how surprising the result remains after discounting the opportunity to find a lucky winner among many attempts.【https://sonar-sci.com/research/glossary/deflated-sharpe-ratio】
That directly supports the article’s claim: if a researcher tweaks enough knobs, the distribution of reported Sharpe ratios becomes biased upward because only the most favorable outcomes survive selection. Sonar’s strategy-validation material explicitly treats this as a multiple-testing problem in backtests, where repeated specification changes can create the appearance of robustness without genuine predictive content. The visible consequence is inflated in-sample performance and weaker credibility out of sample.【https://sonar-sci.com/research/strategy-validation/】
Sonar’s backtest-overfitting audit tool is presented as a way to quantify this distortion. The tool evaluates a backtest in the context of overfitting risk and applies corrections intended to reduce the gap between apparent and defensible performance. In practical terms, its role is to take a strategy result that may look strong on a raw basis and reassess it under multiple-testing awareness, so that the remaining evidence better reflects how much of the result could plausibly be due to selection rather than skill.【https://sonar-sci.com/tools/backtest-overfitting-audit】
The same logic explains why apparent performance often declines after overfitting adjustments. A raw Sharpe ratio is a descriptive summary of one chosen backtest. A deflated Sharpe ratio asks a harder question: after accounting for the fact that many specifications could have been tried, how much statistical credibility is left? Sonar’s glossary makes clear that this adjustment is specifically intended to reduce false discoveries in strategy research. When the deflated measure is materially lower than the raw one, that gap is evidence that the original backtest benefited from search and selection effects.【https://sonar-sci.com/research/glossary/deflated-sharpe-ratio】
For validation workflows, the implication is straightforward. A high in-sample Sharpe ratio should not be treated as stand-alone evidence when the research process involved repeated iteration. The relevant question is not only "what Sharpe ratio did the selected strategy achieve?" but also "how many opportunities were there to discover a favorable result by chance?" Sonar’s research and audit tooling both point to the same conclusion: without correcting for multiple testing, backtests can transform randomness into apparently significant skill, and the reported strategy quality will be overstated.【https://sonar-sci.com/research/strategy-validation/】【https://sonar-sci.com/tools/backtest-overfitting-audit】【https://sonar-sci.com/research/glossary/deflated-sharpe-ratio】
Quantitative examples are not included in the cited material.
Covered in depth in the Strategy validation & overfitting pillar hub.