How to audit someone else's backtest

A rapid backtest audit should ask four questions: how many trials were run, whether costs were modeled realistically, whether the sample spans enough market conditions, and what data selected the parameters. These serve as core checks for overfitting risk and weak validation practice. They justify using tools such as out‑of‑sample testing and deflated Sharpe ratio, but do not provide broad live‑performance case studies or universal numeric cutoffs.

A fast audit of a backtest should focus on four questions:

1. How many trials produced the reported result? 2. Were transaction costs modeled realistically? 3. Is the sample span long and varied enough to support the claim? 4. Did the same data both choose and validate the parameters?

1) Trial count: ask how many variants were tested before the final one was shown

The Sonar Sciences material on strategy validation identifies multiple-testing and data mining as central causes of backtest overfitting. When many rules, filters, lookbacks, thresholds, or asset subsets are tried, the best‑looking result can emerge partly from chance rather than from a persistent effect. The strategy‑validation research page specifically frames this as a validation problem: the more alternatives explored, the less trustworthy a naive in‑sample performance statistic becomes unless it is adjusted for selection bias and tested out of sample.

The glossary entry on the deflated Sharpe ratio is directly relevant here. It describes the deflated Sharpe ratio as a method intended to account for the inflation in Sharpe ratios that arises when many trials are run. In other words, a reported Sharpe ratio cannot be interpreted in isolation if the strategy was chosen from a large search process. An audit should therefore ask:

• How many parameter combinations were evaluated?
• How many strategy ideas were discarded before this one?
• Were alternative universes, timeframes, or filters tried?
• Is the reported performance accompanied by a deflated Sharpe ratio or another adjustment for multiple testing?

If the presenter cannot state the trial count, or treats it as irrelevant, that is a major weakness.

2) Costs: ask whether the edge survives realistic frictions

The Sonar Sciences validation materials also emphasize realistic implementation assumptions. A backtest that omits commissions, fees, spread, slippage, borrow costs, or market‑impact assumptions can materially overstate performance. The backtest‑overfitting audit tool highlights transaction‑cost realism as one of the immediate checks because fragile strategies often depend on gross returns that disappear once frictions are introduced.

A practical audit question is simple: what exact costs were modeled, and how sensitive are results to higher‑cost assumptions? Relevant follow‑ups include:

• Were commissions and fees included?
• Was bid‑ask spread modeled or only mid‑price fills assumed?
• Was slippage estimated, and if so, how?
• Were turnover and holding period reported?
• For shorting, were borrow costs or locate constraints considered?

3) Sample span: ask whether the test period is long enough and diverse enough

The strategy‑validation source stresses that robustness requires evidence across meaningful market conditions, not just a convenient slice of history. A short or unusually favorable sample can produce misleading confidence. The audit tool likewise treats sample span as a first‑pass filter because strategies fit to one regime often fail when conditions change.

Questions to ask:

• How long is the sample?
• How many distinct market regimes does it contain?
• Does it include stress periods, low‑volatility periods, and different trend environments?
• Is the sample size large in terms of number of independent bets, not just calendar time?
• Were the start and end dates justified before the test, or chosen after inspecting results?

4) Parameter selection: ask what data chose the settings

Perhaps the fastest way to expose a weak backtest is to ask whether the same data selected the parameters and validated the final result. The Sonar Sciences strategy‑validation page treats this as a core methodological issue: when model design, feature selection, or parameter tuning are informed by the full dataset, reported performance is contaminated by look‑ahead bias or selection bias, even if labels such as “validation” are used afterward.

The audit tool points toward straightforward checks:

• Were parameters selected using only training data?
• Was there a separate validation set for model or parameter choice?
• Was there a final untouched test set?
• Was walk‑forward or rolling out‑of‑sample analysis used?
• Were any rules added after seeing disappointing periods in the backtest?

Why these four checks work together

• High trial count inflates apparent performance.
• Unrealistic costs can turn a gross edge into a non‑edge.
• Narrow sample spans can confuse regime luck with robustness.
• Data‑driven parameter selection can make in‑sample fit look like predictive power.

A backtest that fails several of these checks at once is exactly the kind of result the Sonar Sciences materials classify as vulnerable to overfitting. In that sense, a systematic audit can expose many weak backtests in minutes: not by proving the strategy will fail, but by showing that the evidence presented is insufficiently protected against known failure modes.

Drafted with AI assistance from cited sources. Reviewed and approved by Sonar Sciences Quant & Research Team.