How to compare backtesting engines fairly

Fair backtesting-engine comparison requires an identical, fully specified strategy, tightly aligned data and assumptions, and trade-by-trade fill reconciliation before trusting portfolio-level metrics.

A fair comparison of backtesting engines starts with one rule: hold the strategy constant, then inspect where the engines disagree before interpreting summary metrics.

Sonar’s comparison framework is explicit about this. The comparison setup is built around running the same strategy specification across engines under controlled settings so differences can be attributed to engine behavior rather than to strategy drift or mismatched assumptions. The emphasis is not just on matching headline settings, but on fully specifying the strategy and execution assumptions closely enough that any divergence can be investigated at the fill level rather than explained away at the portfolio-summary level.

That matters because aggregate statistics can conceal implementation differences. If two engines produce different trade sequences, then performance metrics such as returns, drawdowns, or Sharpe-like measures are downstream consequences of those fill differences. In that context, a trade-by-trade reconciliation is the right diagnostic step: align entries, exits, timestamps, prices, and quantities; identify where the first discrepancy occurs; then trace whether the cause is data handling, event ordering, fill modeling, order semantics, or another engine‑specific rule. Sonar’s comparison material supports this workflow by framing engine comparison as a controlled research exercise rather than a leaderboard exercise.

Once fills are reconciled, summary metrics become interpretable. If discrepancies remain, they should be treated as evidence that the engines are not yet being compared on identical terms. If discrepancies disappear after specification and fill alignment, then remaining differences in portfolio statistics are more likely to reflect reporting conventions or metric definitions rather than substantive execution‑model differences.

Sonar’s backtest overfitting audit tool complements this process by shifting attention from a single attractive backtest result to the validity of the research process. Its role is to test whether a result is robust to multiple testing and selection effects, which is directly relevant when comparing engines: if the comparison itself is influenced by repeated tuning, selective parameter choices, or post hoc exclusions, then even a careful engine match can produce misleading conclusions. Best practice, therefore, is not only to harmonize the strategy and reconcile fills, but also to audit the surrounding research workflow for overfitting risk.

The same logic applies to risk‑adjusted performance statistics. Sonar’s glossary entry on the deflated Sharpe ratio describes it as a Sharpe‑ratio adjustment intended to account for selection bias and multiple testing when evaluating strategy results. In an engine‑comparison setting, this is useful as a reminder that a small difference in conventional Sharpe ratios across engines should not be over‑interpreted, especially if many variants, parameterizations, or implementation choices have already been tried. A deflated Sharpe ratio is therefore more appropriate than a naive Sharpe ratio when the comparison sits inside a broader research process with repeated experimentation.

Taken together, the sources support a practical standard for fair engine evaluation:

1. Specify one strategy completely and identically across engines. 2. Keep data and assumptions aligned as tightly as possible. 3. Reconcile fills trade by trade before trusting aggregate metrics. 4. Only then compare portfolio‑level statistics such as drawdown or risk‑adjusted measures. 5. Audit the broader research process for overfitting and selection bias.

Drafted with AI assistance from cited sources. Reviewed and approved by Sonar Sciences Quant & Research Team.