How to make a backtest reproducible

A backtest is reproducible when every result-determining input is fixed and recorded, including pinning the data snapshot, fixing the code version, logging all parameters, and recording random seeds for any stochastic step.

A reproducible backtest is one that can be run again and produce the same output from the same inputs. In practice, that means fixing every input that can change the result and recording enough metadata to recreate the run exactly.

The core mechanism is straightforward. A backtest depends on data, code, parameter choices, and any randomized procedure used during simulation or validation. If any one of those elements changes between runs, the output can change as well. Reproducibility therefore requires pinning the exact data snapshot, fixing the code version, specifying all parameter values, and recording random seeds.

Sonar documents its reproducibility standards in its internal guide.[1]

Pinning the data snapshot means tying the backtest to the exact dataset version used at the time of the run. If the underlying data is later revised, extended, cleaned, or otherwise modified, a rerun against the newer dataset is no longer the same experiment. Sonar's research-to-publishing workflow emphasizes preserving the lineage of a result from research through publication, including the research artifacts needed to reproduce what was originally computed.[1] That framing supports the idea that the dataset used for the run must be captured as part of the record rather than treated as an implicit input.

Fixing the code version means identifying the precise implementation that generated the result. Even small changes in business logic, data handling, defaults, or numerical routines can alter outputs. In a reproducible backtest, that implies storing a code identifier such as a commit hash or equivalent immutable version reference.

Specifying all parameter values means recording every configurable input used by the strategy and the backtest engine. A strategy definition is incomplete if parameter values are left to defaults that may later change or to undocumented choices made during experimentation. Sonar’s Backtest Overfitting workflow focuses on evaluating a strategy across trials and configurations, which only works if parameterized runs are explicitly tracked and comparable.[2] That supports the operational need to log the full parameter set for each run.

Recording random seeds is necessary whenever any step in the workflow uses randomness. Examples include randomized search, resampling, Monte Carlo procedures, or stochastic validation routines. Without a recorded seed, rerunning the same code on the same data with the same parameters can still produce different outputs because the random number generator starts from a different state. Sonar’s Backtest Overfitting workflow discusses repeated trials, multiple tests, and statistical corrections that are sensitive to how candidate results are generated and compared.[2] In workflows that include randomized components, recording the random seed (often referred to as seed capture) is part of making those trials reproducible.

These four controls work together to ensure reproducibility. Data pinning fixes the historical inputs. Code versioning fixes the transformation logic. Parameter logging fixes the explicit choices. Seed recording fixes the stochastic path. When all four are captured, the backtest becomes a fully specified computational experiment rather than a result that depends on unrecorded context.

Sonar maintains a glossary of technical terms, including the Deflated Sharpe Ratio, to clarify statistical concepts.[3]

The same principle appears in Sonar’s discussion of the statistical evaluation. The Deflated Sharpe Ratio glossary entry explains that significance depends on the number of trials, non normality, and other features of how a strategy was tested.[3] Those quantities cannot be interpreted reliably unless the testing process itself is documented in a stable and reproducible way. Reproducibility is therefore not just an engineering convenience. It is part of preserving the meaning of performance statistics by making the underlying experiment reconstructable.

Within Sonar’s workflow, the practical implication is to attach reproducibility metadata to every backtest artifact. The research to publishing material supports preserving the chain from a research run to its published form.[1] The overfitting workflow supports explicit tracking of trials and testing context.[2] Together, these sources support a process where a backtest record includes the data version, the code identity, the full parameter configuration, and any random seed values used in the run.

Drafted with AI assistance from cited sources. Reviewed and approved by Sonar Sciences Quant & Research Team.