A holdout set is the portion of data kept out of the research loop until a strategy is fully specified.

A holdout set is the portion of data kept out of the research loop until a strategy is fully specified. In systematic trading research, that means the rules, parameters, filters, and evaluation procedure are fixed before the holdout is examined. The reason is simple: once holdout data influences any decision, it is no longer a clean out-of-sample test and should be treated as spent.

Sonar Sciences’ strategy-validation framework describes this separation as part of a broader defense against backtest overfitting. The core idea is that repeated trial-and-error on the same history can make a strategy appear stronger than it is, even when the apparent edge comes from noise rather than a persistent effect. A locked holdout helps by reserving data that was not used to design the strategy, so the final evaluation is less exposed to that selection bias and more informative about out-of-sample behavior [1].

This is closely related to the purpose of a backtest-overfitting audit. Sonar’s audit tool focuses on whether a research process has likely capitalized on noise through repeated testing and selection. A strategy that is developed on all available data has no untouched segment left to challenge the final specification. By contrast, a process that keeps a holdout locked until the strategy is final preserves an independent check. That does not guarantee validity, but it improves the auditability of the research path because there is a clearer distinction between model-building data and evaluation data [2].

The same logic matters for the Deflated Sharpe Ratio, which Sonar defines as a Sharpe-ratio adjustment intended to account for multiple testing and non-normality when evaluating whether an observed Sharpe is statistically credible. If researchers optimize repeatedly on the full dataset, the selected strategy’s raw Sharpe can be inflated by search effects. A locked holdout does not remove the need for deflation, but it reduces one major source of inflation by limiting how much the final estimate has been shaped by iterative fitting on the evaluation sample [3].

From a workflow perspective, the implication is strict: the holdout is only useful while it remains uninvolved in decision-making. If the holdout result causes a researcher to revise parameters, alter filters, change the signal definition, or choose among candidate variants, then that data has participated in model selection. At that point, it no longer serves as an untouched final check and should be considered consumed. A new holdout or a new validation design is then needed if the researcher wants another clean out-of-sample assessment [1][2].

Covered in depth in the Strategy validation & overfitting pillar hub.