Evaluating a strategy on data that played no part in choosing its rules or parameters, held back in advance so that the test is capable of failing.
Partition first, decide the rules, then open the holdout once. Common practice keeps roughly the final third of the sample unseen, and treats a second look as a new experiment on a burnt sample: after one peek, the holdout is training data with a delay.
protocol: partition first · rules frozen · one evaluation · re-use means re-partition
The value of out-of-sample testing comes entirely from the order of operations. Data does not become out-of-sample by being labelled so; it becomes out-of-sample by having had no influence, direct or indirect, on any decision about the strategy. That includes the quiet channels: choosing which ideas to pursue after glancing at recent years, or re-running a sweep because the holdout result disappointed.
Size the holdout to the question. Too short, and one favourable stretch of market carries the whole verdict. The working rule in the audit on this site asks for at least a third of the sample, and for the sample to contain at least one regime change, because a strategy tested in only one kind of market has not really been tested.
One honest evaluation is the entire budget. That is what separates out-of-sample testing from walk-forward validation, which spends the same discipline repeatedly by rolling the boundary forward. When a holdout has been consumed, the alternatives are new data, a new partition, or the humility to report the result as in-sample.
Covered in depth in the Strategy research fundamentals pillar hub.