The null hypothesis is the default claim that a trading strategy has no true edge.

The null hypothesis is the default claim that a trading strategy has no true edge. In this setting, no true edge means the strategy’s expected performance, after costs and realistic assumptions, is not distinguishable from zero. A backtest does not overturn that default on its own. It only produces a sample outcome. The job of validation is to test whether that outcome is unlikely enough under the no edge assumption that rejecting the null hypothesis is justified with room to spare.

Sonar’s strategy validation framework treats this as a statistical question rather than a visual or narrative one. The process starts by defining a testable hypothesis, a performance measure, and a significance threshold before examining the result. A p value then measures how surprising the observed result would be if the true edge were zero. A confidence interval provides the same idea in interval form by showing the range of plausible true effects consistent with the data. If the interval still includes zero, or if the p value does not clear the predefined threshold, the backtest has not established evidence strong enough to reject the null hypothesis. In that case, the result remains compatible with noise rather than a persistent edge.[1]

That logic matters because backtests are vulnerable to data mining and selection bias. When many variants are tried, some will look strong by chance alone even if none has real predictive power. Sonar’s backtest overfitting audit addresses this by checking whether apparent performance survives procedures designed to expose noise fitting. The audit includes out of sample testing and cross validation style checks that separate model discovery from model evaluation. If performance weakens materially once the strategy is tested on unseen data, that is evidence that the original result may have been an artifact of overfitting rather than a rejection of the no edge null.[2]

A related reason to be cautious is that standard performance statistics can be inflated when many trials, short samples, or non normal effects are involved. Sonar’s glossary on the deflated Sharpe ratio describes a bias adjusted metric intended to account for multiple testing and the tendency to overstate significance when the best result is selected from many attempts. The purpose of the deflated Sharpe ratio is not to make a weak strategy look better. It does the opposite. It asks whether the observed Sharpe ratio remains statistically credible after adjusting for the number of trials and related distortions. If that adjusted measure does not support significance, then the ordinary Sharpe ratio is not enough to claim an edge.[3]

Another useful mechanism is comparison against a null distribution built from randomized or bootstrapped returns. Under the null hypothesis, the strategy’s observed outcome should resemble what can arise from chance when return order, labels, or resampled paths are varied in ways consistent with no true signal. Sonar’s validation material uses this logic to frame rarity directly. If the observed performance sits deep in the tail of the null distribution, that supports rejection of the no edge assumption. If it falls in the body of the distribution, the result is not rare under noise and therefore does not establish a meaningful edge.[1][2]

Taken together, these tests answer one question. Is the backtest result sufficiently inconsistent with zero true edge that chance is an inadequate explanation. A legitimate validation case therefore needs more than a positive equity curve or a favorable summary metric. It needs a predefined statistical test, a p value or confidence interval that clears the chosen significance bar, a bias adjusted metric such as the deflated Sharpe ratio, and an overfitting audit showing that the effect persists outside the discovery sample. Without those elements, the null hypothesis has not been rejected with enough margin to make the backtest result meaningful in a statistical sense.[1][2][3]

Covered in depth in the Strategy validation & overfitting pillar hub.