How to keep a strategy research log

A complete strategy research log records every tested variant, including failures, because the number and structure of trials are necessary to interpret the final result. Full trial histories support Sonar's overfitting audit workflow and the use of the deflated Sharpe ratio, both of which depend on knowing how much multiple testing occurred. Recording all variants improves auditability, validation, and research decisions by preserving the full search process rather than only the chosen outcome.

A strategy research log is a complete record of every idea, variant, test, and result produced during the research process. It includes the variants that looked promising and the variants that failed. The purpose is not only organization. It is evidence. The total number of trials matters because a strategy result cannot be interpreted correctly without knowing how many alternative versions were explored before that result was selected.

A complete log starts with the research question and the original hypothesis. It then records each strategy variant as a separate entry. Each entry should state what changed, why it changed, what data and settings were used, and what result was observed. If a parameter range was scanned, the log should record the full scan, not only the best setting. If filters, exits, position sizing rules, or asset selections were modified, those changes should be recorded even when they worsened the outcome. A failed variant is still evidence about the search process.

This matters because overfitting is a property of the research process, not only of the final backtest. Sonar describes the path from research to publishing as a process that turns ideas into documented, reviewable research outputs rather than isolated results. A log supports that process by preserving the sequence of decisions and rejected alternatives, which makes the work auditable and easier to review later. A published conclusion is stronger when the underlying trial history is preserved rather than compressed into a single chosen chart or metric [1].

The mechanism is straightforward. When many variants are tested, the chance of finding an apparently strong result by luck increases. If only the winner is kept, the final metric can look more convincing than it really is. If all trials are logged, the final result can be evaluated in the context of the full search. That is the key reason the trial count is part of the evidence.

Sonar's backtest overfitting audit tool is built around this logic. The tool evaluates the degree to which a selected backtest may reflect a search over many alternatives rather than durable signal. To do that, the audit requires the research history, including the number of variants tried and the outcomes across those trials. Recording every variant, including losing ones, therefore enables the audit. Omitting failed variants removes part of the evidence the audit is meant to measure [2].

A complete log also supports use of the deflated Sharpe ratio. Sonar's glossary defines the deflated Sharpe ratio as a Sharpe ratio adjustment that accounts for selection bias from multiple testing and non-normal returns. The concept directly depends on the fact that a researcher may have tested many alternatives before selecting one strategy. The more extensive the search, the more important it becomes to adjust the apparent Sharpe for the number of trials and the distributional properties of returns. Without a record of how many variants were tried, the adjustment cannot be grounded in the actual research process [3].

In practice, this means the research log should capture the count and structure of the search. A useful log records each hypothesis, parameter sweep, feature addition, regime filter, instrument universe change, and execution assumption change. It should also preserve timestamps, code versions, dataset versions, and evaluation windows. These details help distinguish a genuine improvement from a result that emerged after repeated experimentation. They also make it possible to revisit whether the chosen variant was selected after a narrow, disciplined test program or after a broad search through many combinations [1][2].

The log improves model validation because it separates discovery from justification. During discovery, many variants are explored. During validation, the researcher needs to know what was explored in order to judge whether the selected model was a rare survivor of a large search. Full trial histories help reviewers ask better questions. Was the final rule stable across nearby parameter values. Did related variants fail. Was the apparent edge concentrated in one branch of the search tree. These questions are hard to answer if only the final specification survives [2][3].

The log also improves decision-making. A complete history shows whether progress came from a consistent mechanism or from repeated opportunistic tweaks. It reveals dead ends, which can prevent teams from retesting the same failed ideas later. It helps analysts compare branches of research fairly because each branch includes both attempts and outcomes, not only the best surviving example. This creates a better basis for deciding whether to continue, pause, combine, or retire a line of research [1].

A practical logging format can be simple. For each trial, record the date, hypothesis, exact strategy definition, changed variables, data sample, execution assumptions, performance metrics, validation notes, and next action. Add a trial identifier so that related variants can be grouped into families. Keep failed runs in the same system as successful ones. Do not delete unpromising results. The point of the log is to preserve the denominator as well as the numerator.

The core principle is that robustness cannot be judged from the selected strategy alone. It must be judged in the context of the search that produced it. A complete research log provides that context. By recording every variant, including failures, it creates the evidence needed for overfitting audits and for multiple-testing-aware metrics such as the deflated Sharpe ratio. The trial count is therefore not administrative detail. It is part of the statistical meaning of the result [2][3].

Drafted with AI assistance from cited sources. Reviewed and approved by Sonar Sciences Quant & Research Team.