How to verify a data vendor's history

Before trusting a vendor dataset, quantitative teams should verify sampled records against exchange archives, run a full gap census, and maintain revision logs.

A market data history earns trust only after direct verification. For quantitative teams, that verification can be organized into three checks: spot checks against exchange archives, a gap census across the full history, and revision tracking over time. These checks answer different questions. Spot checks test whether individual records match a primary source. A gap census measures missing coverage systematically. Revision tracking shows whether the history is stable or keeps changing after delivery.

Spot checks are the first line of defense. The basic mechanism is simple. Select a sample of dates, symbols, and venues from the vendor dataset, then compare those records to exchange published history for the same fields and timestamps. The goal is not to prove perfection from a few examples. The goal is to find direct mismatches early and to estimate whether errors are isolated or structural. Sonar’s cross venue data research illustrates why this matters. That work shows that venue level detail changes the interpretation of market activity because trade distributions, spreads, and quote behavior can differ materially across venues. When venue distinctions matter, a vendor history that silently aggregates, drops, or mislabels venue information can change downstream analysis in ways that are not obvious from headline totals alone. A useful spot check samples both ordinary periods and stressed periods to ensure coverage of different market conditions, because market microstructure can behave differently when activity shifts across venues or liquidity conditions change.

A gap census is the second check. A gap census is a systematic count of missing observations across the dataset. Instead of looking at a few records, it scans the full history and asks where expected data is absent. The mechanism depends on the dataset’s clock and structure. For trades or quotes, the team defines expected continuity by symbol, venue, session, and timestamp grid, then counts missing intervals, broken sequences, and abnormal stretches of inactivity. For bars, the team checks whether every expected bar exists for every instrument and whether session boundaries align consistently. The output should be statistical rather than anecdotal: counts, rates, durations, and concentrations of gaps by symbol, venue, and time period. Sonar’s cross venue data work supports the importance of this census approach because it emphasizes that market behavior is not uniform across venues. A gap concentrated in one venue or one market regime can distort any analysis that depends on cross venue comparisons, liquidity measures, or microstructure features.

Revision tracking is the third check. A revision log records when a vendor changes historical data after initial delivery and how large those changes are. The mechanism is straightforward. Store periodic snapshots or hashes of the dataset, compare each new version to the prior one, and classify differences by field, instrument, date, and magnitude. This turns silent history changes into measurable events. A trustworthy review asks at least three questions. How often do revisions occur. How far back do they reach. How large are the changes relative to the values being studied. Without this log, a team cannot distinguish a stable history from a moving target.

The effect of gaps and revisions should then be measured in research terms, not just operational terms. Sonar’s backtest overfitting audit provides a useful framework for that step. The core idea is that apparent research strength can be overstated when many variants are tried or when the observed result is not adjusted for selection effects. The tool explains the expected maximum Sharpe ratio, the probability that the best backtest appears to beat the benchmark by luck alone, and the Deflated Sharpe Ratio, which adjusts an observed Sharpe ratio using the number of trials, the sample length, skewness, and kurtosis. This methodology is relevant to data verification because gaps and revisions can create exactly the kind of hidden variation that changes which strategy variant appears best. If a dataset revision alters returns, signals, or execution assumptions, then the selected backtest may owe part of its appeal to the specific data version rather than to a robust underlying effect.

The Deflated Sharpe Ratio glossary reinforces this point. It defines the measure as an adjustment to the observed Sharpe ratio that accounts for multiple testing and non normal return distributions. In plain terms, it asks whether an apparently strong Sharpe ratio remains credible after considering how many attempts were made and how the return distribution behaves. When applied after a data audit, it helps separate two questions that are often mixed together: whether a strategy looked good on one dataset version, and whether that result remains persuasive after accounting for trial multiplicity and distributional effects. If data gaps or revisions materially change the observed Sharpe ratio, the deflated measure may also change, which is evidence that the research conclusion is sensitive to data quality.

A practical verification workflow follows naturally from these concepts. First, run spot checks on a sampled set of symbols, dates, and venues against exchange archives. Second, perform a gap census over the full history and summarize missingness by frequency, duration, and concentration. Third, maintain revision logs by snapshotting the vendor history and measuring every subsequent change. Fourth, rerun key research and backtests on the corrected or revised data and evaluate whether the conclusions survive methods such as the overfitting audit and Deflated Sharpe Ratio. If findings change materially across data versions, that is not merely a data engineering issue. It is a research validity issue.

Venue level detail matters, backtest conclusions can be overstated without adjustment for selection effects, and deflating an observed Sharpe ratio is one way to test whether a result remains credible after those adjustments. Taken together, this justifies a disciplined trust process: verify records against exchange archives, count gaps systematically, track revisions continuously, and measure whether any data defects change research conclusions.

Drafted with AI assistance from cited sources. Reviewed and approved by Sonar Sciences Quant & Research Team.