A publication date is not a knowledge date

A backtest answers one precise question: if the engine had run that day, what would it have decided? The entire value of the answer rests on one detail — what the engine knew that day.

It is also the exact point at which a backtest turns flattering without anyone noticing. A replay that sees, even by a few hours, what the live system had not yet received produces results better than reality. And the flaw raises no alarm: the tests pass, the curves look good, and the published figure is wrong.

We went looking for ours. We found it.

The defect

Every event in our database carries two timestamps: when the article was published, and when we received it.

When our replay reconstructed the state of the world at a date T, it only bounded the first one. So it counted every event published before T — including those that had not yet reached us.

Our news sources do not deliver instantly. The free tiers we use hold their content back: one of our sources delivers with a little over 24 hours of delay, another with a little over 12. These are not incidents, they are the terms of service.

The result: 65.3% of our corpus arrives more than 24 hours after publication. The replay saw every event between 12 and 24 hours before the moment the real system would have known about it.

What it was worth, measured

We swept 365 days, comparing two counts on the same database: the one bounding publication alone, and the one also bounding receipt.

Ratio between the two counts	×1.72
Days whose count changes	156 out of 365 — 42.7%

Overcounting does not, by itself, manufacture a false decision. It only does so by pushing a count over a threshold. Several of our rules fire when the number of events exceeds a given figure, and that is where the measurement becomes telling.

Rule threshold	Spurious trigger days
at least 2 events	11.1%
at least 3 events	19.0%
at least 5 events	36.3%

The more demanding the rule, the more it was fooled. At five events required, more than a third of the days on which the replay saw it fire would not have existed under real conditions.

It makes sense once written down: clearing a high threshold requires a concentration of events that only the artificial pile-up of several hours of advance could produce.

The fix fits in one word

Four queries, across three functions, each given one extra condition: only count what had already been received.

The phrasing that follows is more useful than the code: the window and the visibility are two different things. The window is read on the publication date — what period is this event about. Visibility is read on the receipt date — did we have it?

Conflating the two means answering "what had happened?" while believing you are answering "what did we know?".

The guard matters more than the fix

Fixing three functions is easy. Preventing the fourth from being written tomorrow with the same defect is another matter.

So we wrote a test that does not check a result but the text of the code. It reads the two files concerned, isolates each query, and fails if one bounds the publication date without also bounding the receipt date.

That is unusual, and deliberate. A function can be correct today while a new data reader reintroduces the defect tomorrow, in another file, with nobody making the connection.

Because that is exactly what had happened. The delivery delay of one of those sources had been measured here in July and written down in our engineering log during an outage — the outage had lasted less time than the source's delay, and had therefore gone unnoticed. The fact was known. It simply was not anywhere someone would encounter it while writing a query.

What holds a lesson in place is not memory. It is the guard.

We declared a break in the series

A fix like this breaks comparability. Numbers produced by the old pipeline cannot be compared with those from the new one.

We neither withdrew nor rewrote them: they were true for the pipeline that produced them. They now carry their break, and the first replay after the fix produced the new benchmark, published alongside the old one, never in its place.

Before launching that replay, we wrote down the direction we expected the effect to take: downwards. Fewer visible events, fewer threshold crossings, fewer decisions. Writing it beforehand meant forbidding ourselves from adjusting it afterwards.

The result surprised us

Cadence did fall, from 64 to 62 decisions per year. The expected direction is confirmed.

But the magnitude is −3.1% — for a defect that overcounted events by a factor of 1.72.

The explanation lies in the engine's refusals: the look-ahead was massively inflating events that would not have cleared the other filters anyway — in that replay, 709 abstentions came from the crisis filter alone. The bottleneck was not where the defect was.

In other words: this fix was a debt of honesty, not a performance lever. And it had to be measured to know that — had we announced an effect without measuring it, we would have been right about the sign and wrong by a wide margin about the size.

As for the accuracy of decisions, it moves from 67.6% to 62.5% between the two replays — a gap that is not significant given the sample sizes. We explain why in another article.

The next day, the same trap, avoided

The following day we were connecting a new family of macroeconomic data. An observation there is published five to six days after the date it measures.

Wiring that table into our dated reads would have manufactured exactly the same defect, on a larger scale. This time we saw it before writing the line — and the table now carries three distinct timestamps: the date of the measurement, the date of its first publication, and the date we received it.

Where the publication date cannot be known, it stays empty. Never guessed. An empty cell says "we do not know"; a cell filled by estimation says something false with confidence.

The full measurement report is readable online, with its method, its numbers and its limits — including this one, which we wrote ourselves: the report does not measure the effect on decisions issued, only on counts. The following replay measured that, and contradicted us on the magnitude.

A backtest whose look-ahead nobody has gone looking for is not a cautious backtest: it is a backtest whose error is unknown. Going looking for ours cost us a day's work and two decisions of announced cadence. That is a far lower price than publishing a result you cannot defend.