Three numbers, not one

A success rate on its own is a claim. With its number of observations and its confidence interval, it becomes a measurement — something you can judge, compare and challenge.

That is the difference between "our signals are 60% accurate" and this:

60%, 95% confidence interval [42.3 ; 75.4], across 30 resolved decisions.

or this:

60%, 95% confidence interval [56.8 ; 63.2], across 900 resolved decisions.

The first statement tells you nothing. The other two display the same rate and say very different things: they tell you how many times we were put to the test, and by how much the true figure might differ from the one we display. Across thirty decisions, that "60%" is compatible with a coin toss as much as with a remarkable rule; across nine hundred, it only moves by about three points either side. That is what a large number of observations buys you, and it is exactly what is missing from most performance figures you will encounter elsewhere.

Note, 16 October 2026. Both examples are textbook illustrations. The first version of this article illustrated the point with the result of a replay of our rules over a long market history, with a link to its report and the detail of its verdicts. We have removed both: that replay judges its decisions on market prices licensed for internal use only, a licence that does not allow results derived from them to be published. The report stays in our archive, dated and untouched; it is no longer served, and its address now answers with the reason it is withheld.

What the interval lets you do

It gives you one specific power: knowing when not to believe us.

Take two successive measurements of the same engine, before and after an internal fix — a textbook illustration again:

Measurement	Resolved decisions	Accuracy	95% interval
Before	30	70.0%	[52.1 ; 83.3]
After	30	63.3%	[45.5 ; 78.1]

Almost seven points apart. A press release would write "performance down", or quietly drop the line. The intervals, however, overlap along almost their entire length: with thirty decisions on each side, that gap is indistinguishable from sampling noise. That is what we write in our reports when the case arises — do not read that gap as a decline.

Both figures then stay published side by side. The older one is neither withdrawn nor rewritten: it was true for the system that produced it.

Note, 16 October 2026. The first version of this article illustrated the point with two of our own replays, before and after a fix. We have removed their figures: each of those replays judged a few decisions on market prices licensed for internal use only, a licence that does not allow results derived from them to be published. The criterion for this removal is the licence of the data, never what the result says.

The same reasoning cuts the other way, and that is where it gets uncomfortable for us. Last week one of our internal criteria came back at 100% — across two cases. The matching interval runs from 34% to 100%. In other words, that reading cannot tell complete success from mediocre success. We published it with the caveat spelled out, because a "100%" without its sample size is the most misleading number of all.

The rule that makes this possible

It fits on one line, and it has been in our internal rules from the start: no figure is published without its number of observations and its confidence interval. Publication is unconditional — including when the result contradicts us.

This is not a pose, it is an engineering constraint. A single module computes these intervals for the entire product, and the interface never recomputes one of its own. We use the Wilson interval rather than the normal approximation most spreadsheets offer: the latter produces nonsensical bounds as soon as the sample is small or the rate approaches 0 or 100 — precisely the cases where caution matters most.

Two consequences we accept:

• A threshold is set before the measurement, never after. We write down and record the criterion, then we measure. Without that, the threshold quietly adjusts itself until the verdict passes, and nobody notices — least of all the person writing it.
• A refuted rule leaves the product. This month, one of our seasonal rules was refuted over sixteen years of history: wrong in every period examined. We pulled it from the catalogue. The report that demolishes it is the one described in the note above: its verdict stays on record here, its figures no longer go out.

What you can check yourself

This is the part that matters, and it is what separates a method from a sales pitch.

Our measurement reports are served online exactly as written, with their method, their numbers and their limitations — except those computed on data licensed for internal use only: those answer with the reason they are withheld, rather than vanishing without a word. Our measurement protocol is public, and the verification page gives you the commands to recompute our figures from our raw data and compare them, line by line, against what is displayed.

We are not asking you to trust us. We are giving you the means to check — and when a measurement proves nothing, we are the ones telling you so.

In a field where most of what gets published cannot be verified, that is the only promise we know how to keep.