Alerts that actually fire: rebuilding the notification pipeline

An alert that arrives late is worse than no alert, because it arrives with the authority of a system and the information of a rumour. Our old pipeline had a median delivery of 5.8 seconds from bar close, a tail that occasionally stretched past a minute, and a small but non-zero rate of duplicate and missed notifications around venue reconnects. All three had the same root cause: the pipeline was polling.

The old design evaluated every active alert rule on a fixed cadence, re-reading each rule’s inputs to decide whether it had triggered. That is simple and it scales terribly. As rule count grew, each sweep took longer, and the cadence had to lengthen to keep up, which pushed latency up for everyone including the users with a single simple rule.

The rebuild inverts it. Rules are compiled once into a dependency graph keyed by the series they read. When a bar closes, we walk only the rules that actually subscribe to that symbol and timeframe, evaluate them against a bar that is already in memory, and emit. Work per bar became proportional to the rules that could possibly have changed, rather than to every rule in the system.

Evaluating on completed bars only was the second decision, and it was contentious internally. It means a rule cannot fire mid-bar on a wick that later reverses. Some users genuinely want that behaviour, and we now offer it as an explicit intrabar option, but the default is bar close because the overwhelming majority of "my alert fired and the setup was gone" reports were exactly this: a condition that was true for four seconds inside a bar that closed somewhere else entirely.

Delivery is idempotent end to end. Every notification carries a deterministic identifier derived from the rule, the symbol and the bar timestamp, and every channel deduplicates on it. A worker that crashes after emitting but before acknowledging replays safely, which is what finally eliminated the duplicate storms around crypto venue reconnects — the same alert can now be produced twice and delivered once.

Bundling was the feature users asked for most and we resisted longest. Several conditions can now compose into one rule with a single notification, because the honest failure of the old system was not latency at all; it was that a setup needing four confirmations produced four alerts, three of which were noise. One setup, one message. Fewer notifications turned out to be the feature.

Where it landed: median delivery of 0.9 seconds from bar close, a 99th percentile under 3 seconds, and duplicate rates that no longer register in our monitoring. None of that came from faster hardware. It came from doing far less work per bar and from being much more careful about what deserves to interrupt you.