At 02:41 on a Tuesday, the p99 latency on a payments endpoint went from 180 ms to 2.1 seconds and stayed there for six minutes. Nobody was paged. This was the correct outcome, and explaining why is most of what we believe about alerting.
The endpoint had a 99.9% success-rate objective measured over thirty days. Six minutes of degradation consumed roughly 0.4% of the month’s error budget. The service had spent 11% of its budget with nineteen days remaining. There was no trajectory on which those six minutes mattered, and a human woken to look at them would have found a slow dependency that had already recovered.
Thresholds answer the wrong question
A threshold alert encodes a guess someone made about a number, usually in a hurry, usually months ago, and usually never revisited. It fires when the number crosses the guess. It cannot distinguish a spike that resolved itself from the opening minutes of a four-hour outage, because it does not model the difference. Both are simply the number, over the line.
The consequence is familiar to anyone who has carried a pager. You accumulate rules. Each one made sense when it was written. Together they produce a stream of notifications with no ranking, and the on-call engineer becomes a human filter — the least reliable component in the system, operating at three in the morning with the worst possible context.
Every alert you do not act on is training the person who receives it to not act on the next one.
Burn rate is a question about the future
An error budget converts an availability objective into a quantity you can spend. A 99.9% monthly objective is about forty-three minutes of failure. Burn rate asks a different question than a threshold does: at the current rate of consumption, will this service exhaust its budget before the window closes?
That question has a defensible answer, and the answer changes with severity in the way a person’s attention should. A total outage burns the month in minutes and deserves to wake someone immediately. A slow elevation in error rate that would exhaust the budget in nine days deserves a ticket on Monday. The same signal, at two magnitudes, produces two entirely different responses without anyone tuning a rule.
- Fast burn — budget exhausted in under an hour at current rate. Page immediately.
- Slow burn — budget exhausted before the window closes, but not imminently. Open an incident, do not wake anyone.
- Recovered — the burn stopped before it mattered. Record it, page nobody.
This is not a novel idea; it comes out of the site reliability literature and has been practised at large operators for a decade. What has been missing is a console where the objective, the budget, and the paging policy are the same object rather than three systems that have to be kept in sync by hand.
What the engineer saw instead
The Tuesday incident did produce an artifact. At 08:30 the on-call engineer opened Meridian to a recorded event: six minutes of elevated latency, the dependency that caused it, the deploy that preceded it by eleven minutes, and the budget arithmetic showing the month was never at risk. Reading it took under a minute. No decision was required.
We think this is the correct shape for most of what a monitoring system observes. The great majority of anomalies are real, are worth recording, and are not worth a phone call. Systems that cannot express that distinction end up making it anyway — through the engineer who learns to swipe the notification away.