There is a strand of engineering culture that treats the busy pager as evidence of importance. The war story, the four-hour night, the heroics — these are told as achievements. They are usually descriptions of a system that was allowed to become unpredictable, and of an organisation that decided to absorb the cost in people rather than pay it in engineering.
We started Meridian with a narrow ambition: that being on call should be an ordinary week. Not a week off, and not a week of dread. An ordinary week, in which the phone is carried and almost never rings, and when it does, the reason is real.
The cost is not the interruption
Teams measure on-call load in pages per week, which understates it considerably. The real cost of carrying a pager is the anticipation — the constraint on where you go, how much you drink, whether you sleep deeply, whether you can be somewhere without signal. That cost is paid in full whether or not the pager goes off. It is set by how much you trust it.
I stopped dreading on-call. Meridian decides what is worth waking me for, and it is almost always right.
M. OKAFOR · PLATFORM LEAD · COMPASS
The operative phrase is “almost always right”. Trust is not built by reducing volume. It is built by the alert you received being, every time, something you were glad to have been told. Ten justified pages a month cost a team less than two unjustified ones, because the unjustified pages are what teach you to distrust the device, and distrust is what you carry all week.
Three things that made it boring
- Page on the objective, not the cause. Users experience failed requests, not saturated CPU. A saturated host that harms nobody is a Tuesday task.
- Give every page a decision. If there is no action available at three in the morning, it is not a page. It is a ticket wearing one.
- Close the loop in writing. An incident with no record is one you will have again, with the same surprise.
None of these require our product. They are practices, and teams have run them with three tools and a spreadsheet. What a console can do is make the correct behaviour the default rather than a discipline that erodes the first time a quarter gets difficult.
The claim
Quiet is the product. Not fewer notifications for their own sake — a system that pages nobody while the service burns is worse than a noisy one. What we are claiming is a specific and testable thing: that the number of times you are woken should equal the number of times something is genuinely wrong, and that the gap between those two numbers is where most of the misery of running software at night has been hiding.