Basin runs the settlement path for a few thousand merchants. Nine services, a 99.95% objective on checkout, and a four-person rotation that had been carrying a pager for two years without anyone stopping to ask what it was actually telling them.
By the time they came to us they were taking about two hundred and forty alerts a week and fourteen pages, of which — by their own audit — three were things anyone acted on. The rest were CPU saturation on hosts that were doing their job, retry bursts that resolved before anyone opened a laptop, and a long tail of rules nobody could account for.
The audit came first
Before any of our software was in the paging path, Chen's team spent an afternoon listing every rule they had and marking each one with the last time it had produced an action. Forty-one rules had never produced one. They deleted those before the migration started, which removed about a quarter of the volume without any technology at all.
Half our alerting was archaeology. Somebody had a bad night in 2024 and we had been paying for it every week since.
L. CHEN · SRE LEAD · BASIN
What the objective changed
The remaining rules were replaced with three objectives: checkout success rate, settlement latency, and webhook delivery. Each got a monthly error budget, and paging was attached to the burn rate rather than to any individual number.
The immediate effect was that most of what used to wake somebody now opened an incident room and waited. A slow elevation in webhook retries that would have exhausted the budget in eleven days became a Monday ticket. A saturated host serving traffic correctly stopped being an event at all.
- Week one — one page, at 04:12, for a genuine regression in the settlement path.
- Weeks two to six — between two and four alerts a week, none of them at night.
- Week seven onward — steady at roughly three alerts and one page a week.
The post-mortem that was already written
The week-one page is the one Chen tends to describe. The on-call engineer acknowledged from a watch, rolled back the implicated deploy with the second tap, and went back to sleep. By morning the incident room had closed itself and the draft was in Linear: the timeline, the deploy that preceded the burn by nine minutes, the three commits in it, and the budget arithmetic.
The review was finished and merged before standup. Basin's completion rate for written incident reviews went from a little under half to every incident in the window, which Chen attributes almost entirely to not having to reconstruct the timeline by hand.
What did not improve
Basin depends on an upstream card processor with its own reliability problems, and roughly one page a fortnight is still that processor failing. Nothing about the signal model makes those quieter, and Chen is clear that it should not — the budget is genuinely burning and somebody genuinely needs to know.