01 · The noise problem
On-call has stopped trusting alerts.
Static-threshold monitoring fires on every spike, every quiet patch, every "we-expected-this-on-weekends" artifact. Most of those pages are noise — but the rotation has no way to tell which is which until someone reads the dashboard at 2 a.m. By the third burning-pager night, the rotation stops trusting alerts. By the tenth, they stop reading them.
The signal is in there. It is just under a pile of false positives that trained the team to ignore new ones.
02 · Alert-fatigue math
A single noisy stream is enough to lose your team.
Industry data on alert fatigue converges on the same shape: at 10+ alerts per day, on-call response time doubles; at 50/day, the team stops investigating within ten minutes of the page. By the time your service catalog is ten streams deep, the cumulative page volume — even if each stream looks "manageable" — is past the point where humans can prioritize signal.
If every stream you monitor pages 3×/day on thresholds, ten streams generate 30 pages/day. Expected noise floor at that volume is roughly 25 noisy pages for ~5 real incidents. The on-call signal-to-noise ratio is below 1:5.
03 · Baseline vs threshold
Threshold detection is a fixed input. You need a context-aware one.
A threshold says "if value > X, alert". A baseline says "if value is statistically unlike its own recent history, alert". The second is what catches the 4 a.m. revenue dip caused by a deploy nobody paged on — and what skips the scheduled weekend patch your static rule has been paging on for six months.
Constant input, page-driven. Cannot learn seasonality, growth, or growth decays. Surfaces real incidents only coincidentally.
Learns each stream's normal — seasonality, growth, weekly cadence. Flags statistical deviations, not boundary crossings.
04 · What to actually page on
Page on incidents. Everything else becomes an action plan.
The page is for "a human is needed in the next five minutes". Investigation, triage, and post-incident reviews are action plans the system drafts while the team keeps the lights on. Treating every anomaly as a page collapses those two into one — and burns the team's attention budget twice as fast.
- A real revenue-affecting event for which a runbook does not exist.
- A sudden divergence that crosses three correlated streams at once.
- A regression that hampers the on-call rotation’s ability to debug.
See how this drafted-plan shape ships in production. The inventory anomaly library catalogs the exact patterns — low-stockout, SKU velocity drift, supplier delivery lag, dead stock, dead-on-arrival rate, safety-stock breach — and renders the same affected systems, root-cause hypothesis, and next three steps your team will see in their ticket.
05 · MTTR: what moves it, what doesn’t
MTTR halves when the team arrives with a plan, not an alarm.
The bottleneck on incident resolution is rarely the engineer. It is the time between "something looks wrong" and "this engineer knows what to look at". Static monitoring optimizes for the first half of that window — and does nothing for the second. A drafted corrective-action plan moves the second down by 40–60%; the team arrives with affected systems, a likely root cause, and a list of the next three steps. That is what a 15-min walk-through shows your team.
ClearOps ships with the same baseline-vs-threshold logic above, plus a draft-action-plan step on every flagged anomaly. The proof is in a 15-min walk-through.