All projects

Reliability, 2021

Observability rollout

One metrics and alerting stack across a hybrid estate, with SLOs that matched customer experience.

Control room wall with monitoring screens

The problem

Alerts fired from four tools with no shared severity, and the on-call engineer spent the first ten minutes of an incident finding context.

What I did

  1. Consolidated metrics into Prometheus with federation across clusters and the on-prem estate.
  2. Rewrote alerts around symptoms customers notice instead of CPU thresholds.
  3. Published one dashboard per service, linked directly from the alert message.
  4. Ran postmortems with a fixed template and tracked follow-up actions to closure.

Results

Mean time to resolve
95 min to 28 min
Alert noise
Down 60%
Services with SLOs
12

Want the detail behind this case?