All projects
Reliability, 2021
Observability rollout
One metrics and alerting stack across a hybrid estate, with SLOs that matched customer experience.

The problem
Alerts fired from four tools with no shared severity, and the on-call engineer spent the first ten minutes of an incident finding context.
What I did
- Consolidated metrics into Prometheus with federation across clusters and the on-prem estate.
- Rewrote alerts around symptoms customers notice instead of CPU thresholds.
- Published one dashboard per service, linked directly from the alert message.
- Ran postmortems with a fixed template and tracked follow-up actions to closure.
Results
- Mean time to resolve
- 95 min to 28 min
- Alert noise
- Down 60%
- Services with SLOs
- 12
Want the detail behind this case?