All projects
Reliability, 2021
Disaster recovery automation
Tested failover automation that replaced a manual, once-a-year recovery rehearsal.

The problem
The documented recovery plan had not been executed in eighteen months and assumed infrastructure that no longer existed.
What I did
- Rebuilt the recovery target as Terraform so it could be created and destroyed on demand.
- Automated database restore and DNS cutover with a scripted, observable runbook.
- Scheduled quarterly game days with a timed checklist and recorded results.
Results
- Recovery time objective
- 4 h to 45 min
- Failover rehearsals
- Quarterly
- Recovery steps automated
- 85%
Want the detail behind this case?