All projects

Reliability, 2021

Disaster recovery automation

Tested failover automation that replaced a manual, once-a-year recovery rehearsal.

Engineer checking hardware in a server room

The problem

The documented recovery plan had not been executed in eighteen months and assumed infrastructure that no longer existed.

What I did

  1. Rebuilt the recovery target as Terraform so it could be created and destroyed on demand.
  2. Automated database restore and DNS cutover with a scripted, observable runbook.
  3. Scheduled quarterly game days with a timed checklist and recorded results.

Results

Recovery time objective
4 h to 45 min
Failover rehearsals
Quarterly
Recovery steps automated
85%

Want the detail behind this case?