What was our backup policy (and Disaster Recovery Plan)?
- use a RAID array, instead of a single disk, to make it possible to recover all data if a disk fails within the server
- duplicate data to a Secondary server in case an entire server is destroyed: the servers can be flipped relatively quickly with just a bit of time needed to test everything.
- duplicate data to a disk (not server) in a separate location in case the entire datacenter is destroyed. Rebuilding from this takes time because very significant amounts of data must be copied to a new location.
Now, on this occasion, the Secondary server was in the datacenter that was burnt to the ground (SBG2), and our Primary server was in a different building. One would have thought that was safe enough but, alas, the electricity was shut down across the whole site, not just in the burning building. As a consequence both our servers are down.