I wanted to give you an update on the status of data recovery.
To reiterate what we previously mentioned, our policy was to have nightly replication to a secondary server (in a different building in Strasbourg) and also to a remote site (in Roubaix).
We were particularly unlucky that both the building containing the Primary server and the building containing the Secondary server were affected by the same event: only one of them was destroyed by fire, but the other suffered knock-on effects with the power supply and communications being affected. Although we have restored data from a January snapshot, it increasingly looks like our further incremental backups won’t be available due to a glitch introduced during our migration to Strasbourg. Basically, while we had multiple backups in place, we have been hit by a particularly shocking set of circumstances that is slowing down the full recovery of the data.
When will all the data be recovered?
We are assured that the data in Strasbourg will be available again when the power and networks are restored, although OVH are now saying they expect that to be “from Monday 22nd March” whereas we were hoping it would be sooner than that. When those data are available again, we will start merging the new data/experiments/users with the data from January up to the fire. This will be done by us automatically, as far as possible, but we are being very cautious about how to handle potential conflicts and in those cases where there is uncertainty about how the data should be merged, we will provide you with both sets of files and allow you to merge them manually.
We appreciate your patience and support following this disaster. Again, in future we will keep our Primary and Secondary servers further apart (probably one in Frankfurt and one in Strasbourg) as well as a 3rd off-site backup (in Roubaix).