Recovery of Pavlovia services and data following the OVH fire

Announcements Banners
23

Het @fgasking,

It will be restored to the point of where things were at when the fire happened.

Best, Thomas

4 Likes
24

Thanks Thomas for the confirmation, much appreciated.

1 Like
25

Amazing work - thank you so much!

1 Like
26

Well thatā€™s a clear vindication of the infrastructure, at least. Everything worked almost exactly as it should (except for them shutting down every building in the data center instead of just the one on fire). Not something you want to have to do, but sort of reassuring that we now have proof pavlovia can survive the worst-case scenario!

4 Likes
27

OVH currently anticipates restoring power to SBG3 on Friday 19th (to my understanding @Moonman157 its SBG1 and SBG4 that they hope to power up on Monday).
@fgasking at that point, we should certainly have access to all data right up until the point it was shut down. In the meantime we are looking into additional partial data that has been backed up incrementally to see what more can be retrieved from those.

1 Like
29

Thanks again to all the Pavlovia team for your hard work on this!

One quick point of clarification - if we need to run participants in the next week (I have some people currently enrolled in a multi-assessment study so weā€™d really like to do this if possible) - and the task code data was established prior to January - should we now be able to do this via https://run.pavlovia.org/? I am currently getting a 404 error for both my tasksā€¦ (e.g. Study game 1)

Thanks

Agnes

1 Like
30

Hey @agnesnorbury,

In that case, the task should run fine, but youā€™ll need to activate it via the dashboard; all restored tasks are set to inactive by default.

Best, Thomas

1 Like
31

Thank you Thomas - that did the trick! I will try to avoid using except where necessary :slight_smile:

1 Like
32

Hi all, I switched from ā€œinactiveā€ to ā€œrunningā€ state and all of my projects are working properly.
Hope this is helpful for whom need to conduct an experiment immediately.

2 Likes
33

sorry to bother you but I canā€™t log in with my credentials??
error: invalid login or password

34

If you registered relatively recently (or changed your password recently), that might explain why. Could that be the case?

35

I registered about 1 1/2 month and never change my password. Will I be able to log in? I have been traying to change my password but I do not get even an email from pavlovia.

36

The only experiments on our dashboard are ones that have been deleted. None of the ones that we were running are there currently. Is this something that will be fixed after all data is retrieved?

38

Yes, thanks, that was a pretty extreme case to ā€œtestā€ on but we got through it. Weā€™ll make sure we can get back online faster if this worst-case ever happened again (surely 2 datacenters could burn to the ground in one decade?!) but in the grand scheme of things, for a young small company, I think we did OK (much respect to Alain for his work, and to the whole team)!

2 Likes
39

@Jon_Aisi my guess is that you registered after 3rd Jan and so your data (including username) are not there yet. If you really need to collect data within the next week, Iā€™d recommend you re-register with a different name (and email if poss?) so that when data are merged they donā€™t clash.

Just to reiterate, anyone that can avoid using pavlovia for the next week (and has used it in the last 6 weeks) should avoid using it for now if possible to avoid data merge problems later

40

ok got it thanks!!! I wonā€™t re register, I rather wait and tray it in a week or so. My thesis in not due yet.
Also thank you and the Pavlovia team for your efforts!!

1 Like
41

OK, Iā€™m going to close this thread at this point I think - we can start another to discuss tips on how to get studies back up (and then how to merge them) where thatā€™s needed urgently.

1 Like
46

I wanted to give you an update on the status of data recovery.

To reiterate what we previously mentioned, our policy was to have nightly replication to a secondary server (in a different building in Strasbourg) and also to a remote site (in Roubaix).

We were particularly unlucky that both the building containing the Primary server and the building containing the Secondary server were affected by the same event: only one of them was destroyed by fire, but the other suffered knock-on effects with the power supply and communications being affected. Although we have restored data from a January snapshot, it increasingly looks like our further incremental backups wonā€™t be available due to a glitch introduced during our migration to Strasbourg. Basically, while we had multiple backups in place, we have been hit by a particularly shocking set of circumstances that is slowing down the full recovery of the data.

When will all the data be recovered?

We are assured that the data in Strasbourg will be available again when the power and networks are restored, although OVH are now saying they expect that to be ā€œfrom Monday 22nd Marchā€ whereas we were hoping it would be sooner than that. When those data are available again, we will start merging the new data/experiments/users with the data from January up to the fire. This will be done by us automatically, as far as possible, but we are being very cautious about how to handle potential conflicts and in those cases where there is uncertainty about how the data should be merged, we will provide you with both sets of files and allow you to merge them manually.

We appreciate your patience and support following this disaster. Again, in future we will keep our Primary and Secondary servers further apart (probably one in Frankfurt and one in Strasbourg) as well as a 3rd off-site backup (in Roubaix).

2 Likes