Pavlovia is currently down: fire at datacenter

Update: We can confirm that the pavlovia server was fortunately in a building that was not affected by the fire. We are currently waiting to hear when OVH will be able to turn the power back on and we will update you as soon as we know more.

We have some rather shocking news right now that the company (OVH) providing servers supporting pavlovia.org has had a major incident - a fire in one of their buildings - and that has caused pavlovia.org to be shutdown temporarily.

We are currently trying to determine:

  • whether our servers were actually in the affected building (or simply affected by the power outage that resulted)
  • how long it will take to retrieve the data
  • how quickly we can get an alternative server up and running if the original is lost

Sorry for the inconvenience. The datacenter was selected on the basis of its ISO27001-certification, which requires a range of safeguards, including fire safety. Sadly these event can still apparently happen.

We will keep you posted as we find out more and we will be working to get Pavlovia back up and running as fast as we can

14 Likes

Thanks for the details. In this kind of situation I think it helps to understand the why as well as the what.

3 Likes

I hope the block which burned down is not the one Pavlovia services are hosted.

yikes. fingers crossed, hopefully we will be back up and running soon. we’ll contact our participants. hopefully know more later today?

1 Like

Here are some links about the incident:

http://travaux.ovh.net/?do=details&id=49471

5 Likes

Picture of the fire at 1 of the 4 buildings https://twitter.com/xgarreau/status/1369559995491172354?s=09

Despite my best efforts our PI is in a bit of a panic – I know this has been discussed in a number of threads but is there a definitive guide/thread to running tasks on your own server in case it takes more than a few days to get a new pavlovia up and running?

Totally understandable to panic. Unfortunately we don’t have a definitive guide and (although I have not set up a server myself) I would suggest that it could take longer to set up a server than to wait it out to hear more here. Be rest assured that the team will keep you posted as soon as we know information!

1 Like

Oh dear. Hope noone was harmed in the incident, and the datacenters of Pavlovia are safe too. Finger crossed, and will keep praying for the best outcome for all of us!

2 Likes

yes, totally agree, this is what I told my boss but want to have a contingency in case this takes more than a few days!

Hi Becca,

Are you able to reassure users that you have regular offsite backups?

Clearly and understandably it may be some time (hours, days or possibly even weeks – who knows) before online experiments can resume. What feels more important at the moment is knowing what is likely to be there when the site resumes. I have the latest version of all of my experiments of course (from the last sync) but not database data since the last download or commit histories, etc.

5 Likes

According to the news report there was no loss of life, so we can be thankful for that.

10 Likes

Do you know in which building the pavlovia data was stored?

2 Likes

We are currently in the process of contacting OVH and will let you know as soon as we can be sure.

4 Likes

Good news that crucially no-one has been hurt.

Regarding the backups - OVH provide standard backup facilities within their hosting plans, copying to other separate storage systems and triple replication etc. Could we confirm that this was set up for Pavlovia.org as a regular task?

Hi everyone we can confirm that the pavlovia server was fortunately in a building that was not affected by the fire. We are currently waiting to hear when OVH will be able to turn the power back on and we will update you as soon as we know more.

18 Likes

This suggests that they said they would, but I don’t doubt it will probably take them a while to retrieve everything for everyone affected.

3 Likes

Thanks Wakecarter! I thought that was the case, but couldn’t see the GDPR page via Google cache/Web archive.

Data are indeed backed up but we have a lot of data and that will take time to build the server back from that. We moved the server from London to Strasbourg (now looking like a shame!) in Jan 2021 and the data transfer alone took several days. We’re hoping that OVH will be able to reconnect power to our server long before the data-from-backup solution has completed, although we will start moving data to a new server as well, just in case.

8 Likes