Ongoing effects of excessive AI crawlers, and our mitigations, on Pavlovia

Online experiments
,
1

Pavlovia interface performance issues Over the past week, AI bots have been making huge demands on Pavlovia services (particularly Gitlab). You may have noticed similar issues on sites all over the internet (and a sudden increase in the number of “Are you a bot?” checks) over the last few weeks. Details here

3

Bots and crawlers have always played a part in the traffic to any website, but over May/Jun 2026 the activity of these bots, at least those irresponsible ones that just constantly poll all website endpoints, has gone through the roof. Rather than thousands of requests per minute, which our servers are designed to handle, we now see a roughly constant barrage of thousands of requests per second. That impacts genuine users and occasionally prevent them from being able to access the site. What is more, those insidious bots are designed to evade standard counter-measures (e.g. rate-limiting).

We have been working on a variety of mitigation approaches to reduce the strain on the servers, and occasionally had to temporarily restrict some edge-case functionality and permissions settings on gitlab.pavlovia.org.

While most functionalities have now been restored, some changes remain in place and may slightly impact how you interact with the platform. We wanted to make you aware of these updates, as below, and we will continue to update this thread as further developments occur.

Changes and current status

  • Access to demo experiment code
  • We temporarily restricted access to the source code of public Pavlovia demo repositories, as these repositories were being heavily targeted by the bots. During this period, the View Code option was unavailable for demo projects. This restriction has now been lifted and access has been restored.
  • Viewing CSV files through GitLab
  • You may currently be unable to view .csv files (for example, individual data files) through View Code → Data → [CSV file]. This access remains restricted. However, all data files can still be accessed either by using PsychoPy (or git) to fetch the files to your local machine or using downloaded using the Download Results button on your Pavlovia project page.
  • Setting projects to public
  • As part of our response, project visibility was temporarily limited to Private and Internal (accessible only to users with a Pavlovia/GitLab account). It is now once again possible to set projects to Public.
  • GitLab Explore and Network pages
  • The GitLab pages /explore and /network (which you were unlikely to use, these are different to the Pavlovia Explore page, for instance) are still currently disabled.
  • You can still browse demos and access experiments directly through Pavlovia.org; only access through these two GitLab pages is currently unavailable.

We apologise for any inconvenience these temporary restrictions may have caused and appreciate your patience while we addressed the issue. We expect any remaining restrictions to have minimal impact on normal workflows, and we will continue to monitor the situation closely.

Thank you for your understanding, we hope everything is smooth from here on!

The Open Science Tools Team

8

As an update to this, we have had to continue fighting bots swarming the site. They have been coming in waves and when we shut down one avenue for them they continue searching for others.

We had hoped that Cloudflare anti-bot measures (the challenge where you tick to confirm you aren’t a bot) could protect the site with minimal intrusion, but those aren’t going to work. Cloudflare blocks uploading large experiment changes (anything over 100mb) but we also suspect the success would also be shortlived - there are certainly blog sites describing ways to evade the Cloudflare challenges too.

So, after a few weeks of trying to keep the site as open and friction-free as possible, we have concluded that we need to convert all Public projects to be Internal projects. That means those projects will now require the user to be registered in order to view the materials (but running the study as a participant will still not require a username, just as with Private projects are still visible to participants). Accessing the code and data for all projects including demos will now require a log in, at least until we find a better way to defend against bots.

We are very confident that preventing fully-public access will solve the problem - we were just looking for less restrictive ways to solve it.

Thanks you for your understanding and patience as we deal with these new and evolving challenges. We appreciate it has been a frustrating experience for users (and for us! :rofl: )

Best wishes,
Jon