Hi everyone,
our teams are getting worried about LLM/“AI” bots taking part in our online studies, so I would like to start a thread about practical tips to prevent or at least detect that. I hope this is the right place for this topic. If there’s a better forum for it, please let me know, and I’m happy to continue the discussion elsewhere.
To illustrate the threat, Westwood (2025) built an AI agent that completed surveys on its own, passed almost all standard attention checks, and gave believable, consistent answers. It could also be instructed to bias results, and it sometimes guessed the study’s hypothesis and answered in line with it. The author sees in-survey checks as a losing battle, partly because harder checks may exclude real participants. But does the community agree here?
I’d like to use this thread to collect practical tips for preventing or detecting AI bot participation in Pavlovia surveys and PsychoPy experiments. If you share a tip, it would be great to include:
- What you did (for example, a type of question, a timing check, or a data check)
- Why it helps against bots, and whether it might also flag real participants by mistake
- A reference, if you know of research or a guide on it
Experiences are welcome too, such as suspicious data you noticed and how you handled it.
In my view, this public forum could be a good place to share experiences, practical tips, and useful references, so that everyone can learn from them, including researchers who are new to online studies. There is some natural tension with discussing countermeasures publicly, as this could be used by people who want to cheat on studies. Westwood resolved this by sharing his analysis code but withholding the scripts that enter answers into surveys automatically. I guess we’ll have to see if this discussion is deemed useful. And if someone finds a weakness in a specific platform, I would suggest reporting it privately to the platform team first, if they offer responsible disclosure.
I’m looking forward to learning from you!
Reference
Westwood, S. J. (2025). The potential existential threat of large language models to online survey research. Proceedings of the National Academy of Sciences, 122(47), Article e2518075122. https://doi.org/10.1073/pnas.2518075122