The timing mega-study (2020)

Blog
,
1

Timing mega-study: comparing a range of experiment generators, both lab-based and online

You can finally see the preprint of our long-awaited timing study at psyarxiv.com/d6nu5/ and view the data at https://psychopy.org/timing/2020

Audio, visual and response timing, on many packages, OSs and browsers.
On the desktop: PsychoPy, Psychtoolbox, Presentation, E-Prime, Open Sesame, and Expyriment
Online: PsychoPy, GorillaPsyc, jsPsych, Lab.js, and Testable

Short story:

  • Best timing is on desktop still, but online studies getting better.
  • PsychoPy in Python achieved sub-millisecond precision almost across the board (except Mac visual presentations). Similar timing for Psychtoolbox, E-Prime, Presentation. Less good timing in OpenSesame and Expyriment.
  • PsychoPy online (version 2020.1) achieved RT std dev under 3.5 ms on every browser/OS combo!! (Other packages had online RT std dev under 10ms in most cases)
  • Although variability (std dev) is generally low online, lags are bigger (for all pkgs)
  • None of the online experiment packages really manage good audio timing as yet (consistently across browsers)
7 Likes
2

Hi John,

Thanks for this comparison. Did you expected the MacOS results?! I had for sure it would be the best platform.

Thanks for the comments.

3

We became aware some months ago that, since MacOS 10.13 (10.12 was fine), there was a 1-frame lag and were trying to work out why that was occuring. At least a 1-frame constant lag has minimal effect on most experiments though.

But these measurements show that lag is also variable on some machines and that was new to us.

We’ve got some more work to do to work out why, and whether there’s anything that can be done about it (can this new “Feature” of the operating system be turned off or mitigated) but the fact that none of the packages have a real solution isn’t encouraging.

4

I am very insterested whether running PsychoPy on Linux with RT kernel patchset (i.e. https://wiki.archlinux.org/index.php/Realtime_kernel_patchset) would improve timing on desktop (especially in audio domain). In theory it can have an effect but Python is a complicated garbage-collected beast and I am not sure. It is also the case that it should be rather easy to test it. You just need to compile the kernel with realtime patchset (https://stackoverflow.com/questions/51669724/install-rt-linux-patch-for-ubuntu). What are your thoughts about it?

5

I have now read the paper. And it was clear there, sorry for the useless question, but thanks a lot for the comments.

I do actually think that 1 frame can bother me. I work a lot with attention paradigms (e.g. Posner) and the usual 10 to 20 ms validity effects, so 1 frame can be bad. Especially with that lack of precision. Anyhow, I think I will stick to Windows 10 for now, upon which I have already been working on. Macs prices in Brazil has been prohibitives also… :frowning:

Thanks a lot for the answer and congrats for all the efforts on Psychopy!

Best regards.

6

Hi John,

Do you plan to upload all the experiment scripts? It would be great to see how you wrote the whole script (you only mentioned some details in the paper, but not the whole script).

Best regards,
Max

7

I agree. For all these reasons I’m moving away too. But I still love the days I go back on my lovely retina imac :frowning:

8

Yes, you can already get them: https://osf.io/3kx7g/

We’ve posted all experiment code (for the packages that make that possible) all data files and all anaysis code to get frmo raw data to plots. All subject to change (and almost certainly cleanup) before the final version of the paper is published.

1 Like
9

I agree and I’m interested to know the answer but we decided to make our own measurements with stock ubuntu under the assumption that that’s what other users would have. Given that showed such high precision I wouldn’t personally feel the need to switch to realtime. I think the answer is to test timing of your experiment and, if that’s not good enough for your needs, then try the realtime kernel. i.e. I have a general philosophy to optimise things only when needed

10

The timing mega-study is now published at PeerJ: