Description of the problem:
Hello forum! We are running an online experiment and are running in an issue where many (but not all) participants record a very low frame rate, in the order of 3-10 fps. Given that the experiment aims at capturing the timing accuracy of a sequence of keyboard key presses, the low frame rate is a significant issue as it drastically reduces the time resolution with which we can detect key presses.
General description of the setup:
The experiment was created in PsychoPy Builder (the latest build was created in version 2022.2.4) and is hosted on pavlovia. Participants are recruited from Amazon MTurk and then routed to pavlovia for the actual experiment (the MTurk HIT contains a link to the pavlovia experiment and also passes several URL parameters). Participants are asked to use Chrome or Edge browsers, and so far everyone has complied with this request.
How we calculate the frame rate:
Frame rate is calculated in two ways. First, we use the standard call to psychoJS.window.getActualFrameRate(). This is the usual function call added automatically by Builder and executed at the very beginning of the experiment. We also added a custom fps calculator that updates throughout the experiment, and the results are consistent with the output of psychoJS.window.getActualFrameRate() - therefore we are pretty confident that the reported fps values are correct.
Since getActualFrameRate is called before any stimuli are presented, we are also inclined to believe that this may not be an issue with rendering graphically demanding components (e.g. object features changing every frame, or high resolution images).
More details on the issue:
While we observe the low fps values in many of our participants, for some participants we see “normal” values that correspond to typical monitor refresh rates (e.g. 30 Hz, 60 Hz and in some cases 100-144Hz).
We did not observe this problem when testing on our own systems: we tested on Windows systems (desktops and laptops), Mac laptops (both Intel and M1 processors). We tested on high powered desktops and also on very low powered laptops. In all cases, our test results showed frame rates close to the monitor refresh rate. While there is certainly room for optimization in our code, this makes us believe that the experiment should run correctly under the right conditions.
While it is hard for us to reproduce the issue, one very helpful participant was willing to run some tests for us. Initially this participant was getting a low frame rate (~6 fps), even when using a high performing laptop (Intel i7 processor, RTX 3060 GPU and 144 Hz monitor). We found that switching from Chrome to Edge fixed the issue, and ultimately we also made it work in Chrome by enabling hardware acceleration. We are now able to reproduce low frame rates on our systems as well by disabling hardware acceleration in the browser settings. After some digging in the documentation, it seems like hardware acceleration is required to make PsychoJS work, so all this makes sense.
The question:
This all makes us wonder why there seems to be such a large percentage of participants that may have disabled hardware acceleration. Has anyone else had similar issues, and are there any recommendations on how to address this?
We are also looking for other potential causes for low frame rates. We would appreciate any suggestions on what we could try to address this.
Sorry for the long post, and thank you very much for your help!
Thanks for your careful testing, this is not an issue that we were aware of (I myself have never encountered this in my experiments) this indeed reflects the many unknowns regarding timing “in the wild”.
Please correct me if I misunderstood anything - I do have a few questions:
Was there a reason to measure framerate independently? rather than using the “frameRate” variable that is automatically calculated at the start of the study?
Is there any possibility participants were running a range of other functions in other windows on their computer?
is there any chance participants clicked away from the experiment window during the task?
do you have a “minimally working demo” of the problem so that we can try to replicate too.
Thank you for your reply! To answer your questions:
We initially were only using the automatically calculated frameRate. After observing our first few participants with low frame rates, we added our frame rate calculation to see if the low fps values were only at the beginning of the experiment or if they persisted also in the later stages. We found that the low frame rates are consistent throughout the experiment, and are not tied to a particular routine.
Yes. Although we added instructions not to do this, we don’t have a way to control what other processes and windows are open.
Yes. This is pretty much the same problem as #2.
I’m working on this and will share as soon as it is ready. What is the best way to share this with you? Should I give you access to the gitlab repository, make it public, or is it sufficient to share the link and set the experiment to “running” on pavlovia?
Just to follow up, I made a new repository with visibility set to “internal” - this should allow you to access the experiment. I shortened the experiment by removing some parts, but this version still calls most of our functions and loads all the same resources. The routines that immediately follow the built-in frame rate calculation are unchanged. It still exhibits the same frame rate issue when hardware acceleration is disabled.
Hi! Did you find a solution to the frame rate issue? I have the same problem with some online participants. They even sent me screenshots of their display settings which were set to 60 Hz but the frame rate recorded in the task output file was 30 Hz. Hardware acceleration was enabled when this happened. Switching from Chrome to Edge also fixed the issue in this case.
Sorry for the long silence. We are actually in the process of writing up a short manuscript summarizing the various problems/lessons learned when running online experiments through MTurk. Part of that will focus on this frame rate issue. I will add a link when it gets published.
In our case a big factor in the low frame rates was that over 80% of workers/participants had hardware acceleration disabled. We added reminders to enable hardware acceleration in the description of the MTurk HIT and added multiple checks inside the experiment that would prompt participants to enable it - clearly without success.
The default renderer of the PsychoPy version we used (2022.2.4) does not perform well when it runs on the CPU, hence the low frame rates. I also think that we did not notice it as much when running previous versions of PsychoPy, as the frame rate calculation was not as accurate.
The PsychoJS development team was very helpful and provided us with a beta build of PsychoJS that detects if hardware acceleration is enabled or not (that’s how I can now say that 80% of participants had it disabled). If disabled, PsychoJS switches to a different renderer that runs more efficiently on the CPU. This increased our frame rates substantially, to the point where HW acceleration off and on are comparable in terms of frame times. If I understood it correctly, this feature should make it into the main PsychoJS branch in the future.
@JohanH The code to check this within the experiment was already added by 2023.1.0.
For participants that didn’t have hardware acceleration supported or turned on you should be seeing a column in your data files labelled hardware_acceleration containing 'NOT SUPPORTED'. Maybe link your project if you want us to look more deeply in your own study
Hello. I am currently collecting data for a study and started to notice that a couple of my participants have abnormally low frame rates (< 3). From what I am reading in this thread, it seems like the solution that people came to is to ask participants to not use chrome. Is this correct? Are there other more recent solutions on the matter?