I am experiencing data loss (~15% of the submissions) using the Pavlovia plugin inside JSPsych.
My question here is, is there anybody we could ask to to try to nail down this issue?
It does not seem to be tied to a specific browser, it is perhaps related to the quality of connection but I cannot really tell if there is actually an actual reason with the data not loading in the server.
I have seen other posts on the topic with no responses. I know this is a forum mainly for Psychopy but it seems to me the only place to refer to in case of issues. If this is not the right place to ask could anybody please point me at the right place to do that?
Many thanks for your help.
I and my colleague also have still been experiencing this issue as of December 2020. This link reaffirms there is a bug with jsPsych + Pavlovia and seems to suggest it is partly a matter of inserting a delay after the submission but before experiment’s conclusion. But I’m really hoping there is another workaround because that seems like an error-prone solution. That post also mentions saving data more than once to pavlovia’s server, which in itself may have created data loss problems for me (although I did not try pausing the experiment to save).
I just ran a couple of tests to see what happens if data sent via the Pavlovia plugin to Pavlovia gets delayed. The plugin then waits until it gets a confirmation from Pavlovia that the data is received before continuing. If something goes wrong with sending the data, the plugin reports an error.
Hence, I don’t think the delay that they suggest would help. Also, as far as I can tell, the plugin works fine. Actually, more robust than I would have expected before my test :).
If you do have this problem and you’re able to reproduce it (perhaps with help of participants), perhaps it could be useful to check whether the experiment itself doesn’t cause any issues?
Thanks both for following up on this topic.
In my experience, things have improved a lot for data loss in the past couple of months. However, in a sample of say 50 people I still have at least 1-2 datafiles lost in translation!!
I never could understand why this happened. However, I convinced myself that it has to do with unstable connection. Some of my own data while debugging could not be pulled when my wifi was jumpy. This was the only time I finally managed to reproduce the data loss, after having tried multiple browsers, operating systems and browser versions without managing to lose my own data!
I am still not 100% sure that is the reason why such data loss occurs but it may help to disclaim that participants should make sure to be under a stable connection before starting the experiment (e.g. on Prolific instructions).
Note that this is not a definitive explanation. The inconsistency of this issue convinced me that is not something in my code or in the functioning of the plugin but I am not entirely sure.
I also came across that link and felt somewhat relieved that I wasn’t the only one having this issue, but still I never implemented it in my code as my percentage of data loss has been extremely low.
Thanks @thomas_pronk for running the tests! this is reassuring!! Do you think unstable connections or page refreshs could cause the plugin to fail transferring the data from the server?
Do you think unstable connections or page refreshs could cause the plugin to fail transferring the data from the server?
If that happens, I’d expect you’d also see something happening on the side of the task. On a refresh I’d expect the task to restart; on issues transferring the data I’d expect an error message.
In general, 1-2 out of 50 is quite a good score. The annoying thing with web is that it involves a bewildering variety of platforms; some are bound to go wrong. Well… at least at the scale and budgets that us web-developers for social science operate; we’re doing rather funky stuff, but we ain’t no Facebook
With PsychoJS we’re in the process of setting up a rather large test suite to find and fix issues, but even then, something will definitely slip through the cracks.
Thanks so much. OK, my data loss percentage was more like 20%, so it has been a big headache for me, and I have heard a similar ratio from my colleague running a completely different experiment. In her case, though, most of the time (but not always) the data would be downloaded to the partcipant’s computer as a csv file when it did not get sent to the server, which is not happening with mine. Is there a way to cause the data to download to the participant’s computer as a back-up, as well as submit to the server? I can’t figure out why her experiment code produces such a different pattern than mine. I will say that my experiment files are about 6 MB in size, which is likely larger than most, on account of our tracking mouse movements during the experiment, so the unstable internet theory makes sense to me. The problem does not seem to clearly track particular browsers or even whether people are abroad or domestic, although I would have to ask everyone to be sure. I will say that adding the delay seemingly did help in my case although I know you are saying it shouldn’t matter. I really appreciate your help and advice; if I could make sure the data is reliably downloaded to the participant’s computer that would resolve most of the issue for me although it would still be an annoyance. Thank you so much for helping to address this issue!
My pleasure! Yeah, I got options for you, based on a little update of the jspsych pavlovia plugin. JsPsych Saving Options on Pavlovia
Using the filter function in this modified pavlovia plugin, you could do two things:
- Use the filter function to perform some data reduction on your mouse tracking data, so that less data is sent to Pavlovia
- Offer the file for download. I see that jsPsych has a function for that: https://www.jspsych.org/core_library/jspsych-data/#localsave
Merry Christmas btw
Thank you so much, really appreciate you Thomas! Hope you had a happy holiday and a nice new year