No Data Files Created

Online experiments
1

URL of experiment: Pavlovia

Description of the problem: Hello! One of our team members made a post about one year ago discussing an issue with missing data files. Participants in our study are successfully completing the task and surveys and receive a completion code from prolific but the data files do not exist online or when we try to download them. For a period of time, we had been more successful with retaining data by allowing participants to complete part 2 of the study one at a time however, in our most recent batch of data collection we only retained 40% of the data. If anyone has any experience with this or suggestions for how to fix the problem we would appreciate your input!

2

How are your participants getting back to Prolific? Are you showing them the completion code on screen or is it in the redirect URL? My experience is that when the data doesn’t save there’s an error and there shouldn’t be a redirect.

3

Thank you very much for your reply. Participants are redirected from the behavioral task to qualtrics to complete a few surveys and from there redirected back to Prolific. I should also say that this issue occurs inconsistently, meaning sometimes we get data but after a few participants, it starts to be lost so I am unclear what could be different or malfunctioning from one to another. Could you say more about what you mean about an error and shouldn’t be a redirect?

4

Sorry for the double message - I just went onto our URL for where the experiment is hosted and I noticed that in the sessions box its registering that the last active session was on 9/3/24, this is consistent with what is down on Prolific, but the data files for those participants are not in the data folder when I refresh the repository nor are the in the data folder when I download it directly off the experiment URL. Also - I should add that we have inquired with participants and they do not experience any errors while completing the tasks and there does not seem to be any consistent pattern with browsers used. Is there any possibility that these files are being organized/created in a different location than the intended data folder with the potential of retrieving them?

Thanks for your assistance!

5

So – to confirm. You have participants who finish the experiment and are successfully redirected to Qualtrics via the completion URL in PsychoPy (not custom code or an incomplete URL). They didn’t receive any errors and they didn’t get a warning that their data wouldn’t be saved if they continued.

6

yes, that is correct.

7

When you checked the data files downloaded from the project page, did you also check in the db database folder just in case your experiment spent a period set to database instead of CSV?

8

Yes, I have been downloading the results periodically to check for the missing data and that folder is identical to the one in the repository.

9

Have you turned on periodic saving?

This is a new feature which I think could result in data loss.

10

No periodic saving is off and save incomplete results is on

11

Hi, and thank you for your suggestions. I’m just chiming in on this thread as Jordan is my grad student and running this project in my lab.

I’m quite concerned with the fact that our data is only intermittently being saved. We have, in total, attempted to collect data from 63 participants. We only have complete data from this task for 32 of these participants––so essentially 50% of our data is lost and we cannot use these participants. What prompted this most recent post to the forums was that in the last batch of data we were attempting to collect, 11 people completed the task on Pavlovia with no reported errors and were successfully redirected to Qualtrics and then to Prolific. Of these 11 people, we have task data for only 2 of them. There are no datafiles to be found from what we can see in the study’s gitlab repository or when clicking ‘Download Results’. Based on some of the suggestions we have received in the past, we did try making the study inactive for a few hours and then reactivating it, but that has not helped––the next participants we ran after doing this also did not have their task data saved.

This is not the first time members of my lab have messaged about this kind of issue with data not saving, and in searching the forums there are a non-zero amount of posts from others on similar issues that, at least to my eye, haven’t been explicitly resolved.

I am concerned because this a federally funded study for which funding is expiring and limited, and because people are being redirected to Prolific, reporting the appropriate completion code and with post-task Qualtrics data, I clearly have to pay them for their time. I do not have unlimited funds to continue pouring into this grant-funded project to keep collecting data at a 50% rate, and I am uncertain as to why data is saving for some people and not others when there is literally nothing changing in the experimental code. Is there any way that these missing data files are floating around somewhere in the ether on Pavlovia’s servers? @Becca @wakecarter @apitiot

12

Dear @jordandejoie , @dfareri ,

I just had a look at our server logs. It appears that almost all (> 99%) of the participant results are marked as incomplete. You can also ascertain that it is the case on your end, on the experiment page, where Jordan found the date of the last session:

sessions

The logs show the typical sign of experiment sessions interrupted before the data are saved.

This looked very odd to me since it is not possible for participants to be redirected at the end of a session without the session completing and the results being uploaded, i.e. PsychoJS waits for the data to be uploaded and, if all went well, redirect the participants.

So I had a look at the experiment code.
The problem is that contrary to Jordan’s answer to Wake’s question about using PsychoPy’s completion URL and not custom code, this experiment is indeed using custom code and not PsychoPy/PsychoJS approach to redirection.
Indeed in the quitPsychoJS function of WTP_SM_Choice_legacy.js, we find

psychoJS.quit({message: message, isCompleted: isCompleted});
window.location.replace(weblink);

with window.location.replace(weblink); as the manually added code.

The issue here is that PsychoJS.quit is an asynchronous function, and you are not using await. So, what happens, is that your experiment calls quit, which starts taking care of results, but, almost immediately, you navigate to the completion URL. This results in a variety of session data not being saved since, as mentioned in the “Save incomplete results” of the experiment page we cannot guarantee that data will be saved when experiments are suddenly interrupted (e.g. when tabs or browsers are abruptly closed, or when the tab suddenly navigates to a new URL).

The solution would be to remove the manual relocation code and use PsychoPy/PsychoJS approach. That should do the trick.

Cheers,

Alain

1 Like
13

Dear @apitiot,

Thank you SO much for your detailed response and for investigating this. We apologize for the miscommunication, this indeed is customized code, as you noted. We did initially push/sync from PsychoPy to Pavlovia in order to convert to PsychoJS, but because not everything was working as we had intended it to after the conversion, we had to manually edit the code quite a bit. This was programmed by personnel who have since moved on to graduate school, making it a little more difficult to troubleshoot, since they were more adept with javascript than either Jordan or myself.

In any case, I very much appreciate this and am wondering if you might be able to provide a little additional assistance––would you be able to tell me precisely how to edit those last lines of code (3888-3900) within the quitPsychoJS function to correctly incorporate ‘await’?

Thank you so much,
Dominic

14

I am not an expert in this and I strongly discourage editing the JS files directly, but if your experiment is otherwise working and it would be too difficult to go back to using PsychoPy to generate the code then you could try just adding await to the code.

await psychoJS.quit({message: me....
15

Hi again @wakecarter @apitiot,

Thank you for the suggestion. We added await as suggested in the last reply, and attempted to collect data from two more participants. We have data for one, but absolutely nothing for the other participant.

It is not a possibility to regenerate the code from PsychoPy, unfortunately. But the experiment works completely as it is supposed to except for this frequent data loss issue.

Is there any other way in which await would need to be implemented in that line of code to ensure that data is being consistently saved?

Thank you,
Dominic

16

@dfareri May I ask whether you or your student ran the experiment yourself? As you know, participants cannot always be relied upon. I would be curious to know whether you experience data issues when you know with certainty that you have gone all the way through and allowed for the data to be uploaded to the server, i.e. the last pop-up window has come and gone.