Description of the problem: when I download the results the data files are not in the folder. Sometimes the .log file is there but not the CSV.
I just put the project in pilot mode to save credits, but it was in running mode earlier and it did manage to save 1 participant’s data today.
The experiment was built in PsychoPy with a few manual code components. Redirect URLs were set in the Builder and participants are asked to wait until they have been redirected.
I’ve tested the programme extensively in running mode, and it appeared to save data just fine before, I’ve made some changes but they were minor and seemed to be working well. Would anyone know what might be happening and how I could fix this?
no, the experiment was in running mode so it won’t be pilot data and one participant’s data was saved successfully. It was in running mode for a week or two, until I changed it to “pilot” mode last night after realising some data weren’t saved to prevent people from doing the experiment.
If this is useful information: the saving format is set to “CSV”
Not really. The .log files add up with the number of credits consumed.
At the end:
Participants are shown a debrief page, they’re asked to confirm that they are happy to submit their data by clicking “submit” or “withdraw”. This is saved as any other mouse click response, the same thing happens regardless of which button they click on.
Then there’s a dialogue from Pavlovia (asking them to wait and press OK)
Finally, they’re automatically redirected to a website (this was specified in Builder, in the Experiment settings dialogue)
I just tried testing myself again, and it saved my data correctly. So it seems to work, just not consistently.
Before I started testing, on Tuesday, I was a bit concerned because two of the csv files were missing but I found one under a different name (Database_AllParticipants…csv) rather than the usual filenames. After that, I changed the saving format to CSV, and tested it again and it saved everything.
I’m using database instead of CSV for my experiments. I don’t get individual CSV files but instead get all the data in the all participants file. I do get log files.
Check the all participants file to see if they’re all there
Just checked again, the missing CSV files from Tuesday are definitely there in the 'All participants file". However, the number of CSV files still doesn’t match up with the reserved credits… it was in CSV format during testing yesterday.
It is possible that participants pressed Esc/closed browser prematurely, in that case, would results be saved for the trials they did do or credits be released automatically? i.e. can I rule out the possibility that data is not being saved correctly and assume that if participants get to the end of the experiment, the data will be saved properly?
(sorry I am really new to this and not sure how credits and data saving works yet)
Credits are reserved when a participant starts and either consumed when they finish or released after 24 hours. Data is only saved when credits are consumed. Pressing escape or closing the window loses the data
I’m having a similar problem where CSV data seems to be selectively not saving for some participants. For these participants, I can see that they submitted the correct code which they could only have obtained by finishing the experiment, but there are no corresponding CSV files… For other subjects, I can see partial data where they aborted the task early, so it seems as though aborted runs are being saved at least sometimes.
I am now trying again with the ‘database’ format, so we will see how that works — but is there anyway to recover the CSV data that was not saved though? It seems very problematic that a subject could spend time doing the task, and a credit could be consumed, but no data would be saved in the end.
I think the problem might have been fixed now (I’ll copy and paste the email below, I’m sure there’s a discourse thread for it somewhere but can’t seem to find it!).
I haven’t tested since the update… but I think my issue was also probably due to participants exiting the programme prematurely e.g. if they press ESC or close browser before the programme ends, even if they’ve ‘completed’ the experiment, the programme would see it as the participant withdrawing. I did also get partial data though, so I’m not really sure whether there were other issues…
See email from PsychoPy below (I received it 27 June)
Hi there,
We are writing to inform you about an upcoming change to how we handle experiment results when participants fail to complete a study (‘incomplete runs’).
TL:DR We have introduced a new setting on the Pavlovia projectpages that determines whether data from incomplete runs are stored (and whether a credit is consumed in this case), taking effect on Tuesday, June the 30th.
Up until now, when a participant pressed the Esc key to interrupt a study, the partial data were saved (without a credit being consumed) whereas when a participant closed the browser tab we treated that action as consent withdrawal and, consequently, did not save the data (and did not consume a credit).
We have identified three potential downsides to this policy:
Did the participant intended to leave the experiment or merely closed the window by mistake?
Is it appropriate to keep the data from an incomplete run? We opened a discussion on the forum a month ago (Discussion: What to do with partial data when a participant quits? - #4 by wakecarter) and have come to the conclusion that this would best be left up to the researcher. We urge you to think carefully about whether a participant expects their data to be deleted when they withdraw, and what your ethics approval/information/consent form says about this scenario. If you intend to keep the partial data when your participants exit we believe that, morally, it would be best to inform them that you do. Legally, it will depend on your local rules and laws.
Should credits be consumed for partial data? Those among you working with Participant Credits rather than a Site License may or may not want to spend a credit for incomplete data.
We will be implemented the following solution:
We will ask the browser to warn participants before they leave the page that data are about to be lost. You might have seen such warnings before when leaving a web page with a partially filled form. That technology is somewhat limited: it’s browser specific, we can’t change the text included in the warning, and if a browser doesn’t permit the message at all (now or in the future) we can’t do anything about that. As you may know, browsers are constantly evolving and changing what is or is not permitted, to make sure that programmers don’t abuse the functionality to prevent people from leaving their malicious site. At first, this will apply to the latest versions of the library (from 2020.1 onward). We will propagate this behaviour back to older versions in the coming weeks.
2 & 3. To make it possible for you to decide how to handle partial data, you will find a new control in the Dashboard, on the projectpage, in the ‘Saving Results’ section, asking you whether to save incomplete results. You can then decide to:
save incomplete results: a credit will be consumed for those of you not covered by a license
not save incomplete results: no credit will be consumed
That new control will take effect on Tuesday, June the 30th.
Unfortunately, we weren’t able to fix this problem entirely… However, adding a warning at the beginning and end of the experiment about pressing the escape key did seem to help this problem occur less frequently.
I think the issue was that some participants were exiting the task by closing their browser window rather than hitting ‘esc,’ as we could see that they had the correct unique task code, but never received their data.
Weirdly I am still having this problem, but have only told participants to press ‘esc’ - still not getting any unique codes, just “PARTICIPANT” as the name of the file (and the log and csv files are ~1-2 bytes, nothing in them)
Sorry to clarify- I basically tell them “IMPORTANT: finish the full experiment, redirect to Qualtrics, which will then redirect to SONA for automatic credit granting. If you wish to stop at any time, press esc.” I have a long and boring task so the abandonment rate is pretty high- people are definitely quitting early. This is the first time I’m running this task so I’ll probably cut down the trials.
I was saying that I have ‘only told participants to press esc’ referring to the closing the browser tab vs pressing esc solution (which didn’t work for me, as this is still happening)
have you switched on “Save Incomplete Results” on Pavlovia?
Are you using CSV or database?
are you sure that data is not being saved? e.g. have you replicated the issue on your own device? I am wondering whether some people might be exiting the experiment early (but after expInfo dialogue box), meaning the CSV files will be small anyway.
I think it’s worth adding a clear, simple warning at the end of the experiment, asking participants to not close their tab or window, and wait until they are re-directed, and say that this is to make sure that their data is saved.
I worry that some participants might see ‘press esc’ and, without reading the whole thing carefully, will assume that that’s what they have to do after they’re done.
Since the ‘save incomplete results’ was rolled out I haven’t really had this issue, I still have a warning message at the end though…
CSV, although I briefly switched to database and was having the same issue so I switched back to CSV
For a couple participants, it did save their incomplete data with their ID…but that only happened for about 2 of the participants who abandoned. The others have PARTICIPANT (not ID number) in the file name of the saved data and the csv files are completely empty (logs are empty, very short, or not there)
I do have a waiting message at the end and have told them that their data will be lost if they close out the tab (both in the instructions and in an email prior to starting the experiment).The message says that they must wait until the experiment redirects them to Qualtrics, which will then redirect to SONA for automatic credit.
The fact that incomplete data is being saved is promising though. In theory, even if participants don’t wait till the every end, the data should still be saved.
If you are getting empty csv files (without participant ID) but do have the log files, I think that might suggest that those participants may have just clicked on the link, but didn’t go past expInfo (the dialogue at the beginning).
I just did a quick test with my experiment. Instead of clicking ‘OK’, I clicked ‘cancel’, and that created a log file, but no data. I’m using database rather than csv (it was recommended to me by someone, I forgot who…), it also takes much less time to process as I only have to open a single csv file in Matlab.
It might be worth trying to recreate the issue on your device or doing similar tests to see how Pavlovia behaves to rule out possible scenarios. Unfortunately with online experiments I expect higher dropout rates, and sometimes people might prefer to do the experiment on another browser or device, and that could also produce empty data files.