Data Not Recorded Properly Despite Smooth Pilot and Full Run

Online experiments
1

Hi everyone,

I’m hoping someone can help me understand what went wrong in my study on Pavlovia.

After piloting my experiment with 17 participants and confirming that everything worked fine, I launched the full study online. For the real run, I deselected the “save incomplete results” option to avoid using credits for unfinished sessions.

Now, with nearly 300 participants recorded on Prolific, I checked the data and was surprised to find only 100 spreadsheets, all of which are incomplete and don’t resemble the data structure from the pilot. For example, none of them contain the consent form response (which is the first screen shown in the experiment), nor the start and end display times for it. Additionally, the order of variables in these datasets is confusing.

I’m puzzled because:

  • The pilot worked well, with no reported issues from participants. I didn’t get any complaints from participants about being unable to continue.
  • The experiment itself runs smoothly for me when I try it using the paid link, just as it did with the pilot link.

This was my first time using Pavlovia, so I realize I should have checked the data earlier. Still, I’m surprised that I didn’t get any complete data at all—or that I even have 100 incomplete spreadsheets despite having deselected “save incomplete results.”

Could anyone help me figure out what might have caused this? Or if there’s any way to retrieve usable data?

Thanks a lot

2

Hi Otávio,

Sorry to hear you’re running into issues with data saving. Based on my own experience, it sounds like this could be related to participants not fully completing the task.

A few suggestions that might help:

  1. Check participant completion
    You mentioned: “The experiment itself runs smoothly for me when I try it using the paid link, just as it did with the pilot link.”
    This suggests that the issue isn’t with your task setup itself, but may be with participants dropping out before completion. It’s fairly common if you have a long online study (how long does it take to complete your study?).
  2. Recruit in small batches
    When I run studies, I usually recruit participants in smaller batches. This allows me to monitor data quality as the study progresses, and adjust recruitment if needed.
  3. Manually review payments
    After each batch, I review submissions manually before approving payments. Prolific allows you to reject incomplete submissions if participants fail to provide complete data.
  4. Set up automatic redirects
    How are you currently redirecting participants back to Prolific?
    We typically recommend using automatic redirects instead of relying on participants to copy and paste a completion code (which can mean participants leave the experiment tab prematurely to copy and paste, and therefore don’t end your experiment properly).
  • Automatic redirects reduce dropouts caused by participants not copying codes properly.
  • You can set up different redirect URLs for:
    • Full completion
    • Partial completion or early exits
      This ensures that only participants who fully complete your study get marked as complete on Prolific.

Hope this helps!

Becca

3

Hi Becca,

Thanks for your reply. I just ran the study again (using “run,” not “pilot”), this time with the “save incomplete results” option checked—and in this case, the data was saved.

However, I still don’t understand why the data wasn’t being saved at the end of the experiment in previous runs when I had “save incomplete results” turned off. This has me completely puzzled, as data was being properly recorded when using the pilot button. It’s honestly quite disheartening—aside from my mistake of not running the study in smaller batches to verify that data was being saved, I had no reason to suspect any issues, especially since the pilot phase ran smoothly without any problems.

When running the study with the “save incomplete results” option enabled (which successfully saved the data), I received the following error message. I wonder if this could be related to the issue I experienced in previous runs—despite no error message appearing at the time:

image
image551×342 21.9 KB

This is what I’m seeing on Pavlovia. As you can see, 0 for submitted:

image
image446×102 6.89 KB

Just to confirm: have I definitely lost the data from all 300 participants, or is there any chance that it might be stored somewhere on the server?

Cheers,
Otávio

4

Hi Otávio,

OK so since the data are only being saved when “Save incomplete” is enabled, it sounds like the experiment may not be ending properly (i.e., it’s technically finishing, but PsychoPy isn’t recognising it as a completed session). Could you please share your PsychoPy Builder file with us?

A few additional questions that might help us troubleshoot:

  • What version of PsychoPy were you using to run the experiment?
  • Could you share a screenshot of the project dashboard on Pavlovia? (We’d like to check the saving settings for the project.)
  • How does the task normally end? For example: is there a final screen where participants press a key (e.g., SPACE) to finish? Do you usually see the green “Thank you for participating” message before the browser closes?
  • Are you able to share the experiment link with us?

Unfortunately, there’s usually no way to retrieve data from aborted sessions on Pavlovia. Many researchers enable the “allow incomplete data” option to give participants the right to withdraw at any time, which is why aborted sessions don’t save data by default when participants exit early. If we have the link to your project though I can see if we can get someone to look at the logs, this can’t guarantee restored data but it might tell us a bit more info on what is happening (just as a note, I don’t personally have access to check that, I would need to contact another team member so it will take a bit longer sorry)

Thanks,
Becca

5

(post deleted by author)

6

I also added some new information to my previous message—sorry for the confusion! I didn’t realize you had already replied before I made those edits.

It’s quite surprising to learn that researchers usually prefer to keep “save incomplete data” enabled, as I had simply followed Prolific’s recommendation to disable it. They state the following:

If you choose to allow incomplete responses to be saved, and the “Incomplete” redirect URL does not contain a Prolific completion code, this may lead to a mismatch between the number of spaces available on Prolific and the number of credits available on Pavlovia (i.e. participants have consumed credits despite not completing, so the space is still marked as “available” on Prolific). This could lead to participants starting the study on Prolific only to see a message that the study does not have sufficient credits. We highly recommend choosing not to save incomplete responses, or else making sure you have enough credits to cover additional participants. (Pavlovia integration guide)

In any case, I’d really appreciate it if you could check the logs, as you suggested.

Thanks a lot for your help!

7

Hi Otávio,

Ok so from the screenshot I can now see that you’re using a completion code on the final page, rather than an automatic redirect link. Given that setup, I suspect what may be happening is this:

Participants reach your final page → copy the completion code → switch to Prolific to paste the code → but never return to your experiment tab to actually close the experiment properly.

As a result, the experiment ends from their point of view, but PsychoPy/Pavlovia never receives the final signal to save and upload the data.

To help confirm this:
If you return to this final page yourself and click to properly end the experiment, do you see the “Please wait” message, followed by the green “Thank you” message? That green thank-you screen is the key indicator that data have been successfully saved and uploaded.

Regarding the Prolific advice: I think I may not have explained it clearly earlier. The warning on Prolific says:

We highly recommend choosing not to save incomplete responses, or else making sure you have enough credits to cover additional participants.

This is correct - if you allow incomplete responses to save, aborted or withdrawn sessions could still use up credits. Many researchers disable saving incomplete data specifically to avoid paying for partial/incomplete runs or because they want to fully honor participants’ right to withdraw. The trade-off, however, is that if participants exit the experiment incorrectly (without completing the final steps), their data won’t be saved.

In Prolific’s integration guide here, they also mention the issue I think we are encountering here:

When using manual completion codes, make sure to instruct your participants to complete your task once they have copied and pasted these codes (e.g., “Once you’ve copied the code, hit space to complete your submission”. This will ensure data is fully submitted back to your project.

So, I think participants may not be returning to submit properly. That said, you mentioned that even when you submit properly yourself, sometimes the data still don’t save - which is why I’m asking about the green thank-you screen as a quick check.

Hopefully this helps clarify what’s happening, even though I completely feel for you that this must be a stressful situation.

Becca

8

Hi Becca,

Thanks! I’ll go through your points one by one, quoting your message below:

Ok so from the screenshot I can now see that you’re using a completion code on the final page, rather than an automatic redirect link. Given that setup, I suspect what may be happening is this:

Participants reach your final page → copy the completion code → switch to Prolific to paste the code → but never return to your experiment tab to actually close the experiment properly.

As a result, the experiment ends from their point of view, but PsychoPy/Pavlovia never receives the final signal to save and upload the data.

Unfortunately, I don’t think that’s the cause—because even when clicking the redirect link (in the “Click HERE to return to Prolific” message), the data still isn’t saved. I know this because I tested it multiple times myself today.

To help confirm this:
If you return to this final page yourself and click to properly end the experiment, do you see the “Please wait” message, followed by the green “Thank you” message? That green thank-you screen is the key indicator that data have been successfully saved and uploaded.

Yes, that screen did appear for me very quickly before the Prolific website opened.

When using manual completion codes, make sure to instruct your participants to complete your task once they have copied and pasted these codes (e.g., “Once you’ve copied the code, hit space to complete your submission”. This will ensure data is fully submitted back to your project.

I agree that the final slide allows participants to manually enter their completion code. However, as I mentioned before, my data still wasn’t saved even after clicking the redirect link on that last slide.

That said, you mentioned that even when you submit properly yourself, sometimes the data still don’t save - which is why I’m asking about the green thank-you screen as a quick check.

The data was never saved, even after the green “thank you” screen appeared. What I meant earlier is that I did receive some datasets from participants, but they were essentially empty and didn’t match the expected data structure for my study.

Hopefully this helps clarify what’s happening, even though I completely feel for you that this must be a stressful situation.

Thank you. It’s pretty nerve-wracking, I may not be able to finish my thesis. Aside from not running the study in batches, there was no way to anticipate this issue, especially after so many successful pilot runs.