Pavlovia credits after failing attention check

URL of experiment: n/a

Description of the problem:

We have two attention check questions – if a participant gets either of them wrong, it essentially terminates the experiment. The issue is that we want to 1) get access to the participant ID if a participant fails the attention check so they dont redo study and 2) get this information without using a Pavlovia credit.

In our initial studies, we had over 35% of our Mturk participants fail these checks. Even if we dont pay them for their participation, using these Pavlovia credits is costly with many participants. We now have it so, if they fail, it sends them to red screen that says “you failed attention check”. It then asks them to exit the program so we dont use a Pavlovia credit. However, now we can’t store their participant ID with the rest of the completed data files.

Our current work around is that we have them enter their ID before starting the Pavlovia experiment, so we can at least see if the participant tried to do the experiment at some point (but dont know if they just stopped or if they were booted because they failed the attention check). But it’s not optimal to try to verify things this way.

Is there any solution to this issue? That is, can we get a data file with their ID number without having to use the credit? Or if we want to verify they failed the check, we just have to suck it up and pay for it?

You are gathering data from the subject, so…

That is, Pavlovia can’t tell how much of an experiment is “real”. What if there was half an hour of questions before an experiment decides to reject a participant, but the researcher still wanted to retain all of that exclusion data?

Pavlovia is a largely automated system, which is how it can be so cheap. If there were to be exemptions for things like this, it would have to be a on a per-study basis, and that would require extra staffing to administer and implement. Any credits you might save would have to be balanced by the need for the credits themselves to be become more expensive accordingly.

You should be able to have an incomplete URL containing their participant ID that goes to a website that will collect the numbers for you. I could quite easily write such a site (based on my previous tools) or you could use Qualtrics or anything else that can log a URL variable?

Thanks, Michael. I definitely understand why things are the way they are. Just wondering if there was any potential work around for it :slight_smile:

Thanks, @wakecarter.

Our University just transitioned to QuestionPro. So from MTurk/Prolific, they go to the QuestionPro survey where they enter their ID and then it assigns them to one of the conditions on Pavlovia. On the Pavlovia startup, they again enter their ID. When they successfully complete the experiment, Pavlovia generates a code and they manually enter it into QuestionPro. They are then redirected back to Mturk/Prolific, which confirms their submission.

If they fail the attention check, there is currently no redirect - it just goes to the red screen I mentioned that asks them to exit. So if there was a way to redirect after the failure screen that could automatically log their ID (e.g., back to a different QuestionPro that logs their ID from Pavlovia) and also not use a credit, that would be great. I say “automatic” because that would be better than saying “you didnt pass attention check…but before we dont pay you, please type in your ID again!” But I guess we could always redirect them first, tell them to enter their ID, and then say “sorry you failed”.

What exactly were you thinking?

Ideally you shouldn’t be asking people to manually type anything in.

Ideally MTurk tells QuestionPro the MTurk ID.
Then QuestionPro tells Pavlovia the MTurk ID and the condition.
Then Pavlovia tells MTurk or QuestionPro the MTurk ID and the condition.

If you have a URL to Pavlovia which includes ?participant=MTurkID&condition=ConditionID and those are the only two variables in the expInfo dialogue box, Pavlovia will go straight into the experiment.

The completed URL would look something like:

https://mturk.com/completion?id="+expInfo[‘participant’]

and the incomplete URL would look similar but might also include expInfo[‘condition’]

I’d need to know more about the formatting of the URLs and whether QuestionPro can receive and send embedded data to be able to help further.

@wakecarter - thanks for the info.

I definitely understand (conceptually) what you’re saying. But we’re new to all of this – Pavlovia, Prolific, and QuestionPro – so it’s going to take us a bit of time to try to figure all this out. If you have any guidance, that would be great. But otherwise we’ll plug along and see what we can figure out over the next few days. If we cant come up with anything I might ping you then.

Thanks,
Hunter

Hi Hunter,

If you want help here then you need to tell us what you know and what you’ve tried.

This link may be useful:

Best wishes,

Wakefield