An approach to SONA integration and by-participant counterbalancing

Online experiments
1

I wasn’t able to find comprehensive guidance elsewhere for the problem of SONA recruitment, counterbalanced list assignment, and SONA credit granting, so I’m posting my current solution in case it helps others.

Description of the problem:
We typically design psychology experiments with a set number of participants in mind, creating n counterbalanced lists for m*n subjects, and we then try to run an equal number of participants on each list. When one participant fails for some reason (they don’t complete the task, or they complete it in an invalid manner), then we would try to run a replacement on that same list. Unfortunately, online research methods aren’t currently set up to do such counterbalanced assignment, let alone accomplish the replacement, so we need to find some way to approximate it.

My current solution
In a nutshell, I’ve set up SONA to link to (and pass its subject ID through) an external php script (https://moryscarter.com/vespr/pavlovia.php; thanks!) that simply increments a participant number assignment when redirecting to Pavlovia. Then my Psychopy/Pavlovia script uses the php participant number to select a corresponding counterbalanced list, and after the experiment it uses the passed-through SONA subject ID to redirect the participant back to SONA for automatic credit assignment. Replacement is accomplished in Psychopy by explicitly creating an array of the names of counterbalanced lists for Psychopy to cycle through based on the php participant number; if you start with an array of {1,2,3,4,5,6} and only need to re-run lists 1 and 4, then it’s not terribly onerous to just (manually) reduce that array to {1,4} so Psychopy assigns future participants to just those lists. It would of course be nicer to automatically identify lists to re-run, but I don’t think that’s currently possible without setting up your own php server.

So it has three parts and four steps:

  1. SONA: Recruit participants and link them to Morys-Carter’s php redirect (instead of linking to Pavlovia directly as SONA’s guidance suggests), including the SONA subject code (%SURVEY_CODE%) in the ‘id’ field.
  2. Morys-Carter’s php redirect (https://moryscarter.com/vespr/pavlovia.php): no configuration necessary, beyond a properly formed php call. Though it could be better documented, I think this is actually how he intends people to use the ‘id’ field in his php query.
  3. Pavlovia/Psychopy:
    a. Give your experiment an expInfo[‘id’] field so it can accept the SONA id in the link from the php redirect, and set your experiment’s online options to use that (instead of expInfo[‘participant’]) in your ‘Completion URL’ option.
    b. In a code chunk at or near the beginning of your experiment, use your existing expInfo[‘participant’] field (set sequentially by moryscarter’s php redirect) to assign the participant to a list for which you still need data. This is most easily done by creating an array that enumerates all of the lists when you first deploy the experiment, and then manually removing entries from that array when you have run them successfully. That is, instead of directly transforming expInfo[‘participant’] into a filename (as I had done in the lab, e.g. myItemList = ‘trialOrders/ListNumber’ +${parseInt(expInfo['participant'])} + ‘.csv’), you can then index that array:
    var arrayOfListsToRun = [1, 2, 4, 6, 53];
    var listNumberToRun = arrayOfListsToRun[(parseInt(expInfo[‘participant’]) - 1) % arrayOfListsToRun.length];
    myItemList = ‘trialOrders/ListNumber’ +${listNumberToRun} + ‘.csv’;
  4. SONA: Should automatically grant credit when participants return via the ‘Completion URL’ after completing the experiment.

I hope this helps others, and please post below if you can think of ways to improve it.

6 Likes
2

Thank-you for sharing your experience and documenting this to help others.

You will be pleased to hear that actually a similar solution to this is currently listed as a pull request for the psychopy documentation https://github.com/psychopy/psychopy/pull/3272 so the online documentation should be updated as soon asap. You will be able to see the finalised docs here https://psychopy.org/online/

Your explanation is actually more integrated with users of SONA systems though, and makes use of the id field - so thanks for sharing this information.

Becca

3

Hi - thank you so much for posting this, I’ve found it really hard to find out how to increment participant numbers for counterbalancing.

However, I was wondering if you wouldn’t mind explaining step 3. around Pavlovia/PsychoPy. I’m a novice at both (and at Python) so I don’t understand how I get my experiment to accept the SONA id (actually, in my case it’s Prolific, but I presume the theory is the same). At the moment I just want to check Morys-Carter’s redirect is incrementing participant numbers AND carries the Prolific ID through to Pavlovia.

Any help would be greatly appreciated, thank you!

4

My web app increments the participant variable and passes id , session and researcher unchanged (which you can use for whatever you need).

I’d be happy to accept suggestions of improvements to the documentation on the page.

5

Hi Rach,
To add an ‘id’ input argument to your script via the builder interface, you’d just click on the gear icon in the menu bar to open an “Experiment Settings” panel (see screenshot below). Within that panel, you’ll see some fields labelled ‘Experiment Info’, and it lists your script’s expected input arguments. By default, it lists ‘participant’ as the first, and ‘session’ as the second, but you can change these names, or add extra fields by clicking on the + sign to the right of them. So that’s how you would add the expInfo[‘id’] field.

image
image781×619 86.9 KB

(As wakecarter just confirmed, his php script only increments the ‘participant’ number; it passes the rest of the variables through unchanged, so you can use the id, session, and researcher variables to pass through any short string of alphanumeric characters, such as a unique participant ID that SONA or Prolific will offer to generate.)

Once you’ve prepared your experiment to expect your participant and id input arguments via the URL, then you need to tell your experiment what to do with it. I used a ‘code component’ with a few lines of javascript (because the online version uses javascript instead of python) to convert the participant argument into filename. And the only thing that I did with the id argument in Psychopy was to use it set the ‘Completed URL’ in the ‘Online’ tab of the ‘Experiment Settings’ panel.

6

Thank you @goppenh2 , that’s really useful - and thanks again for posting the original.

@wakecarter; your web app and the ability to consistently increase participant number is likely to save our entire experiment, thank you as well.

7

Hi @wakecarter, thank you very much for your script and crib sheet!

Re: documentation, I think the main point of confusion is that SONA offers its own instructions for integrating with Pavlovia directly, and it also claims the ‘participant’ field in its instructions, so using your script with SONA requires knowing to contradict/modify SONA’s own instructions. It’s not such a big deal once you understand the various scripts and how they work together, though.

But the other part of the challenge is that your suggested javascript snippet (condition = int(expInfo[‘participant’])%x, which I think should actually be condition = parseInt(expInfo[‘participant’])%x, right?) will only try to assign participants equally to all conditions. That is, your script will increment and redirect every time it is called, but because online studies produce a fair amount of waste (e.g. participants who click through before realising that they can’t do the study on their phone, or who just mash keys randomly, or who apparently complete the full experiment but close their browser before uploading their results), just assigning redirects equally won’t guarantee equal numbers of valid datasets across conditions. The potential imbalance is not such a problem if you only have four conditions, but if your counterbalancing requires 48 different sequences then it becomes more concerning. So that’s when it becomes useful to use that modulus to index a second array that you can update as needed:

// var arrayOfListsToRun = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48]; // ORIGINAL ARRAY
var arrayOfListsToRun = [1, 4, 6, 35, 48]; // UPDATED ARRAY TO RE-RUN JUST THESE LISTS
var listNumberToRun = arrayOfListsToRun[parseInt(expInfo[‘participant’]) % arrayOfListsToRun.length];
8

I think my other script https://moryscarter.com/vespr/survey.php has a tweak (apparently undocumented) whereby if you sent it a value for participant it will increment id instead. I’ll apply that to pavlovia.php which should help.

I code in Python wherever possible, so the int(…) is referring to Python code which will be auto translated into parseInt(…)

One day I’ll probably be commissioned to write a new app which will code with participants who don’t finish but in the meantime I agree that your solution is a good one.