Participant data is initially stored on the run server and then transferred to the GitLab server.
If you download data from the project page you are downloading from the run server. N.B. If you are saving in database format then only the log file gets transferred, and data can only be downloaded from the project page.
If you change your experiment to inactive then it gets deleted from the run server along with any data stored there. This isn’t usually an issue, but if for any reason the data hasn’t transferred to the GitLab server this would result in data loss.
Therefore download your data from the project page before setting the project to inactive. You can switch it to piloting to prevent additional participants from launching it.
It is good practice to delete old experiments from the run server by switching them to inactive, but do not be hasty. Check you have all of your data first, and don’t assume that everything has been transferred to Gitlab automatically.
N.B. If you add the data folder to .gitignore, it will stop data being added from the run server, as well as from your local data folder.
Random allocation to groups is easy, but by it’s very nature can easily result in unbalanced group sizes. For counterbalancing, the allocation needs to take into account the allocation of other participants.
I have written three external tools which offer solutions to this problem.
This tool adds a consecutive value for participant to the study URL which can be used to systematically allocate participants to different groups using the modulus function, e.g. group = int(expInfo['participant'])%3 would allocate participants to groups 0, 1 or 2.
This uses the same principle, but sends the participant to different study URLs on the same domain.
Both of these tools allocate equal numbers of participants to each group at the start of the experiment, but have no knowledge of whether the participants completed the study or not, so unequal numbers of completed participants in each group are still possible.
This is a more powerful external tool which allows the researcher to host the participant information sheet (so the study is only launched by participants who consent). The licensed version (£10 per year) allows counterbalanced allocation to groups independent of the consecutive participant number and takes into account non-finishers by releasing their group allocation back into the pool if they have not reached a debrief page within the time limit. This method (and indeed any method) will not cope well with most of the participants starting at the same time, since the allocations would be happening before it becomes clear whether other participants have finished or not.
However, the introduction of the Pavlovia Shelf in 2022 gives the possibility of a counterbalance solution without the use of external tools.
Introduced in 2024, I had hoped that this would provide an easy solution to using the Pavlovia Shelf for counterbalancing online. Unfortunately, it still isn’t stable enough for me to recommend it since it doesn’t yet take non-finishers into account, and seems to result in uneven distributions, unless you have a high completion rate and know how many participants you are going to get.
Bespoke counterbalancing using the Pavlovia Shelf.
I developed this code earlier this year for cases where I wanted even allocation to groups for two different types of participants (based on their responses to an embedded survey). However, it will also work for simple cases and also takes non-finishers into account and, with the help of Claude Sonnet 4.6 (Anthropic), I’ve extended the functionality to work locally as well as online.
Step 1: Create a list entry on the Pavlovia Shelf called group_allocations and populate it with one more zeros than the number of groups.
Step 2: Add a Both code component. The Python code is only needed if you would also like the experiment to run locally.
Begin Experiment Python
import json, tempfile
# Get the folder containing the psyexp file
experiment_folder = os.path.dirname(os.path.abspath(__file__))
allocations_file = os.path.join(experiment_folder, "group_allocations.json")
Begin Routine Python
if os.path.exists(allocations_file):
try:
with open(allocations_file, 'r') as f:
groupAllocations = json.load(f)["group_allocations"]
except Exception as e:
logging.warning(f"Warning: Could not load group allocations ({e}), starting fresh.")
groupAllocations = []
else:
groupAllocations = []
Begin Routine JavaScript
groupAllocations = await psychoJS.shelf.getListValue({key: ["group_allocations"], defaultValue: []}); // Get values
Step 4: Add an Auto code component Begin Experiment
numberOfGroups = 4 # Groups will be numbered 1 to x
maxAllocationGap = 3 # 3 means groups can become 3 apart after allocation
incrementWhenPiloting = True
Begin Routine
if len(groupAllocations):
expInfo["session"] = groupAllocations[0] # Add session number to data file
minAllocation = min(groupAllocations) # ... needed to indicate groupAllocations is a list
else:
expInfo["session"] = 0
minAllocation = 0
groupAllocations = [0]
availableGroups = [] # Define list
for Idx in range(1, numberOfGroups + 1):
if len(groupAllocations) == Idx:
groupAllocations.append(0)
availableGroups.append(Idx)
elif not groupAllocations[Idx]:
groupAllocations[Idx] = 0
availableGroups.append(Idx)
elif groupAllocations[Idx] < (minAllocation + maxAllocationGap):
availableGroups.append(Idx)
shuffle(availableGroups)
group = availableGroups[0] # group takes positive integer values
expInfo["group"] = group # Add group allocation to data file
Finally, add a Both code component to your last routine. This ensures that the allocation for the current participant is only saved to the shelf if they complete the experiment.
End Routine Python
# If this code is in End Experiment, aborted sessions
# will count if escape is pressed twice. To avoid
# counting any aborted sessions, this code should be
# in End Routine of the final routine.
groupAllocations[0] += 1
groupAllocations[group] += 1
try:
dir_name = os.path.dirname(os.path.abspath(allocations_file))
with tempfile.NamedTemporaryFile('w', dir=dir_name, delete=False, suffix='.tmp') as tf:
json.dump({"group_allocations": groupAllocations}, tf, indent=2)
temp_path = tf.name
os.replace(temp_path, allocations_file)
except Exception as e:
logging.warning(f"Warning: Could not save group allocations: {e}")
try:
os.remove(temp_path)
except:
pass
End Routine JavaScript
if ((incrementWhenPiloting || (! PILOTING))) {
groupAllocationsCheck = await psychoJS.shelf.getListValue({key: ["group_allocations"], defaultValue: []}); // Update list from shelf
if (groupAllocationsCheck.length) {
groupAllocations = groupAllocationsCheck;
groupAllocations[0] += 1;
}
else {
groupAllocations[0] = 1; // First participant
}
groupAllocations[group] += 1;
await psychoJS.shelf.setListValue({key: ["group_allocations"], value: groupAllocations}) // Await needed here
}
The selectively show or skip an embedded survey, put a loop around the survey routine and set the number of repeats to 0 / False to skip and 1 / True to show. If you have created a group variable (for example using Counterbalancing online) with two groups (0 and 1) then you can show a survey to group 1 using group repeats and to group 0 using 1 - group repeats. If group has values 1 and 2 then these become group - 1 to show to group 2 and 2-group to show to group 1.
