Counterbalancing online
Random allocation to groups is easy, but by it’s very nature can easily result in unbalanced group sizes. For counterbalancing, the allocation needs to take into account the allocation of other participants.
I have written three external tools which offer solutions to this problem.
This tool adds a consecutive value for participant to the study URL which can be used to systematically allocate participants to different groups using the modulus function, e.g. group = int(expInfo['participant'])%3 would allocate participants to groups 0, 1 or 2.
This uses the same principle, but sends the participant to different study URLs on the same domain.
Both of these tools allocate equal numbers of participants to each group at the start of the experiment, but have no knowledge of whether the participants completed the study or not, so unequal numbers of completed participants in each group are still possible.
This is a more powerful external tool which allows the researcher to host the participant information sheet (so the study is only launched by participants who consent). The licensed version (£10 per year) allows counterbalanced allocation to groups independent of the consecutive participant number and takes into account non-finishers by releasing their group allocation back into the pool if they have not reached a debrief page within the time limit. This method (and indeed any method) will not cope well with most of the participants starting at the same time, since the allocations would be happening before it becomes clear whether other participants have finished or not.
However, the introduction of the Pavlovia Shelf in 2022 gives the possibility of a counterbalance solution without the use of external tools.
Introduced in 2024, I had hoped that this would provide an easy solution to using the Pavlovia Shelf for counterbalancing online. Unfortunately, it still isn’t stable enough for me to recommend it since it doesn’t yet take non-finishers into account, and seems to result in uneven distributions, unless you have a high completion rate and know how many participants you are going to get.
- Bespoke counterbalancing using the Pavlovia Shelf.
I developed this code earlier this year for cases where I wanted even allocation to groups for two different types of participants (based on their responses to an embedded survey). However, it will also work for simple cases and also takes non-finishers into account and, with the help of Claude Sonnet 4.6 (Anthropic), I’ve extended the functionality to work locally as well as online.
Step 1: Create a list entry on the Pavlovia Shelf called group_allocations and populate it with one more zeros than the number of groups.
Step 2: Add a Both code component. The Python code is only needed if you would also like the experiment to run locally.
Begin Experiment Python
import json, tempfile
# Get the folder containing the psyexp file
experiment_folder = os.path.dirname(os.path.abspath(__file__))
allocations_file = os.path.join(experiment_folder, "group_allocations.json")
Begin Routine Python
if os.path.exists(allocations_file):
try:
with open(allocations_file, 'r') as f:
groupAllocations = json.load(f)["group_allocations"]
except Exception as e:
logging.warning(f"Warning: Could not load group allocations ({e}), starting fresh.")
groupAllocations = []
else:
groupAllocations = []
Begin Routine JavaScript
groupAllocations = await psychoJS.shelf.getListValue({key: ["group_allocations"], defaultValue: []}); // Get values
Step 4: Add an Auto code component
Begin Experiment
numberOfGroups = 4 # Groups will be numbered 1 to x
maxAllocationGap = 3 # 3 means groups can become 3 apart after allocation
incrementWhenPiloting = True
Begin Routine
if len(groupAllocations):
expInfo["session"] = groupAllocations[0] # Add session number to data file
minAllocation = min(groupAllocations) # ... needed to indicate groupAllocations is a list
else:
expInfo["session"] = 0
minAllocation = 0
groupAllocations = [0]
availableGroups = [] # Define list
for Idx in range(1, numberOfGroups + 1):
if len(groupAllocations) == Idx:
groupAllocations.append(0)
availableGroups.append(Idx)
elif not groupAllocations[Idx]:
groupAllocations[Idx] = 0
availableGroups.append(Idx)
elif groupAllocations[Idx] < (minAllocation + maxAllocationGap):
availableGroups.append(Idx)
shuffle(availableGroups)
group = availableGroups[0] # group takes positive integer values
expInfo["group"] = group # Add group allocation to data file
Finally, add a Both code component to your last routine. This ensures that the allocation for the current participant is only saved to the shelf if they complete the experiment.
End Routine Python
# If this code is in End Experiment, aborted sessions
# will count if escape is pressed twice. To avoid
# counting any aborted sessions, this code should be
# in End Routine of the final routine.
groupAllocations[0] += 1
groupAllocations[group] += 1
try:
dir_name = os.path.dirname(os.path.abspath(allocations_file))
with tempfile.NamedTemporaryFile('w', dir=dir_name, delete=False, suffix='.tmp') as tf:
json.dump({"group_allocations": groupAllocations}, tf, indent=2)
temp_path = tf.name
os.replace(temp_path, allocations_file)
except Exception as e:
logging.warning(f"Warning: Could not save group allocations: {e}")
try:
os.remove(temp_path)
except:
pass
End Routine JavaScript
if ((incrementWhenPiloting || (! PILOTING))) {
groupAllocationsCheck = await psychoJS.shelf.getListValue({key: ["group_allocations"], defaultValue: []}); // Update list from shelf
if (groupAllocationsCheck.length) {
groupAllocations = groupAllocationsCheck;
groupAllocations[0] += 1;
}
else {
groupAllocations[0] = 1; // First participant
}
groupAllocations[group] += 1;
await psychoJS.shelf.setListValue({key: ["group_allocations"], value: groupAllocations}) // Await needed here
}
If you use this code in your studies, please cite: Morys-Carter, W.L. (2026, April 21). Allocate Group [Computer software]. Pavlovia. https://gitlab.pavlovia.org/vespr/allocate-group
The code is also available from this thread: