Random vs Counterbalanced
There are three main levels of randomisation that typically exist within an experiment.
- Random order of trials
- Random order of conditions
- Random assignment to groups
Order of trials is set either by setting the loop to sequential, random or fullRandom. When the loop is sequential, then the trials themselves could be an a specific order (e.g. of increasing difficulty), a pre-randomised order or a shuffled order of selected rows. Having a random order has the advantage of being easier to set up – you don’t need to worry about randomising your spreadsheet. On the other hand, it does add noise to your data. Some participants may get longer runs of similar trials than others. If you have four trial types then there is about a 25% chance that you could get a run of at least five in a row of the same type in a block of 100 trials.
If all your participants get the same random order, this will reduce the noise in your data, but you risk having artefacts caused by the specific random order you chose.
Randomising the order of your conditions is a far bigger issue. Later conditions may have worse performance due to fatigue or improved performance due to practice. If you have a small number of different conditions (two or three) and a large number of participants, the easiest option is to randomise the order and record the order chosen as a variable you can include in your analysis. With a larger number of conditions you may need to select a subset rather than allowing all possible permutations if you want to include the order as a variable. Alternatively, you may decide to fix the order, especially if your conditions are different tasks so practice is less likely to be important.
If you think order is likely to affect your results and you have tens rather than hundreds of participants, the best option is probably to counterbalance the order to ensure that similar numbers of participants are assigned to each order. This advice also goes for other between-participants variables. If you are assigning 100 participants to four groups then there is a 95% probability that each group will contain between 17 and 33 participants, but that still means that your largest group could be nearly twice as big as your smallest. If this is too much potential variation then you need to counterbalance.
I won’t go into the details of how to counterbalance in this tip but there are three basic methods.
- Assign each participant a consecutive participant number and allocate based on participant number % (modulo) number of groups.
- Assign each participant to a group using a counterbalance routine or app.
- As above but also take into account non-finishers so that if a participant doesn’t complete your experiment their group assignment can be reassigned to a later participant.