Simplest code to select random rows of two alternating excel files (avoiding delays along the experiment)

[Paula]

Win10 / version 2021.1.2

Hello,

I am stuck with a problem in my experiment. It seems that the number of calculations I want to perform prior to each trial is generating a delay that gets bigger as the experiment advances. The consequence is that my ITI gets longer than it should and that the stimuli are presented on the screen during less time thay they should.

In previous versions of this experiment the number of calculations for each trial were considerable (I guess),
After I asked for help in the forum (Inter-Trial Interval (ITI) gets longer as the experiment advances - Builder - PsychoPy and Most efficient way to set the position and orientation of stimuli randomly on every trial - #5 by Paula - Coding - PsychoPy) I have simplified the code a lot, but still, I need (or I think I need) a minimum code to do what I want to do.

Participants must do one task (the CATEGORIZATION task) in odd trials and a different task (the SELECTION task) on even trials.

This is how the flow of these two tasks looks like

The CATEGORIZATION trials are specified in the first most inner loop. The loop is linked to the CAT excel file, which has 16 different conditions.

The SELECTION trials are specified in the second most inner loop. The loop is linked to the SEL excel file, which also has 16 different conditions.

I want the program to present the CAT conditions randomly without replacement on odd trials and the SEL conditions randomly without replacement on even trials.

To do so, a code component prior to this flow specifies the following:

cat_indices=list(range(16)) #creates a list of 16 numbers
shuffle(cat_indices) #suffles them
cat_selected_rows=cat_indices[-1:] #takes one number that will be used in the field "row" in the following cat loop. 

sel_indices=list(range(16)) #same with the SEL conditions.
shuffle(sel_indices)
sel_selected_rows=sel_indices[-1:]

When the program has run one CAT trial and one SEL trial, I need to eliminate the row that has just been used and select a new one. To do so, after one repetition of those inner loops has occurred, I have a code component as this one:

cat_indices=cat_indices[:-1] #eliminates the CAT row that has just been used.
cat_selected_rows=cat_indices[-1:] #takes a new number from the shuffled list.

sel_indices=sel_indices[:-1] #same for SEL conditions
sel_selected_rows=sel_indices[-1:]

As I am eliminating one number from the shuffled lists after each trial, once the 16 cat trials and the 16 sel trials have been run (once a big block of 32 trials has been run), the shuffled lists would be empty. Since I want to run several times that big block of 32 trials, once the 32 trials have passed, the program goes back to the first code of this post.

This is pretty much all the code I have in the experiment. Another code is used to provide the correct feedback on each trial.

I case it is useful, this is how my inner loops are specified.

I have tried to:

1. replace the code
   cat_selected_rows=cat_indices[-1:]
   cat_indices=cat_indices[:-1]
   with
   cat_selected_rows=cat_indices.pop()

but I get the following error message:
File “C:\Program Files\PsychoPy3\lib\site-packages\psychopy\data\utils.py”, line 438, in importConditions
elif len(selection) > 0:
TypeError: object of type ‘int’ has no len()

2. I have also included the second code component of this post in the ITI routine (whith the hope that those calculations would be done during the ITI), and set the ITI to be 2 seconds instead of 1 second, but the problem is still there.

I am a bit confused with this problem, because, as I mentioned, I have simplified my code considerably compared to prior versions (e.g, prior versions determined randomly the position and orientation of my stimuli before each trial, also, the two different condition files were “called” with a code component) but I still have this problem.

To give an idea of the problem… the stimuli that should be on the screen for two seconds, end up lasting around 1.30 secs.

I would appreciate any idea on how to solve this

Paula.

[jderrfuss]

Hi @Paula,

Spontaneously, I can think of two possible solutions (if I have understood your description correctly):

• If your cat and sel trials are similar in structure (i.e., use the same components and the same timing), you might not need two separate loops for these trials
  • Instead, you could randomise lists of your cat and sel stimuli independently and then zip the two lists (such that cat and sel stimuli alternate)
  • This would be conceptually similar to the approach that @wakecarter suggested here
• If cat and sel trials however do differ in their structure, I would tend to put an extra loop around each of the inner loops; the sole function of these extra loops would be to determine whether or not the respective inner loop runs based on a modulo division of a trial counter

Hope this helps.

Jan

[Paula]

It does help a lot!

The cat and sel trials are very similar and I can use a single routine and loop for both.
So I am going to try and use your first approach, either using @wakecarter interleaved lists method, or another way to randomize lists and zip them in an alternating manner.

At this point, I wonder whether these methods can be applied in the second part of my experiment. The structure is the same (alternating 2 types of trials) but, instead of running all trials from the conditions file in a row, I need to run 8 intermixed trials of types A (cat) and B (sel), then 8 intermixed trials of types C (testcat) and D (testsel), then go back again to A and B…, and so on.

That is, I wonder whether I can “pause” the random presentation of conditions after 8 trials, and continue with the random list in the following 8-trials repetitions.

In case it helps, the following figure represents what I want to do in the whole experiment

Do you think that would be possible with either method?

Thanks again!

Paula.

[wakecarter]

You could definitely use my method to pop 8 each from A and B then switch to popping from C and D then back again.

[Paula]

Okey, that “definitely” word makes me very happy

I going to try and do it, and update this topic with the result.

Thank you!

[Paula]

Hello again @wakecarter

I have taken this begin-experiment code you provide…

#this is entirely your code
targetJitter = 5
putTarget = 10+randomint(0,targetJitter)
targetGap = 20 #min gap between targets
fillers=[]
targets=[]
trialIdx=-1

and adapted it to my experiment in this way (i consider your targets to be my CAT trials and your fillers to be my SEL trials)

catJitter = 1
putcat = 10+randint(0,catJitter)
catGap = 1 #min gap between targets
sels=[]
print(len(sels))
cats=[]
print(len(cats))
trialIdx=-1

First thing is that I have replaced your word randomint with the word randint because I was getting the error that “randomint was not defined” (I hope not having screwed anything).

Now, the main problem is that I get the following error:

thisItem=sels.pop()
IndexError: pop from empty list

After reading in the forum I am pretty sure I have to put a list of values inside the [ ] in the cats=[ ] and sels=[ ] part. Since I have 16 cat conditions and 16 sel conditions I have tried with cats=[1:16] and sels=[1:16] but then I get this error.

Item=thisItem[0]
TypeError: ‘int’ object is not subscriptable

and also receive this warning in the routine itself (in the JS part)
/ Syntax Error: Fix Python code /

Could you give me some advise with this?

Also…

In these other parts of the code:

catJitter = 1
putcat = 10+randint(0,catJitter)
catGap = 1 #min gap between targets

I have put numbers based on what I think these variables refer to, but with no idea of what I am doing. If you could give me a hint on what these variables refer to, that would be great. especially the “putcat” variable is a mystery.

I have gone through your paper here to see if the experimental procedure could give me some idea but I could not deduce those variable values based on what is in the paper (which is normal, I just had to try).

Paula.

[wakecarter]

Thanks for spotting my randint / randomint typo.

However, if you are trying to alternate between two lists then you don’t need catJitter, putcat or catGap. Instead you could use.

if trialIdx%2 == 1:
     thisItem=sels.pop()
else:
     thisItem=cats.pop()

If you want some participants to start with cats and some with sels then you could use putcat = randint(0,1)

and then if trialIdx%2 == putcat:

However, the main thing you seem to be missing is appending items from your Excel files to sels and cats. .pop() is failing because you’re trying to select the last item of an empty list.

Begin Routine (in loop attached to fillers)
fillers.append([Item.ItemType,ItemAnswer])
Begin Routine (in loop attached to targets)
targets.append([Target,TargetType,TargetAnswer])

I usually have the fillers and targets in the same spreadsheet and therefore have to add for example 0:12 to Selected Rows if there are 12 targets.

cats=[1:16] is not doing what you want.

[Paula]

Thanks for your response @wakecarter,

Yes, It seems that the problem is in the “append” part. I have tried to do it in several ways but still, I get the same error message.

thisItem=sels.pop()
IndexError: pop from empty list

I think the problem could be in either the content of the code, its location within the loops or even that I have to use certain names in the routines or exelfiles.

I am going to go through these things to see if you notice something wrong. I’ll try to be brief.

INITIAL CODE

As the catJitter, putcat and catgap are not needed, I have adapted your code from

targetGap = 20 #min gap between targets
targetJitter = 5 #max variation in gap
putTarget = 10+randint(0,targetJitter)
fillers=[]
targets=[]
trialIdx=-1

to be simply…

sels=[]
cats=[]
trialIdx=-1

I have that code at the very beginning of the experiment

1ST APPEND CODE (cats)
I have adapted your code from…

Begin Routine (in loop attached to fillers)
fillers.append([Item.ItemType,ItemAnswer])

to…

cats.append([trial_type,color_set,relevant_letter,irrelevant_letter,relevantColor,irrelevantColor,response,orientation,Position,orientrel,orientirrel,posrel,posirrel,prompt_up,prompt_middle,prompt_down,prompt_color,relevant_color,irrelevant_color,allowed_keys,corrAns])

because I understood that within the I have to put the column names of my excel files (each column defines a variable to be used within the trial)

I have placed that code within the cats routine that contains the stimuli to be presented.

Note that the loop also includes the feedback and ITI routines.

This is how the loop is defined

2ND APPEND CODE (sels)
Same as the append code for cats but instead of cats i write “sels”

sels.append([trial_type,color_set,relevant_letter..... etc.

The code is located in the same place as the append code for cats but in the sels routine.

The sels loops is defined as the cats loop but the row columns are 16:31 and the excel file is called sels.xlsx (despite is exactly the same as the cats.xlsx excel file)

ALTERNATION CODE
Finally…
I have adapted your code:

#Begin Routine (in trials loop which should have no spreadsheet and a large number of nReps)
trialIdx+=1
if trialIdx == putTarget and len(targets)>0:
     thisItem=targets.pop()
     putTarget=trialIdx + targetGap + randint(0,targetJitter)
     thisExp.addData('Target',1)
else:
     thisItem=fillers.pop()
     thisExp.addData('Target',0)
Item=thisItem[0]
Type=thisItem[1]
corrAns=thisItem[2]
thisExp.addData('Item',Item)
thisExp.addData('Type',Type)
thisExp.addData('corrAns',corrAns)
if len(fillers)==0:
     trials.finished=True

to be…

trialIdx+=1
if trialIdx%2 == 1:
     thisItem=cats.pop()
     thisExp.addData('cats',1)
else:
     thisItem=sels.pop()
     thisExp.addData('sels',0)
trial_type=thisItem[0] #1st column in my excels contains the variable trial_type
color_set=thisItem[1]#2nd column in my excels contains the variable color_set
relevant_letter=thisItem[2]#etc.
irrelevant_letter=thisItem[3]
relevantColor=thisItem[4]
irrelevantColor=thisItem[5]
response=thisItem[6]
orientation=thisItem[7]
Position=thisItem[8]
orientrel=thisItem[9]
orientirrel=thisItem[10]
posrel=thisItem[11]
posirrel=thisItem[12]
prompt_up=thisItem[13]
prompt_middle=thisItem[14]
prompt_down=thisItem[15]
prompt_color=thisItem[16]
relevant_color=thisItem[17]
irrelevant_color=thisItem[18]
allowed_keys=thisItem[19]
corrAns=thisItem[20]
thisExp.addData('trial_type',trial_type)
thisExp.addData('color_set',color_set)
thisExp.addData('relevant_letter',relevant_letter)
thisExp.addData('irrelevant_letter',irrelevant_letter)
thisExp.addData('relevantColor',relevantColor)
thisExp.addData('irrelevantColor',irrelevantColor)
thisExp.addData('response',response)
thisExp.addData('orientation',orientation)
thisExp.addData('Position',Position)
thisExp.addData('orientrel',orientrel)
thisExp.addData('orientirrel',orientirrel)
thisExp.addData('posrel',posrel)
thisExp.addData('posirrel',posirrel)
thisExp.addData('prompt_up',prompt_up)
thisExp.addData('prompt_middle',prompt_middle)
thisExp.addData('prompt_down',prompt_down)
thisExp.addData('prompt_color',prompt_color)
thisExp.addData('relevant_color',relevant_color)
thisExp.addData('irrelevant_color',irrelevant_color)
thisExp.addData('allowed_keys',allowed_keys)
thisExp.addData('corrAns',corrAns)
if len(sels)==0:
     trials.finished=True

And I have that code placed within the outer loop and before the other two loops

Finally, I am going to run the experiment locally (not online), in case it matters.

Thanks for your time,

Paula.

[wakecarter]

You need to load the spreadsheets in dedicated loops before your trials loop. The trials loop doesn’t then point to the spreadsheets

[Paula]

Okey, I am closer (but still a bit far)

Now the problem is that a single trial is presented. It is as if the trials loop only had one repetition. there are no error messages, it simply presents a single trial.

I have put the append codes in dedicated loops linked to the excel files, and the “alternation” code is within the stimuli routine

I wonder whether the loops are well defined. but I have tried several combinations of “sequential” and “random” and nothing seems to change.

[wakecarter]

You need to give the trials loop a large value for nReps and break out early in code or the correct number for your number of trials

[Paula]

I have tried with 128 (the number of trials I want) and with 500 and still, it presents only 1 trial.

With 5000 I get this error message

** File “C:\Program Files\PsychoPy3\lib\site-packages\openpyxl\utils\cell.py”, line 113, in get_column_letter
raise ValueError(“Invalid column index {0}”.format(idx))
ValueError: Invalid column index 25003**

[Paula]

Sorry, there was an error in one of my codes. Now it seems to work!
I am going to do some checks and update the topic with the results.

Thanks for all your help

Paula.

[Paula]

For anyone having the same problem, the Interleaved Lists
method worked perfectly!

Thanks @jderrfuss and @wakecarter!

Best,

Paula.

[wakecarter]

Your welcome. Please do let me know there’s anything that you think could do with being clearer in my description of it.

[Paula]

For beginners (as I am) it would help to have a kind of glossary of what each variable means, or if variables are expressed in a more general manner, like…

fillers.append([variable1, variable2, variable3,…])

or

fillers.append([excelcolumn1,excelcolumn2,… )

@wakecarter

You need to load the spreadsheets in dedicated loops before your trials loop.

Clarifying this would also help (I had never used a loop to simply load a conditions file)

[wakecarter]

Thanks. I’ve made some edits.

(And thanks for the virtual coffee)

[wakecarter]

You might be interested in my new demo

Independent Randomisation code | try it

Pre-load one column from a spreadsheet so that the contents can be displayed in an independent random order alongside the contents of another column. This demo also uses shuffle to randomise the location of the target.

[Paula]

Thanks @wakecarter! I’ll definitely try it

[Paula]

seems elegant and it is explained in detail for dummies like me