psychopy.org | Reference | Downloads | Github

Inability to call createTrialHandlerRecordTable in several loops

Hello! I am currently playing around with the new builder eye tracking components (they make eye tracking setup a breeze!), and I’ve gotten stuck when trying to create more than one TrialHandlerRecordTable in the same experiment.
Basic info about the task:
I created a basic testing design, with a loop called trials and a routine within, where i called ioServer.createTrialHandlerRecordTable(trials). This loop used an excel file just to register some dummy variables
Description of the problem:
Everything was working just fine, but when i added a second loop that called createTrialHandlerRecordTable a second time to create a second table that gathered the variables of my second loop (say, ioServer.createTrialHandlerRecordTable(trials_2)), it did not actually create the second table; the Runner raises a KeyError (code below).

File "D:\Descargas\EyeTrackerBETATESTING\BETATEST_lastrun.py", line 524, in <module>
    ioServer.addTrialHandlerRecord(thisTrial)
  File "D:\Descargas\Psychopy3\lib\site-packages\psychopy\iohub\client\__init__.py", line 644, in addTrialHandlerRecord
    data.append(cv_row[cv_name])
KeyError: 'TestVariable4'

Perhaps ioHub is unable to create more than one table for appending into the hdf5 file? (thus the KeyError would maybe come from an assignation of the differently named variables of the second loop into the table that has columns representing those of the first one.

If i disabled the creation of the second table, and only called ioServer.addTrialHandlerRecord(thisTrial), the values of the new rows are the same of the last trial of the first loop; image for reference

Is there any other way to create several ioHub TrialHandler tables? Since the main utility i’m trying to accomplish is labeling eye tracking data, a workaround for this problem could be using messages to send the values of the different variables in the loops to ioHub and then using pandas (or R) to extract the different values and correctly label each eye tracking refresh

You are correct, iohub only supports creating one conditions table in an hdf5 data file. When createTrialHandlerRecordTable(trials) is called, a trial in trials must have all the columns that need to be written to during the experiment, otherwise you will get a key error.

As you suggest, a work around would be to use experiment messages to save the variable values to the hdf5 file and then parse them out. Not as easy / nice as using a conditions table, but workable.

Are your two conditions files within nested loops, or are they in sequential loops in the experiment?

Sequential; I could try nesting the two smaller loops within a bigger loop that gathered all the different variables of all loops and creating a single table outside of the smaller loops. I’ll try calling addTrialHandlerRecord to see if the table fills the corresponding variables

I’ll edit this post with the outcome in a second
Edit: Still not working; after creating the table in a routine outside of the smaller loops and creating the table in the “allvariablegatherer” routine,


the same KeyError is raised when ioHub tries to add the trial data to the table

Traceback (most recent call last):
  File "D:\Descargas\EyeTrackerBETATESTING\BETATEST_lastrun.py", line 451, in <module>
    ioServer.addTrialHandlerRecord(thisTrial)
  File "D:\Descargas\Psychopy3\lib\site-packages\psychopy\iohub\client\__init__.py", line 644, in addTrialHandlerRecord
    data.append(cv_row[cv_name])
KeyError: 'TestVariable4'

This could work, but I think you would need to make a single call to addTrialHandlerRecord with all the variable values for the single ‘combined’ trial conditions file you created. The variable names in this file will (I think) need to be different then the ones you actually use in your experiment, so if you are using a column called testVariable1 in your experiment, the combined table column name could be testVariable1_ or something.

I see; I actually managed to fix the problem that was raised in my earlier post!: the inner loop file needs to have the exact same variables than the outer loop file, and their datatype has to be the same (e.g., if you fill one of them with integers and the other with a string you will get another error). I tried calling addTrialHandlerRecord inside each inner loop, and i found the same problem i was getting earlier, where the data from the second loop was getting improperly filled with the values of the last trial of the first loop.


I noticed, however, than the TRIAL_START and TRIAL_END variables were correctly increasing each trial, so i manually called

thisTrial['TestVariable1'] = ''

on routine end, and voila!

So it seems the recipe for having several different loops while using ioHub and recording variable values in the TrialHandlerRecordTable is to: (a) create an outer loop that encompasses your trial blocks, with an excel file that gathers all different excel variables, and dummy values in each column that will have data, of the same datatype as the real variable values, as an example: image
**
(b) Have a routine inside your outer loop, before your inner loops, where you create the table for the outer loop. As my outer loop was called trials_3, i used this in the Begin Routine section of a code component;

if trials_3.thisN == 0:
    ioServer.createTrialHandlerRecordTable(trials_3)

(c) Your inner loop condition files MUST contain the exact same variable names (and order, just in case).
(d) You can register your first loop variables by calling ioServer.addTrialHandlerRecord(thisTrial) at the end of the routine inside the first loop, but your second block will not register the values from the excel at all. In order to do so, you can manually update the values of your variables inside the code component of your routine in the End Routine tab before calling addTrialHandlerRecord (for example: image
You can even use python loops, list comprehension or packing/unpacking to update all the variables at the same time.
Within the hdf5 file, all the variables you update in a code component within your routines will get correctly updated and added to your table, and then you can use it to analyze your data with the ease that the table allows for


Thank you very much for your assistance and the idea to use nested loops, which absolutely ended up solving the problem

edit: it is worth noting that, since you can only have one table per experiment, you will probably need to add a variable to register the block you are in if you want to analyze your blocks in a separate manner

I do not think you need to have the outer trials_3 loop. Here is what I would try:

  1. Create a .xlsx conditions file that contains all the columns you want across both trial loops in your hdf5 condition variables table. I called it hdf5_conditions.xlsx. It only needs to have 1 row / trial worth of data in it since it is only being used to tell iohub what conditions to expect and what the default values that should be if it is not updated for any trial.

  2. In your project custom code where you create the hdf5 conditions table (near the start of the experiment), update it to use the new hdf5_conditions file and access an hdf5_trial instance from it:

    hdf5_conditions = importConditions('hdf5_conditions.xlsx')
    hdf5_trials = TrialHandler(hdf5_conditions, 1)
    io_hub.createTrialHandlerRecordTable(hdf5_trials)
  1. To reset the condition variables to default values, so you do not need to update every column on every trial, get a copy of the first hdf5_trial at the start of each trial loop:
    hdf5_trial = copy.deepcopy(hdf5_trials.trialList[0])
  1. Now, whenever you need to set values for the current trial, use code like:
    hdf5_trial['trial_id'] = t+1
    hdf5_trial['testVariable1'] = 'test1'
  1. to save a row to the hdf5 conditions table, use:
    io_hub.addTrialHandlerRecord(hdf5_trial)

Each time this is called, a row will be added to the hdf5 conditions table containing the current state of hdf5_trial.

Edit: I see you figured out a workaround yourself; nicely done.
Edit2: Updated example code to show how to reset condition defaults at the start of each full trial.

1 Like

That will probably work just as well, since you have to update the variables within the code regardless of nested loop usage, and the custom TrialHandler should do exactly the same as the outer loop TrialHandler (since it is really only used for creating the table)