Pandas functionality from PsychoPy for PsychoJS (pavlovia)

[NicBadcock]

URL of experiment: https://gitlab.pavlovia.org/nicbadcock/phonological-attentional-blink

Description of the problem:
I’m new to PsychoPy and attempting to create an experiment (including bits of code) within the builder to generate a PsychoJS script to use with pavlovia. I’d like to be able to import a stimulus file (different to a condition file that I use TrialHandler for) in order to create random lists of items for presentation (i.e., a rapid serial visual presentation) using matrix selection (i.e., give me all of the item in the ‘target’ column when the ‘target_number’ column is equal to ‘T2’).

I’ve got this working with the pandas library and it functions nicely in the builder (see PAB2020.py / PAB2020.psyexp). For example, creating a list of T2 (second target) items from a ‘target’ column contingent upon a ‘target_number’ column being equal to ‘T2’:

T2list = list(stim['target'][(stim['target_number'] == 'T2')])

But the conversion from PsychoPy to PsychoJS fails with pandas included (consistent with some threads in the forum).

I’ve imported the stimulus file with TrialHandler but I can’t figure out how to handle the data to do matrix selection of stimuli. I am able to access rows nicely but the column functionality has me stumped.

targets = data.TrialHandler(nReps=0, method='sequential',
        originPath=-1,
        trialList=data.importConditions("C:\DATA\gitlab\PAB\PAB_stimuli_targets.xlsx"),
        seed=None, name='targets')
first_row = targets.trialList[0]

I have attempted converting the sheet to a ‘wide’ format but I’m not able to select multiple columns using logical tests (i.e., just end up with a single FALSE) value.

Any help for matrix selection of items within native PsychoPy functions would be ideal but maybe there are some libraries that operate with PsychoJS. Very open to suggestions.

Thanks,
Nic

PS I haven’t attempted to only code this in PsychoJS yet as I wanted to build it up piece by piece within the builder so I could learn PsychoJS but doing it a little more blindly is my fallback

[wakecarter]

Hi,

Please have a read through this thread

Alternative to pandas for reading csv/excel files online

Hello PsychoPy community, I have an experiment which uses the pandas package to import an excel file with lots of stimuli and associated distracter words. During the experiment I’m also using pandas-specific methods like .loc() and .sample() to shuffle and check my trials/words/conditions/etc. Since we can’t import extra packages for our online experiments, what alternatives would you recommend for these functions?

Best wishes

Wakefield

[NicBadcock]

Thanks Wakefield,
This is helpful lead. It gets the data into lists that I can work with but the logical tests are only applying to the first item in the lists. It’s clear there’s something else for me to learn here but this is a massive help - thank you!
Nic

[wakecarter]

Hi Nic

Please show me an example of the code you’re using to test items in the list

Best wishes

Wakefield

[NicBadcock]

It’d be great to have your thoughts on this - thank you:

# initialize the big dictionary...
expVars = {}

# import csv file
csvData = data.importConditions("C:\DATA\gitlab\PAB\PAB_stimuli_targets.csv")

# meld all dictionaries together and store in the big one
for k in csvData[0]:
    expVars[k] = [d[k] for d in csvData]

# pulling the lists out for easier access
target_list = list(expVars['target'])
target_numbers = list(expVars['target_number'])

# trying to select a subset of the 'target_list' based on the 'target_numbers' variable
T2list = target_list[target_numbers == 'T2']
# this just applies to the first item, so I just get 1

Here’s a Dropbox link to the csv.

Any thoughts would be very welcome,
Thanks again,
Nic

[wakecarter]

I’m not familiar with functionalities of dictionaries, but it seems like that’s not the appropriate code to create a subset.

[NicBadcock]

I think you’re right - I guess that’s one of the advantages of pandas. I’ll keep playing.
Cheers,
Nic

[aisa2]

@NicBadcock if you are like me, coming from Matlab or R, this can be extremely frustrating, I know! There are some very simple things that we do reflexively that don’t seem to work here.

Try a list comprehension? I’m not sure if there’s a better way, but this has worked for me in some situtaions.

T2list = target_list[n == 'T2' for n in target_numbers]

The other thing that you might want to look into is making your lists of interest into np.array() when you create them, e.g.:

target_list = np.array([x for x in expVars['target']])

… I found that arrays can be acted upon with logical operations more easily.

I don’t know how much work it would be to add some pandas functionality to base psychopy, but it seems like it would be very useful.

[wakecarter]

I don’t think np.array is going to work online.

[aisa2]

@wakecarter Oh? Do you know why that’s the case? I was using it because I saw that psychopy automatically imports the entire np library on line ~23 of each script generated by builder.

import numpy as np  # whole numpy lib is available, prepend 'np.' 

If it doesn’t work, that would be worth noting in the crib sheet more explicitly.

[wakecarter]

Wildo. This message suggests that numpy is Python specific:

Problems with running experiment online

OK, there’s the error message and then there’s something else that’s unrelated and will make this not work, so let’s try to tackle both at once. The error message I played around with, it’s coming from ‘word_{}’.format(). I think that the javascript translator doesn’t know how to handle the string.format function. Happily, you can work around it pretty easily: trials.addData('word_'+str(rts), rtList[rts]) It should stop giving you the error at that point and produce some javascript, but I…

[aisa2]

Drat, I was hoping arrays were going to solve all of my problems

[wakecarter]

What are you trying to do? JavaScript will work with two dimensional arrays in the same way as Python arrayName[index1][index2]

You could have a loop:

subSet=[]

for Idx in range(len(arrayName)):
      if arrayName[Idx][1] == "test":
           subSet.append(arrayName[Idx][0])

[aisa2]

So, I have less experience with Python than some other languages, but I think the problem may be that both @NicBadcock and I imported our csv files as a dictionary of lists. Now we are trying to use some criteria in one column of the csv file (now a list) to access another column. It’s an operation I do quite frequently in my script, and it would be nice to not have to write a loop each time. Unlike in some other languages, logical operations like:

T2list = target_list[target_numbers == 'T2']

won’t work since [target_numbers == 'T2'] only outputs 1 value, I think

…Also don’t Python arrays have to be all the same type of data? Because @NicBadcock and I both have mixed data (some strings, some ints, etc.), which complicates things.

[aisa2]

Also, I don’t know if this helps, but when I use np.array, the Auto->JS translation gives something like this:

wordKeyCheck = np.array(function () {
    var _pj_a = [], _pj_b = wordKey;
    for (var _pj_c = 0, _pj_d = _pj_b.length; (_pj_c < _pj_d); _pj_c += 1) {
        var w = _pj_b[_pj_c];
        _pj_a.push(_pj.in_es6(w, parsedResponse));
    }
    return _pj_a;
}
.call(this));

…so maybe it is translating this function after all?

[wakecarter]

Python arrays can be of mixed type, I.e. value[0][0] can be of a different type from value[0][1]

[aisa2]

Okay, maybe I need to figure out a bit more about regular python arrays then. Thanks! (and sorry for co-opting the thread…)

[NicBadcock]

Thanks @wakecarter and @aisa2 - lots of ideas to play with - just need to figure out how to get the MATLAB and R functionality that I’m used to. I’ll let you know if I come up with anything.