Full Str not Added to Dictionary in Loop

Coding
1

Hi everyone!

This may not be a PsychoPy specific problem, but for some reason is one of my major stumps on this massive experiment using PsychoPy.

Essentially, I am running into an issue where my dictionary is not picking up full strings attached to variables, but instead just the first letter. I made a small example of what happens, but I would be happy to attach my full/real code if that would be helpful.

import numpy as np 
import random

trials = 5

d = {'trial': np.empty([trials]),
     'fruit': np.empty([trials], dtype = str),
     'vegetable': np.empty([trials], dtype = str)}

fruits = ['apple', 'orange', 'banana']
vegetables = ['carrot', 'squash', 'sprout']

for i in range(trials): 
    fruit = random.choice(fruits)
    vegetable = random.choice(vegetables)

    # trial data 
    d['trial'][i] = i
    d['fruit'][i] = fruit
    d['vegetable'][i] = vegetable
    print(d)

Like in my real experiment, this code just takes the first letter of the fruit/vegetable to add to my dictionary. This is messing with me, as I need to record ‘lvf’ and ‘lower’ in terms of visual fields in my real experiment, but they both appear as ‘l’ in my dictionary.

I feel the solution may be simple, I just can’t seem to find it anywhere. I have tried adding ‘list()’ around my variables, but that didn’t help. For example:

fruit = random.choice(list(fruits))

If anyone has insight to this issue, I would greatly appreciate it! Thank you all for your help!

2

Hello @lilih

I think that the issue lies in how NumPy pre-allocates memory for strings. When you use dtype=str without specifying a length, NumPy defaults to allocating space for exactly one character. When you try to assign a longer word like lvf or lower, NumPy strictly truncates the string to the first letter so it fits into that tiny memory slot.

There are two ways to solve this.

Option 1 is to use the object data type. This is usually the best approach for trial matrices. By using dtype=object instead of str, NumPy stores references to standard Python strings of any length. Just change your empty array creation to use dtype=object.

import numpy as np
import random

trials = 5

# that is different to your code
d = {'trial': np.empty([trials]),
     'fruit': np.empty([trials], dtype=object),
     'vegetable': np.empty([trials], dtype=object)}

fruits = ['apple', 'orange', 'banana']
vegetables = ['carrot', 'squash', 'sprout']

for i in range(trials): 
    fruit = random.choice(fruits)
    vegetable = random.choice(vegetables)

    # trial data 
    d['trial'][i] = i
    d['fruit'][i] = fruit
    d['vegetable'][i] = vegetable
    print(d)

Option 2 is to specify a maximum string length. If you want to keep strict string arrays, tell NumPy the maximum number of characters. For example, using dtype=U20 reserves space for strings up to 20 characters long.

import numpy as np
import random

trials = 5

# this is different, set string length to 20 characters
d = {'trial': np.empty([trials]),
     'fruit': np.empty([trials], dtype='U20'),
     'vegetable': np.empty([trials], dtype='U20')}

fruits = ['apple', 'orange', 'banana']
vegetables = ['carrot', 'squash', 'sprout']

for i in range(trials): 
    fruit = random.choice(fruits)
    vegetable = random.choice(vegetables)

    # trial data 
    d['trial'][i] = i
    d['fruit'][i] = fruit
    d['vegetable'][i] = vegetable
    print(d)

Using the object type should fix the dictionary in PsychoPy immediately and record your data correctly.

Best wishes Jens

3

On a related idea, I can’t find a definitive answer on whether pre allocating
a numpy dtype= ‘U20’ or a list of strings is faster

Two issues:

does the preallocation speed up the code, and does pushing, appending, indexing etc as the contents are stored on the fly alter the speed?

Shorter question. If I am going to add contents to some object on each trial, what is the fastest way to do this?

4

Hello @ben

Standard Python lists are inherently fast for storing strings. NumPy is highly optimised for numerical data, so forcing strings into fixed-size C arrays (like dtype=‘U20’) adds unnecessary processing overhead.

In NumPy pre-allocation is strictly required. Using the function np.append() is slow because the entire array has to be completely rebuilt and copied every time.

Python lists: Although the .append() method is highly optimised in Python, pre-allocating the list is slightly faster because the memory space does not need to grow dynamically.

So, it is probably best to use a pre-allocated Python list combined with indexing.

# Pre-allocate before the experiment starts:
fruit_data = [None] * number_of_trials 

# Assign on the fly via indexing:
fruit_data[i] = fruit

However, I assume that the code mentioned above is not executed during the time-critical periods of stimulus presentation and reaction time measurement in the experiment, but rather at the beginning. Therefore, it probably does not matter which approach is taken.

Best wishes Jens

5

That is helpful. I remember an experience I had with Octave years ago where each append had a huge time cost. On hardware from 25 years ago the overhead made the script I wrote almost unusable.

The list I am building has an unknown number of appends on each trial. It is actually timestamps for certain trial events so it is inside the trial loop. The loop is supposed to take 1000ms and I have it optimized for between 1.5 and 0.5 ms slop. Perhaps all the logging is responsible for some of this.

A different list (alltrials) logs per-trial events and is of knowable length and structure. Each trial generates a small list of a few floats and strings which is then appended to the main alltrials list. Given 48 trials, initializing the alltrials list might make small benefit but is still worth trying.

On a related topic of reducing overhead, I found that dropping the screen resolution from 3840x2160 to 1280x720 greatly reduced the overhead. This is primarily an audio expt, so video resolution doesn’t really matter. Also, the tip about “You can make the draw() quick by calling re-setting the text (myTextStim.text = myTextStim.text) when you’ve changed the parameters.” really helped.

Tx

6

Hello @ben

Octave had a different mechanism at that time than Python has today. Octave required reallocation and copying every element each time. Python over-allocates memory when it grows a list.

The slop of 0.5 ms - 1.5 ms could be due to garbage collection (you can turn it off during the trial and turn it on after the trial (gc.disable(), gc.collect()), waiting for vsync, your OS is doing something, audio loading and playing (load before your trial starts).

Best wishes Jens

7

I had the vsync off but I found that drawing a small image ( < 200x200 pixels) added a lot of time on this old hardware. I got rid of the image and it makes the times much tighter.