| Reference | Downloads | Github

Incomplete data saving: only first 101 trials are saved

I’m a new user of Psychopy. It is a Serial Reaction Time task consisting in 10 blocks of 100 trials, however only 101 trials are saved on the .csv file.
Has anyone ever had this kind of issue or has any idea of possible causes?
The experiment displays the 1000 trials, so the problem is really located in the saving process.

Here is the piece of code:

def stim_pres():
    for i in range(len(d)):
        stimvis = visual.ImageStim(w, image = chem_img + '/' + d.Stimulus[i])
        h = core.Clock()
        resp = event.waitKeys(keyList = ['b','d', 'f', 'g', 'h', 'j', 'k'])
        rts = h.getTime()*1000
        d.iloc[i,2] = resp
        d.iloc[i,3] = rts
        if resp[0] == 'b':
            d.to_csv(output, index = False, header = True, sep= '\t')
            sys.exit('Fin de la session')
        elif resp[0] == d.CorrResp[i]:
            d.iloc[i,4] = 1
            d.iloc[i,4] = 0 
            tone = sound.Sound(value = chem_sd + '/beep1.wav')
        for nf in range(15):
            isi = visual.TextStim(w, text = "")

I tried both options to put the “d.to_csv” line in the for loop (saving after each trial) and after the for loop (saving only at the end of the experiment): the result is the same.

Many thanks!

OK, there are a couple of unrelated PsychoPy-style optimisations here to suggest:

  • don’t re-create stimulus objects each time you use them (that is time-intensive). Instead, instantiate them just once and then update their properties with, e.g. stimvis.image = 'something.png' or tone.setSound(blahblah). i.e. the lines stimvis = visual.ImageStim() and tone = sound.Sound() should occur before your loop starts, with just some dummy values provided. Updating their attributes later is much faster than re-creating it each time. That is certainly also the issue with drawing the text stimulus. Its content is constant, and yet your are currently re-creating it every 16 ms or so. Just instantiate it once, and then draw and flip it as needed.
  • in general, use PsychoPy’s core.quit() function to politely finish up, rather than sys.exit().

Now to the actual problem. You don’t describe what d is or what its dimensions are. But the symptoms sound like a classic case of overwriting the previous file on disk. i.e. if you are running 10 blocks of 100 trials, perhaps each block is overwriting the results from the previous one, so you only see 100 rows of data plus a header row from the final block.

You might find that it is worth investing in understanding how to use PsychoPy’s TrialHandler class instead of trying to reinvent the wheel here. It handles all this stuff for you (e.g. it automatically saves the data when you quit) and you don’t have to do all that pesky handling of indices. i.e. indexing values in an array is quite a non-Pythonic way to approach this: instead you could just iterate over a TrialHandler object, so you always have access to the current row of data, without needing to keep track of what that row number is. And you can refer to the variable names rather than by index. e.g. to store a response in a TrialHandler called trials:

trials.addData('response', resp)
trials.addData('latency', rts)

i.e. notice it is not necessary to know what trial number this is, as the TrialHandler simply keeps track of the current trial number for you. Similarly, you can refer to columns/variables by name, rather than the brittle mechanism of a hard-coded column number.

The TrialHandler also makes it easy to save associated meta data with your file (e.g. the subject and other info you might enter into a dialog box at the start of an experimental session). Check out the TrialHandler and ExperimentHandler demos from the demos menu of PsychoPy’s Coder view, and the API below.

In essence, you might find life a little easier if you make more use of PsychoPy’s built-in classes and migrate away from this pandas-esque sort of code. It would be worth looking at some of the provided demos and seeing how they are written.

1 Like

Many thanks for all this Michael! This is helpful. I’ll let you know if Trial Handler solves the problem.