Running independent loops simultaneously with sound and visual stimuli

Coding
1

I am trying to set up a dual task which plays participants a constant stream of tones every 3 seconds and records responses to them via mouse clicks, while at the same time they are viewing stimuli on screen and responding to them via key presses. I currently have this set-up running from 2 separate machines which is extremely clunky and allows a lot of room for experimenter and technical error so I am trying to integrate the scripts into 1, but am having no luck at all in getting the 2 processes to run simultaneously.

The audio task needs to run for as long as it takes participants to complete a set number of trials of the visual task, and the length of time taken to complete the visual task is variable dependent on participant reaction time. The 2 tasks should also be entirely independent of each other, apart from starting and finishing at the same time.

Essentially what I need is:

While (trial number not met):
Play sound + record responses
While sound is playing AND for set no. trials:
Generate visual stims + record responses

I have tried various combinations of while and for loops to make this work, but as I can’t get the 2 loops to run simultaneously I haven’t been able to get any closer than getting the program to alternate between playing 1 tone and doing 1 trial of the visual task.

I have also tried to set it up using multiprocessing (just using basic code to practise with as the full experiment code is very long and convoluted) but am getting a pickling error when I try to incorporate sound. See code below. The same code works fine when both processes are just printing things.

I am too much of a novice to even begin to figure out how to fix the pickling issue on my own! Can anybody help me figure out if it’s possible to use multiprocessing with .Sound (and .Visual, as I will need both!) or if there is a clever way to use a combination of loops to mean it isn’t necessary?

Thank you!

Practise code:

from multiprocessing import Process
from psychopy import sound

one_tone = sound.Sound(value='B',secs=0.2, octave=4, loops=0)
   
   
def sounds(tone):
    for i in range (5):
        tone.play()
        core.wait(0.5)
        
        
def counting(total):
    for i in range (total):
        print i
        core.wait(0.3)

if __name__ == '__main__':
    p1 = Process(target=sounds, args=(one_tone,))
    p2 = Process(target=counting, args=(20,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

error output

File “C:\Users\jud1\Box Sync\Adult WSLS experiments\Experiment 2 - multiple dual task pilots\Audio\multiprocess test.py”, line 26, in
p1.start()
File “C:\Program Files\PsychoPy2\lib\multiprocessing\process.py”, line 130, in start
self._popen = Popen(self)
File “C:\Program Files\PsychoPy2\lib\multiprocessing\forking.py”, line 277, in init
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File “C:\Program Files\PsychoPy2\lib\multiprocessing\forking.py”, line 199, in dump
ForkingPickler(file, protocol).dump(obj)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 224, in dump
self.save(obj)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 331, in save
self.save_reduce(obj=obj, *rv)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 425, in save_reduce
save(state)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 286, in save
f(self, obj) # Call unbound method with explicit self
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 655, in save_dict
self._batch_setitems(obj.iteritems())
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 687, in _batch_setitems
save(v)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 286, in save
f(self, obj) # Call unbound method with explicit self
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 554, in save_tuple
save(element)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 331, in save
self.save_reduce(obj=obj, *rv)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 425, in save_reduce
save(state)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 286, in save
f(self, obj) # Call unbound method with explicit self
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 655, in save_dict
self._batch_setitems(obj.iteritems())
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 687, in _batch_setitems
save(v)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 331, in save
self.save_reduce(obj=obj, *rv)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 396, in save_reduce
save(cls)
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 286, in save
f(self, obj) # Call unbound method with explicit self
File “C:\Program Files\PsychoPy2\lib\pickle.py”, line 754, in save_global
(obj, module, name))
pickle.PicklingError: Can’t pickle <type ‘Sound’>: it’s not found as builtin.Sound

2

I’m (slightly) more familiar with using threading instead of multiprocess, I only know that the former use the same space in memory and hear that the latter can make communication between objects more difficult. Have you tried using threading instead?

But before getting ahead of ourselves, it would be helpful to include the complete error message, and explain where this pickling error is coming from, from what’s here we can’t tell if it’s a symptom of the same problem or a different one.

Lastly, I see you tried to have the code formatted, but you need to use backticks ``` instead of apostrophes. If you could edit your post to fix that it would make it more legible.

3

I think it should be able to work within a single loop. Would an arrangement like below work? It requires looping over ‘time’ rather than over trials.

clock = psychopy.core.Clock()

sound_playing = False
sound_duration = 0.2
sound_iti = 3.0

current_trial = 1

trials_complete = False

while not trials_complete:

    # start playing if it is within the initial period, and isn't already playing
    if np.mod(clock.getTime(), sound_iti) < sound_duration:
        if (not sound_playing):
            tone.play()
            sound_playing = True
    else:
        sound_playing = False

    # handle stimulus etc. for current trial

    # check and handle keyboard and mouse

An alternative would be to generate a 3 second long sound (0.2 seconds of beep and 2.8 seconds of silence, either as an array or as a wav file) and then play it at the start with loops=-1, which will make it play until you tell it to stop (at the end of the trials).

1 Like
4

I have edited the post to include the full error and fixed the backticks.

I have never used threading. I have never previously used multiprocessing either, I just had it suggested to me by a colleague (who is also not very familiar with it) as something to test as we hadn’t managed to get anything working with loops. So definitely open to trying with threading instead, but would have no idea where to start!

Thank you

5

Well @djmannion is suggesting you may be able to avoid threading altogether. Have you tried his suggestion?

6

I managed to get the code you provided to play a series of tones earlier (using a windows 7 desktop PC), but now testing it again on a different machine (windows 7 laptop) I am having trouble getting anything to happen at all. I think the issue is with the following line:

if np.mod(clock.getTime(), sound_iti) < sound_duration:

as this number is never lower than sound_duration and so it never gets to the point where it plays a tone.

Even when it was working earlier however, I don’t think this is a solution to my problem unfortunately, as anything that happens within the while loop (such as the example below), after the tone plays, needs to run to duration before the loop will restart and play another tone. This means it is still alternating between playing one tone per trial, rather than a constant stream of tones while the trials progress.

    sound_duration = 0.2
    sound_iti = 3.0
    current_trial = 1
    sound_playing=False
    trials_complete = False
    
    while not trials_complete:

        # start playing if it is within the initial period, and isn't already playing
        if np.mod(clock.getTime(), sound_iti) < sound_duration:
            if (not sound_playing):
                one_tone.play
                sound_playing = True
        else:
            sound_playing = False
        stim=random.choice(stimsList)
        stim.draw()
        win.flip()
        core.wait(4)
        if current_trial==trials:
            trials_complete=True
        current_trial+=1

playing a constant stream of tones would potentially work within this dummy example, but in the full experiment it will be randomly selecting the sound to play from a list of available sounds so unfortunately I don’t think that would work either.

7

I guess it’s not that clear (to me) what you’re trying to achieve. I know it’s a delicate balance to be brief enough that people will respond, yet with enough of the right information so that we can give useful answers, but could you be a little clearer about what is actually supposed to happen in a given trial?

EDIT

Looking at your code again, I think one big problem might be not understanding what core.wait() does. It freezes that thread for that period of time, meaning any mouse clicks given in that time period will not be registered, for example. If you are interested in visual reaction times, its use is also probably going to mess up those timings. Have you had a good look at the psychopy docs about millisecond precision timing, and monitor refresh rates? Remember that screens don’t refresh every millisecond, so when dealing with reaction times you need to compare with the frame flips, and whenever you call core.wait() on your main thread, you may be causing trouble for calculating reaction times.

It seems like you might not need to deal with threads yet, but an understanding of them may help you figure out why something is not working the way you intended. This tutorial might get you started: http://www.python-course.eu/threads.php

But still, first things first, I think the best way forward right now is to describe the goal well, so that you may avoid unnecessary complications like threading or multiprocessing.

8

Your code is missing the parentheses required to get the sound to play - i.e. you have one_tone.play rather than one_tone.play().

That should be OK - the key thing is that the loop iterates continuously, rather than using the core.wait functionality to interrupt execution. This requires the code to be written in a bit of a different way, but should be possible.

For example, say you wanted your tones to be playing every three seconds while you are collecting the reaction time for an orientation detection task (stimuli presented in random order) and also collecting mouse clicks for a secondary task (I agree with @daniel.riggs1 that a more precise specification of your requirements would be helpful). Something like the below should let you do that:

import random

import numpy as np

import psychopy.visual
import psychopy.sound
import psychopy.event
import psychopy.core

win = psychopy.visual.Window(size=[400, 400], units="pix", fullscr=False)

sound_playing = False
sound_duration = 0.2
sound_iti = 3.0

tone = psychopy.sound.Sound(secs=sound_duration)

stim_oris = [0, 45, 90, 135]

n_stim = len(stim_oris)

stims = [
    psychopy.visual.GratingStim(
        win=win,
        size=[200, 200],
        sf=5.0 / 200.0,
        ori=stim_ori
    )
    for stim_ori in stim_oris
]

random.shuffle(stims)

stim_drawn = [False] * n_stim

mouse = psychopy.event.Mouse()

clock = psychopy.core.Clock()
trial_clock = psychopy.core.Clock()
   
i_trial = 0

while i_trial < n_stim:

    # start playing if it is within the initial period, and isn't already playing
    if np.mod(clock.getTime(), sound_iti) < sound_duration:
        if (not sound_playing):
            tone.play()
            sound_playing = True
    else:
        sound_playing = False

    # handle stimulus etc. for current trial
    if not stim_drawn[i_trial]:
        stims[i_trial].draw()
        win.flip()
        trial_clock.reset()
        stim_drawn[i_trial] = True

    # check and handle keyboard and mouse  
    keys = psychopy.event.getKeys(timeStamped=trial_clock)

    if keys:        
        for (key, key_rt) in keys:
            print "Pressed {k:s} with RT {rt:.3f}".format(k=key, rt=key_rt)
        
        i_trial += 1

    if any(mouse.getPressed()):
        print "Mouse click registered"

win.close()
9

Thanks for the help! I will try to be more specific without being too specific. I will also have a go with the above solution and hopefully that will crack it.

There need to be 2 tasks running at the same time, which are totally independent of each other, apart from starting and finishing at the same time.

task 1: iterates over a set number of trials. On each trial 2 shapes are presented on screen, one on the left and one on the right. After a variable wait time either a target stimulus or a foil stimulus is presented on top of one of the shapes and a participant needs to select either left or right to select the side of the screen that should contain the target using specified keys on either a keyboard or button box. Psychopy needs to record whether a target or foil stim appeared, which side it appeared on, which key was pressed and the reaction time, and whether the correct (target not foil) side was selected. There isn’t a set time limit for the set of trials or for each individual trial, as it will depend on the variable wait time before the stim is presented and the reaction time of the participant on each trial. There are brief wait screens between trials just telling participants to get ready / their score for each trial etc.

task 2: loops continuously from the onset of task 1, until task 1 has ended. Every 3 seconds plays an audio stimulus and waits for a response from the mouse. Psychopy needs to record what audio stim was played, whether the mouse was clicked in response to it and if so how many times, and what the ‘correct’ response to the audio stim should have been, dependent on the participant’s task condition, to give a score. Task 2 needs to play the audio stimuli continuously at the same intervals regardless of what point in a trial task 1 is at. Nothing ever needs to be presented visually.

Currently when running this each task plays off a different machine (task 2 playing off a tablet with headphones and a mouse plugged in) and task 2 is set to start and stop with a keypress and I watch the participant complete task 1 and use a wireless keyboard to start and stop task 2 at the appropriate time. This is fine up to a point, but obviously far from ideal as it is a bit off-putting for participants and leaves the window for experimenter error very wide open!

10

This seems to do pretty much what I need, thank you so much!
The only issue I am having is that when it registers a mouseclick it slows the whole program down and pauses before playing the next tone, sometimes to the point that it freezes everything and gives a ‘not responding’ message for a couple of seconds. That might just be to do with the speed of my machine though. I should be able to test on a different computer on Monday.

It prints the “Mouse click registered” repeatedly until something else happens, so this must be registering the fact that the mouse was clicked in every loop before the next tone is played, rather than registering a click once and starting the next loop with a blank slate. This is happening even with a click reset added at the beginning of the loop. If I could change this I’m sure that would sort out the speed issue, I will see if I can do this and let you know.

I would also need to amend it so it could register whether a click was single or double, and so it associated the tone and the click (as in, have a record of which clicks were made in response to which tones) so it could mark the click responses as correct or incorrect.

However, now that I can see the general conceptual way to get the 2 loops to play simultaneously without waiting for each other to finish, as long as I can stop the delay caused by the clicking, fitting everything else in should (hopefully) not be too difficult!

11

Great! Thanks for the detailed description - I think it should all work OK.

I suspect the slowness issue is caused by lots and lots of print statements, because it was printing a new statement every time it registered that the mouse button was in a down state (not just the action of pressing it). You might be able to resolve both it and the double-click issue by putting some code like this outside of the trial loop:

m_known_down = any(mouse.getPressed())

last_click = -1.0
double_click_range = 0.3

and then replace the mouse handling code with something like:

    if any(mouse.getPressed()):
        if not m_known_down:
            m_known_down = True
            m_click_time = clock.getTime()
            if (m_click_time - last_click) < double_click_range:
                print "Double click registered"
            else:
                print "Mouse click registered"
            last_click = m_click_time
    else:
        m_known_down = False
12

Great, this works fine thank you!

13

Hello! I have a similar problem to this and I’m super confused how the solution djmannion offers will allow the two tasks to be time-independent. The main difficulty I’m having is that if the beep starts playing when a new trial loop starts, and then it gets down to the part of the code where the visual stim is presented and it’s waiting for the key press, and the person takes a long time to key press (let’s pretend 8seconds), won’t the audio beep still be playing beyond its 200ms presentation time because the loop hasn’t circled back for the next trial to re-evaluate whether the tone should be playing or not? If someone could help me understand this, I would greatly appreciate it. I did try something similar in psychopy coder and it is doing what I thought it would do–going through the visual stim-RT before cycling back up to the top of the loop, so yeah, I’m confused how to implement this so it works. Thanks!

14

hi, I haven’t used this code for a very very long time so I can’t remember how all of it works and ufortunately I don’t have capacity at the moment to review it, but I know that I definitely had people do the task that had ludicrously long response times for their key presses (I think one person waited several full minutes) and it didn’t affect the beeps.

If it is at all helpful, all the code I used when using this method to run my PhD studies is online here: OSF | Flexible Social Learning Strategies are Harder than the Sum of their Parts (experiment scripts) and I think the relevant script that used this method of playing tones while waiting for key responses to a different task is WSLS_switchingTones_exp1.py

Feel free to extract useful snippets from that script if it can solve your problem!

15

Thanks Juliet!