Timing & Movies

Builder
1

**MacOS 10.13.6
**PsychoPy version: 3.2.0
**Standard Standalone? (y)
Hi, everyone:

I have a question about timing and playing movies with moviepy (w/ ptb library & portaudio driver).

I need to know the timing of a custom response event while a movie is playing. A routine begins with a movie playing, and when the response occurs, I want to know the timing of the response relative to the start of the movie. (I can’t use any of the established response components like keyboard or button box because it is a custom response signal being read.)

Currently, I’m achieving this by grabbing the current clock time at the start of the routine in which the movie plays and then grabbing the current clock time when the response occurs, and then just subtracting the two. So I have a custom code component set to “Begin Routine” in the routine where the movie plays:

beginTime = clock.getTime()

Then, during the movie when the response occurs, I have the custom code component executed (this is triggered by the custom response signal):

shotTime = clock.getTime() - beginTime

shotTime is then saved as the time the response occurred relative to the beginning of the movie playing.

This works great. The problem is that the movies do not appear to begin immediately at the start of the routine but instead there may be a second blank screen pause before the movie starts. I’m concerned that, because of this delay, I’m not actually getting the response relative to the start of the MOVIE but instead relative to the start of the ROUTINE, which is (possibly) a second off.

My questions are, first, is it possible that movies are beginning to play slightly off from the start of the routine (even if the movie is supposed to begin at the start of the routine), and second, what is the best way to get a timing event relative to the ACTUAL start of the movie if the movies are not beginning at the exact start of the routine? Instead of grabbing the current clock time with beginTime = clock.getTime() at the start of the routine, is there a way to connect it to the flipping of the window to begin the movie or some other event that is certain to be the start of the movie playing?

Thank you for your help!
Joe

2

Yes, it takes a non-zero amount of time to load a movie file from disk, so it won’t start exactly at the specified time unless you have pre-loaded it. You could do this by inserting a “static period” component at some time in which nothing is changing in your task (e.g. during a fixation point stimulus) and then tell the movie to be updated during that named static period rather than the standard “every repeat”. That way it is cued up and ready to go at the desired onset time.

Try that in the first instance, and then your approach described above should work (within one screen refresh period).

1 Like
3

Wonderful, thank you for the suggestion Michael. I will try the static period. (There is a prior screen but it is much shorter than the movies to be loaded, so I’m not sure it will work though.)

If the static screen approach doesn’t solve the problem, could I use a code component to reference the first frame of the movie, something like using frameN as in:

if movie.frameNStart == 1:
     beginTime = clock.getTime()

or

if movie.status == STARTED:
     beginTime = clock.getTime()

or some other variable inherent to the movie component?

thanks! Joe

4

The static period doesn’t need to approach the duration of the movie, just the relatively short period of time required to initialise the movie, which is probably independent of the movie duration (and might be something of the order of some tens of milliseconds).

That sort of approach could work but beware of these particular expressions where you are checking against something that effectively becomes constant: the check will keep passing many times, meaning that your timer will get continually re-set.

So you need to set some sort of flag at the beginning of the routine, say, like:

timer_started = False

and then on every frame:

if movie.status == STARTED and not timer_started: 
    beginTime = clock.getTime()
    timer_started = True # so will only do this once
5

Wonderful, this looks like it will be perfect. Thank you so much! Joe

6

Hi, Michael:

Actually, is a simpler approach just to grab

 movie.tStart

where these were defined earlier as:

 movie.tStart = t

and

 t = routineClock.getTime()

That is, is it the case that t is defined when the routine begins as 0, and movie.tStart is the time (relative to the start of the routine then) that the movie component began? This seems simpler than repeatedly checking frame-by-frame whether movie.status == STARTED and not time_started, but I wanted to check that I’m understanding movie.tStart correctly.

Thank you again!

7

To be honest, I don’t know, without generating a Builder file and reading through the code it generates. But the start time it assigns should be correct, assuming you’ve pre-loaded the movie. Without checking, I couldn’t say if it would be correct if there is a lag due to needing to initialise the movie if it wasn’t pre-loaded.

8

Hi, Michael:

Of course, my apologies – should have pasted it right away. At the bottom of this is the code for the relevant routine. (Note that in everything that follows I’m not pre-loading the movies.)

On my reading, at the start of the routine t is set to 0 and testClock (the clock for the routine “test”) is reset. Then, the movie is updated with:

if t >= 0 and movie.status == NOT_STARTED:
        # keep track of start time/frame for later
        movie.tStart = t

movie.tStart would seem then to be the time the movie actually starts and not the time it’s called to start – otherwise t would just be zero because the movie is set to begin at the start of the routine. Is that not correct? When I store movie.tStart in the datafile, it is in fact a non-zero value: .744, indicating that it’s not just giving me the time it is supposed to play (which again is zero i.e. the start of the routine) but instead giving the actual time the movie begins. I’m interpreting that as the movie taking 3/4 of a second to load.

One weird thing is that when I use the code you suggested earlier:

        if movie.status == STARTED and timer_started == False: 
            movieStartTime = testClock.getTime()
            timer_started = True

movieStartTime gives a slightly different value of .766, which is .022 off from movie.tStart. Why it is that movieStartTime and movie.tStart give different values, I haven’t figured out yet.

Thank you again so much.

For the code below, the routine is called “test” (and is within a loop called “trials”). The movie component is called “movie” (drawn from CurrentFile which is a list established earlier in the experiment). There is a keyboard component called “key_resp_1” which records every time the letter z is pressed. Both the movie and the keyboard component are set to begin at the start of the routine. Routine ends when movie ends.

# ------Prepare to start Routine "test"-------
t = 0
testClock.reset()  # clock
frameN = -1
continueRoutine = True
# update component parameters for each repeat
key_resp_1 = event.BuilderKeyResponse()

movie = visual.MovieStim3(
    win=win, name='movie',units='norm', 
    noAudio = False,
    filename=CurrentFile,
    ori=0, pos=(0, 0), opacity=1,
    size=(2, 2),
    depth=-3.0,
    )
# keep track of which components have finished
testComponents = [key_resp_1, movie]
for thisComponent in testComponents:
    if hasattr(thisComponent, 'status'):
        thisComponent.status = NOT_STARTED

# -------Start Routine "test"-------
while continueRoutine:
    # get current time
    t = testClock.getTime()
    frameN = frameN + 1  # number of completed frames (so 0 is the first frame)        
    # *key_resp_1* updates
    if t >= 0.0 and key_resp_1.status == NOT_STARTED:
        # keep track of start time/frame for later
        key_resp_1.tStart = t
        key_resp_1.frameNStart = frameN  # exact frame index
        key_resp_1.status = STARTED
        # keyboard checking is just starting
        win.callOnFlip(key_resp_1.clock.reset)  # t=0 on next screen flip
        event.clearEvents(eventType='keyboard')
    if key_resp_1.status == STARTED:
        theseKeys = event.getKeys(keyList=['z'])
        
        # check for quit:
        if "escape" in theseKeys:
            endExpNow = True
        if len(theseKeys) > 0:  # at least one key was pressed
            key_resp_1.keys.extend(theseKeys)  # storing all keys
            key_resp_1.rt.append(key_resp_1.clock.getTime())

    # *movie* updates
    if t >= 0 and movie.status == NOT_STARTED:
        # keep track of start time/frame for later
        movie.tStart = t
        movie.frameNStart = frameN  # exact frame index
        movie.setAutoDraw(True)
    if movie.status == FINISHED:  # force-end the routine
        continueRoutine = False
    
    # check if all components have finished
... [deleted after here]....
9

We can’t tell that without seeing code that includes both the movieStartTime and movie.tStart sections.

10

Hi, Michael:

No problem – here it is with a code component (from builder) set to define “timer_started = False” at the start of the routine and the checking of whether movie.status has started at each frame:

    # ------Prepare to start Routine "test"-------
    t = 0
    testClock.reset()  # clock
    frameN = -1
    continueRoutine = True
    # update component parameters for each repeat
    timer_started = False
    key_resp_1 = event.BuilderKeyResponse()
    movie = visual.MovieStim3(
        win=win, name='movie',units='norm', 
        noAudio = False,
        filename=CurrentFile,
        ori=0, pos=(0, 0), opacity=1,
        size=(2, 2),
        depth=-2.0,
        )
    # keep track of which components have finished
    testComponents = [key_resp_1, movie]
    for thisComponent in testComponents:
        if hasattr(thisComponent, 'status'):
            thisComponent.status = NOT_STARTED
    
    # -------Start Routine "test"-------
    while continueRoutine:
        # get current time
        t = testClock.getTime()
        frameN = frameN + 1  # number of completed frames (so 0 is the first frame)
        # update/draw components on each frame
        if movie.status == STARTED and timer_started == False: 
            movieStartTime = testClock.getTime()
            timer_started = True 
        
        # *key_resp_1* updates
        if t >= 0.0 and key_resp_1.status == NOT_STARTED:
            # keep track of start time/frame for later
            key_resp_1.tStart = t
            key_resp_1.frameNStart = frameN  # exact frame index
            key_resp_1.status = STARTED
            # keyboard checking is just starting
            win.callOnFlip(key_resp_1.clock.reset)  # t=0 on next screen flip
            event.clearEvents(eventType='keyboard')
        if key_resp_1.status == STARTED:
            theseKeys = event.getKeys(keyList=['z'])
            
            # check for quit:
            if "escape" in theseKeys:
                endExpNow = True
            if len(theseKeys) > 0:  # at least one key was pressed
                key_resp_1.keys.extend(theseKeys)  # storing all keys
                key_resp_1.rt.append(key_resp_1.clock.getTime())
        
        # *movie* updates
        if t >= 0 and movie.status == NOT_STARTED:
            # keep track of start time/frame for later
            movie.tStart = t
            movie.frameNStart = frameN  # exact frame index
            movie.setAutoDraw(True)
        if movie.status == FINISHED:  # force-end the routine
            continueRoutine = False
        
        # check if all components have finished
...[deleted after here]...
11

OK, this timing discrepancy is due to the relative position of your code component and the movie component. Because you have your code component above the movie component, that custom code gets executed before the movie code. So in this case, the start times will be set on different screen refreshes. i.e. on the screen refresh cycle where the movie will be started, your custom code runs fractionally before the movie actually starts, so movie.status == STARTED will evaluate to False. It will only be on the next cycle that it will evaluate to True. You can remove this problem by shifting the code component to be below the movie component. That way movie.status == STARTED will evaluate to True on the same refresh cycle in which the movie actually starts, because it is checked after the movie code has run.

But this custom code is probably redundant anyway, as you can just access movie.tStart directly.

12

Ah wonderful! Thanks for making that clear. And I see you’re right that movie.tStart can be called directly.

One odd thing about the behavior of the keyboard component, however, is that given the code below the timer for the keyboard component actually appears to be starting NOT at the start of the routine (t=0) but instead at the same time as when the movie begins, even though the keyboard component is set to begin at the start of the routine.

For example, if we store both movie.tStart and key_resp_1.tStart, they are exactly equal and neither one is 0 (generally, they are between .5 and .75 sec). Shouldn’t the key_resp_1 begin at routine = 0 regardless of when the movie loads & plays?

Thanks,

Joe


    # ------Prepare to start Routine "test"-------
    t = 0
    testClock.reset()  # clock
    frameN = -1
    continueRoutine = True
    # update component parameters for each repeat
    key_resp_1 = event.BuilderKeyResponse()
    
    movie = visual.MovieStim3(
        win=win, name='movie',units='norm', 
        noAudio = False,
        filename=CurrentFile,
        ori=0, pos=(0, 0), opacity=1,
        size=(2, 2),
        depth=-2.0,
        )
    
    # keep track of which components have finished
    testComponents = [key_resp_1, movie]
    for thisComponent in testComponents:
        if hasattr(thisComponent, 'status'):
            thisComponent.status = NOT_STARTED
    
    # -------Start Routine "test"-------
    while continueRoutine:
        # get current time
        t = testClock.getTime()
        frameN = frameN + 1  # number of completed frames (so 0 is the first frame)
        # update/draw components on each frame
        
        # *key_resp_1* updates
        if t >= 0.0 and key_resp_1.status == NOT_STARTED:
            # keep track of start time/frame for later
            key_resp_1.tStart = t
            key_resp_1.frameNStart = frameN  # exact frame index
            key_resp_1.status = STARTED
            # keyboard checking is just starting
            win.callOnFlip(key_resp_1.clock.reset)  # t=0 on next screen flip
            event.clearEvents(eventType='keyboard')
        if key_resp_1.status == STARTED:
            theseKeys = event.getKeys(keyList=['z'])
            
            # check for quit:
            if "escape" in theseKeys:
                endExpNow = True
            if len(theseKeys) > 0:  # at least one key was pressed
                key_resp_1.keys.extend(theseKeys)  # storing all keys
                key_resp_1.rt.append(key_resp_1.clock.getTime())
        if len(theseKeys) > 0:
            shotTimeMovieAnchor = testClock.getTime() - movie.tStart
        
        # *movie* updates
        if t >= 0 and movie.status == NOT_STARTED:
            # keep track of start time/frame for later
            movie.tStart = t
            movie.frameNStart = frameN  # exact frame index
            movie.setAutoDraw(True)
        if movie.status == FINISHED:  # force-end the routine
            continueRoutine = False
        
        
        # check if all components have finished
        if not continueRoutine:  # a component has requested a forced-end of Routine
            break
        continueRoutine = False  # will revert to True if at least one component still running
        for thisComponent in testComponents:
            if hasattr(thisComponent, "status") and thisComponent.status != FINISHED:
                continueRoutine = True
                break  # at least one component has not yet finished
        
        # check for quit (the Esc key)
        if endExpNow or event.getKeys(keyList=["escape"]):
            core.quit()
        
        # refresh the screen
        if continueRoutine:  # don't flip if this routine is over or we'll get a blank screen
            win.flip()
    
    # -------Ending Routine "test"-------
[deleted after this point]
13

No, because the test clock is being reset to zero before the movie object is created, which takes some time to initialise. So by this stage:

# -------Start Routine "test"-------
    while continueRoutine:
        # get current time
        t = testClock.getTime()

the clock has been running for the duration required to initialise the movie.

As noted previously, you will necessarily get a lag like this unless you have set the movie to be preloaded, say in a static period before the beginning of the routine, so that it is ready to start playing immediately.

14

Michael, thank you so so much for your help with this. Very helpful and the timing specifics are now very clear.

I see that PsychoPy3 / MovieStim3 seems to handle timing differently. If I’m reading it correctly (below), the line:

win.timeOnFlip(movie, 'tStartRefresh')

assigns the time ON the next screen flip to the object ‘movie’, which means that the time “movie.tStartRefresh” (which is saved as “movie.started” in the datafile) is in fact the first frame that the movie plays. The movie has already been created/initialized by this point in the code so delays in loading the movie should not affect this value. Is this correct?

# ------Prepare to start Routine "movieRoutine"-------
# update component parameters for each repeat
# keep track of which components have finished
movieRoutineComponents = [movie]
for thisComponent in movieRoutineComponents:
    thisComponent.tStart = None
    thisComponent.tStop = None
    thisComponent.tStartRefresh = None
    thisComponent.tStopRefresh = None
    if hasattr(thisComponent, 'status'):
        thisComponent.status = NOT_STARTED
# reset timers
t = 0
_timeToFirstFrame = win.getFutureFlipTime(clock="now")
movieRoutineClock.reset(-_timeToFirstFrame)  # t0 is time of first possible flip
frameN = -1
continueRoutine = True

# -------Run Routine "movieRoutine"-------
while continueRoutine:
    # get current time
    t = movieRoutineClock.getTime()
    tThisFlip = win.getFutureFlipTime(clock=movieRoutineClock)
    tThisFlipGlobal = win.getFutureFlipTime(clock=None)
    frameN = frameN + 1  # number of completed frames (so 0 is the first frame)
    # update/draw components on each frame
    
    # *movie* updates
    if movie.status == NOT_STARTED and tThisFlip >= 0.0-frameTolerance:
        # keep track of start time/frame for later
        movie.frameNStart = frameN  # exact frame index
        movie.tStart = t  # local t and not account for scr refresh
        movie.tStartRefresh = tThisFlipGlobal  # on global time
        win.timeOnFlip(movie, 'tStartRefresh')  # time at next scr refresh
        movie.setAutoDraw(True)
    
    # check for quit (typically the Esc key)
    if endExpNow or defaultKeyboard.getKeys(keyList=["escape"]):
        core.quit()
    
    # check if all components have finished
    if not continueRoutine:  # a component has requested a forced-end of Routine
        break
    continueRoutine = False  # will revert to True if at least one component still running
    for thisComponent in movieRoutineComponents:
        if hasattr(thisComponent, "status") and thisComponent.status != FINISHED:
            continueRoutine = True
            break  # at least one component has not yet finished
    
    # refresh the screen
    if continueRoutine:  # don't flip if this routine is over or we'll get a blank screen
        win.flip()

# -------Ending Routine "movieRoutine"-------
[deleted after here]