Imprecise timing of visual objects overlaid on video

yfonken · March 27, 2018, 3:41pm

OSWin7:
PsychoPy version:1.84.6
**Standard Standalone? y
What are you trying to achieve?:
I’m trying to achieve very precise timing (+/- one frame max) of a visual stimulus overlaid on a video appearing at a precise delay relative to the onset of the video (i.e. 750ms of 1750ms into a 3sec video).

What did you try to make it work?:

I encoded timing in frames
I decreased the resolution of the video’s
I matched the video fps such that the screen refresh rate is a multiple (i.e. 30fps for 60Hz, 25fps for 100Hz)
I preload all stimuli in a 2s ISI
I tested it on my personal mac and on the win7 experimental computer. The timing seems to visually work well on my mac as long as I use the low-resolution videos that are encoded at 30fps (with 60Hz screen refresh). I encoded white squares in the video’s themselves so I can measure the difference between the video and the psychopy generated visual stimulus with a photodiode, this is also how I can tell the ‘rough’ timing of the overlaid visual stimulus.

What specifically went wrong when you tried that?:
THe delayed stimulus arrives >200ms (jittered) later than programmed.

From photodiode measurements the visual object appears at delays between 860 and 960ms instead of 750ms, and between 2099 and 2250ms instead of 1750ms (N=10 for each delay). This is the same as the delays that can be extracted from the psychopy logfiles.

A photodiode square object in psychopy coded to start with the beginning of the video does seem to align with the beginning of the video, so it seems as if the timing imprecision of additional objects gets introduced as the video is playing.

On the windows machine I do get the error: ‘avbin.dll failed to load. Try importing psychopy.visual’, though browsing through the forum I’ve found that this should not be a problem as long as I play video’s with moviepy.

I tested the experiment with one static image instead of the video, to see if the problem was video specific (or to see if I encoded the frame timing wrong), but the timing of the delayed visual stimulus is accurate +/- 20ms on the windows computer (timing here is based on the logfile). The >200ms inaccuracies in timing of the delayed visual stimulus in the video cannot be accounted for.

Is there something w.r.t. programming of timing with video’s I’m overlooking?

Thanks

jon · March 27, 2018, 4:38pm

There are many ways to get this wrong, like trying to load the image at the same time as you present it (it will then take time to load and be delayed in its onset). Impossible to tell what’s gone wrong without seeing the relevant code. Please consider what would be required for a “minimal working example” of the problem

yfonken · March 28, 2018, 2:18pm

Hi Jon,

Thank you for your reply.

I’ve programmed it such that both the image and the video are set during a 2s ISI before the trial. I do this in the pre-trial ISI instead of at the beginning of the experiment because I’m reading in movie and png filenames from a conditions xlsx file for each trial.

Per your suggestion I coded up a simpler version of the experiment, using only one video. Now it (visually) seems to have the correct timing. I will test with a photodiode next.

Is there a difference between initiating a video object (here called ‘movie’) directly referencing the filename (see simple version below) compared to referencing a variable pulled from a condition file (see snippet from original version)? Or, are there other ways to preload video’s than using movie.setmovie during the static period?

Simple version of my code that seems to work:

movie = visual.MovieStim3(
win=win, name=‘movie’,
noAudio = True,
filename=u’Clips/FieldLate30_lowres.mov’,
ori=0, pos=(0, 0), opacity=1,
size=[960,540],
depth=-3.0,
)
# keep track of which components have finished
trialComponents = [text, ISI, movie, image]
for thisComponent in trialComponents:
if hasattr(thisComponent, ‘status’):
thisComponent.status = NOT_STARTED

# -------Start Routine "trial"-------
while continueRoutine:
    # get current time
    t = trialClock.getTime()
    frameN = frameN + 1  # number of completed frames (so 0 is the first frame)
    # update/draw components on each frame
    
    
    # *text* updates
    if t >= 0.0 and text.status == NOT_STARTED:
        # keep track of start time/frame for later
        text.tStart = t
        text.frameNStart = frameN  # exact frame index
        text.setAutoDraw(True)
    
    # *movie* updates
    if frameN >= MovStartFrame and movie.status == NOT_STARTED:
        # keep track of start time/frame for later
        movie.tStart = t
        movie.frameNStart = frameN  # exact frame index
        movie.setAutoDraw(True)
    if movie.status == FINISHED:  # force-end the routine
        continueRoutine = False
    
    # *image* updates
    if frameN >= StimStartFrame and image.status == NOT_STARTED:
        # keep track of start time/frame for later
        image.tStart = t
        image.frameNStart = frameN  # exact frame index
        image.setAutoDraw(True)
    if image.status == STARTED and frameN >= (image.frameNStart + StimDurFrame):
        image.setAutoDraw(False)
    # *ISI* period
    if t >= 0.0 and ISI.status == NOT_STARTED:
        # keep track of start time/frame for later
        ISI.tStart = t
        ISI.frameNStart = frameN  # exact frame index
        ISI.start(ISI_dur)
    elif ISI.status == STARTED:  # one frame should pass before updating params and completing
        # updating other components during *ISI*
        movie.setOpacity(1)
        movie.setMovie(u'Clips/FieldLate30_lowres.mov')
        movie.setPos((0, 0))
        movie.setOri(0)
        movie.setSize([960,540])
        image.setOpacity(1)
        image.setPos((200, 100))
        image.setOri(0)
        image.setImage(u'frog.png')
        image.setSize((100, 100))
        # component updates done
        ISI.complete()  # finish the static period
    
    # check if all components have finished
    if not continueRoutine:  # a component has requested a forced-end of Routine
        break
    continueRoutine = False  # will revert to True if at least one component still running
    for thisComponent in trialComponents:
        if hasattr(thisComponent, "status") and thisComponent.status != FINISHED:
            continueRoutine = True
            break  # at least one component has not yet finished
    
    # check for quit (the Esc key)
    if endExpNow or event.getKeys(keyList=["escape"]):
        core.quit()
    
    # refresh the screen
    if continueRoutine:  # don't flip if this routine is over or we'll get a blank screen
        win.flip()

# -------Ending Routine "trial"-------
for thisComponent in trialComponents:
    if hasattr(thisComponent, "setAutoDraw"):
        thisComponent.setAutoDraw(False)

Code snippet from the original version that has large timing errors:

# ------Prepare to start Routine "trial"-------
t = 0
trialClock.reset()  # clock
frameN = -1
continueRoutine = True
# update component parameters for each repeat
movie = visual.MovieStim3(
    win=win, name='movie',units='pix', 
    noAudio = True,
    filename=vidName, #from conditions file
    ori=0, pos=[0, 0], opacity=1,
    size=[960, 540],
    depth=-2.0,
    )
key_resp_4 = event.BuilderKeyResponse()
# In seconds
ISI_dur = 2. +  random.uniform(0,jitter)
#MovStart = ISI_dur
#StimStart = DelaySecs + MovStart

# Convert seconds to frames
ISI_dur_frames = int(ISI_dur*FrameRate2) # int will round down to nearest integer
MovStartFrame =  1 # starts counting after ISI ends!
Key_frames_dur = int(3*FrameRate2) # now set to full length of video
StimStartFrame = MovStartFrame + int(DelaySecs*FrameRate2)
StimDurFrame = int(StimDur*FrameRate2)
polyFramesDur = int(polyDur * FrameRate2)
FixationFrames = ISI_dur_frames + Key_frames_dur

print(ISI_dur_frames)





# keep track of which components have finished
trialComponents = [ISI, Fixation, movie,  Target]
for thisComponent in trialComponents:
    if hasattr(thisComponent, 'status'):
        thisComponent.status = NOT_STARTED

# -------Start Routine "trial"-------
while continueRoutine:
    # get current time
    t = trialClock.getTime()
    frameN = frameN + 1  # number of completed frames (so 0 is the first frame)
    # update/draw components on each frame
    
    # *Fixation* updates
    if frameN >= 0 and Fixation.status == NOT_STARTED:
        # keep track of start time/frame for later
        Fixation.tStart = t
        Fixation.frameNStart = frameN  # exact frame index
        Fixation.setAutoDraw(True)
    if Fixation.status == STARTED and frameN >= (Fixation.frameNStart + FixationFrames):
        Fixation.setAutoDraw(False)
    
    # *movie* updates
    if frameN >= MovStartFrame and movie.status == NOT_STARTED:
        # keep track of start time/frame for later
        movie.tStart = t
        movie.frameNStart = frameN  # exact frame index
        movie.setAutoDraw(True)
    if movie.status == FINISHED:  # force-end the routine
        continueRoutine = False
    
    
        
        # check for quit:
        if "escape" in theseKeys:
            endExpNow = True
        if len(theseKeys) > 0:  # at least one key was pressed
            key_resp_4.keys.extend(theseKeys)  # storing all keys
            key_resp_4.rt.append(key_resp_4.clock.getTime())
    
    
    # *Target* updates
    if frameN >= StimStartFrame and Target.status == NOT_STARTED:
        # keep track of start time/frame for later
        Target.tStart = t
        Target.frameNStart = frameN  # exact frame index
        Target.setAutoDraw(True)
    if Target.status == STARTED and frameN >= (Target.frameNStart + StimDurFrame):
        Target.setAutoDraw(False)
    

    
    # *ISI* period
    if frameN >= 0 and ISI.status == NOT_STARTED:
        # keep track of start time/frame for later
        ISI.tStart = t
        ISI.frameNStart = frameN  # exact frame index
        ISI.start(ISI_dur_frames*frameDur)
    elif ISI.status == STARTED:  # one frame should pass before updating params and completing
        # updating other components during *ISI*
        movie.setOpacity(1)
        movie.setMovie(vidName)
        movie.setPos([0, 0])
        movie.setOri(0)
        movie.setSize([960, 540])
        Target.setOpacity(1)
        Target.setPos(TargetLocation)
        Target.setOri(0)
        Target.setImage(TargetStim)
        Target.setSize((60, 60))
       
        # component updates done
        ISI.complete()  # finish the static period

yfonken · March 28, 2018, 2:52pm

Update: I may have just pinpointed the problem to potential hardware.

We have a dual monitor setup (one experimenter, one subject). A quick test revealed that when the subject monitor is turned off, the timing seems ok on the experimenter monitor (not accounting for <50ms imprecision). When I turn on the subject monitor, the ~200ms delay appears. When I turn the experimenter monitor off, the delay is still there.

The experimenter monitor is connected by DVI, the subject monitor by display port. Is there a known issue with display port connections in dual monitor setups? The monitors themselves are identical (acer XB270H).

jon · March 29, 2018, 11:22am

That’s really weird. All I can imagine is that the graphics card is struggling to keep up (do you know what the card is?) such that it can load the movie frames and image fast enough when it’s only got only monitor to render to, but can’t keep up when it has to output to both.

Make sure that the monitors are both set to the same resolution in windows’ control panel and then use mirrored displays, to minimize that effort for the card

yfonken · March 29, 2018, 12:40pm

Hi Jon,

‘Make sure that the monitors are both set to the same resolution in windows’ control panel’

We have seemed to narrow down the problem to the resolution. The original setup had 100Hz refresh rate on the subject monitor, 75Hz on the experimenter monitor. When I set both to 60Hz the timing is spot on (with a 30fps video).

Thanks for your help

yfonken · June 7, 2018, 2:49pm

I have an update on the same setup, i.e. I am now experiencing more subtle timing problems after moving to a -supposedly- identical windows machine, now with psychopy 1.90.1

The goal: to present a video with an image overlaid at a specific time delay relative to the start of the video

The problem: The image is set to be presented at frame 46 (in this simple example case, see code below), relative to the video starting at frame 1. At 60Hz refresh rate this should happen at 766ms delay after the start of the video. However, from the logfile I find that the timing is off by a number of frames, and is inconsistent across trials (between 800 and 1100ms instead of 766ms). The video is an .mp4 H264 format/codec <3mb. I load all stimuli in a 2s ISI period.

The timing with the same code running on my mac is correct within one frame.

If anyone could point me towards better video codecs/formats to use, different settings or supporting python packages to try, or options of coding the timing different I would very much appreciate it! Thanks for reading.

The code:

for thisTrial in trials:
currentLoop = trials
# abbreviate parameter names if possible (e.g. rgb = thisTrial.rgb)
if thisTrial != None:
for paramName in thisTrial:
exec(’{} = thisTrial[paramName]’.format(paramName))

# ------Prepare to start Routine "trial"-------
t = 0
trialClock.reset()  # clock
frameN = -1
continueRoutine = True
# update component parameters for each repeat
movie = visual.MovieStim3(
    win=win, name='movie',
    noAudio = True,
    filename='Clips/Version1/BlueBoatsLR.mp4',
    ori=0, pos=(0, 0), opacity=1,
    size=[1280, 720],
    depth=-1.0,
    )
# keep track of which components have finished
trialComponents = [ISI, movie, image, text]
for thisComponent in trialComponents:
    if hasattr(thisComponent, 'status'):
        thisComponent.status = NOT_STARTED

# -------Start Routine "trial"-------
while continueRoutine:
    # get current time
    t = trialClock.getTime()
    frameN = frameN + 1  # number of completed frames (so 0 is the first frame)
    # update/draw components on each frame
    
    # *movie* updates
    if frameN >= 1 and movie.status == NOT_STARTED:
        # keep track of start time/frame for later
        movie.tStart = t
        movie.frameNStart = frameN  # exact frame index
        movie.setAutoDraw(True)
    if movie.status == FINISHED:  # force-end the routine
        continueRoutine = False
    
    # *image* updates
    if frameN >= 46 and image.status == NOT_STARTED:
        # keep track of start time/frame for later
        image.tStart = t
        image.frameNStart = frameN  # exact frame index
        image.setAutoDraw(True)
    if image.status == STARTED and frameN >= (image.frameNStart + 6):
        image.setAutoDraw(False)
    
    # *text* updates
    if frameN >= 0 and text.status == NOT_STARTED:
        # keep track of start time/frame for later
        text.tStart = t
        text.frameNStart = frameN  # exact frame index
        text.setAutoDraw(True)
    # *ISI* period
    if t >= 0.0 and ISI.status == NOT_STARTED:
        # keep track of start time/frame for later
        ISI.tStart = t
        ISI.frameNStart = frameN  # exact frame index
        ISI.start(2)
    elif ISI.status == STARTED:  # one frame should pass before updating params and completing
        # updating other components during *ISI*
        movie.setOpacity(1)
        movie.setMovie('Clips/Version1/BlueBoatsLR.mp4')
        movie.setPos((0, 0))
        movie.setOri(0)
        movie.setSize([1280, 720])
        image.setOpacity(1)
        image.setPos((-200, -200))
        image.setOri(0)
        image.setImage('FrogBlue.png')
        image.setSize((100, 100))
        # component updates done
        ISI.complete()  # finish the static period
    
    # check if all components have finished
    if not continueRoutine:  # a component has requested a forced-end of Routine
        break
    continueRoutine = False  # will revert to True if at least one component still running
    for thisComponent in trialComponents:
        if hasattr(thisComponent, "status") and thisComponent.status != FINISHED:
            continueRoutine = True
            break  # at least one component has not yet finished
    
    # check for quit (the Esc key)
    if endExpNow or event.getKeys(keyList=["escape"]):
        core.quit()
    
    # refresh the screen
    if continueRoutine:  # don't flip if this routine is over or we'll get a blank screen
        win.flip()

# -------Ending Routine "trial"-------
for thisComponent in trialComponents:
    if hasattr(thisComponent, "setAutoDraw"):
        thisComponent.setAutoDraw(False)
# the Routine "trial" was not non-slip safe, so reset the non-slip timer
routineTimer.reset()
thisExp.nextEntry()

Topic		Replies	Views
Inaccuracy in timing of visual stimuli Builder timing , screen	5	367	October 1, 2022
Video stimulus unusually slow Builder	9	3387	June 5, 2017
Stimulus Timing issue Coding	2	629	September 25, 2021
Frames vs timing - will my frame rate be automatically detected? Builder timing , screen	4	2667	June 9, 2023
Time is NOT accurate when it comes to image components Builder timing , screen	4	480	June 11, 2023

Imprecise timing of visual objects overlaid on video

Related topics