psychopy.org | Reference | Downloads | Github

Moviestim3: timing issues with frame-by-frame presentation

timing

#1

hello alltogether,

I played around and did quite some testing, but couldn’t really find a solution for the following puzzle. maybe someone here can clearify to me what’s happening. if not, maybe it helps others, who run into similar problems.

problem:

for an eeg experiment I want to present video stimuli with decent and reliable time-precision. I use moviestim3 and a set of mp4 video clips, all encoded using the same codec specifications.

I first decided for a frame-refresh based presentation loop, since this should in general give you the most control over your stimuli.
the output looks good (not sped up or similar). yet, neither on my coding machine (mac os, ~60hz), nor on my lab pcs (windows, 144hz lcd) I could get this to work with reliable accuracy.

eg. presenting 10s of the clips on my 144hz panel (ie. for 10x144 = 1440 frames) the actual playback duration differs from the defined (10s) in a range between 50 to 500ms! scaling with presentation duration.
using a countdown-timer + while loop, I also have some milliseconds inaccuracies, but these don’t scale up with longer presentation durations, and are much more reproduceable.

note that I check the timing very rudimentally using the system clock and log-file

in both conditions there are barely frames dropped, and most interestingly the frame counter for the while loop usually results in identical frame counts (eg. 1440 frames for 10s).

I really dont understand what’s happening, and in general what moviestim does is not very transparent to me. is it possible that the movie-objects hold some additional information on frame-timing that differ from the displays refresh-rates and can hence not be reproduced presenting them frame by frame?

I am curious to here your comments on that!

below some simplified code:

# init
import os
import logging
import numpy as np
from psychopy import visual, core, event, monitors, gui, parallel 


# parameter
nStim = 2
time_pres = 10  # in s
framerate = 144  # in hz
trig_start = 255  # trigger id
trig_end = 250  # trigger id

dir_stim = 'path/to/stimuli'
stimlist = ['ABC.mp4', 'XYZ.mp4']


# init logfile
logging.basicConfig(filename='logfile.log',
                format='%(asctime)s - %(levelname)s: %(message)s', level=logging.DEBUG)
console = logging.StreamHandler()  # log also to console
logging.getLogger('').addHandler(console)

# init window
screen = monitors.Monitor('testMonitor', width=53, distance=60)
win = visual.Window(size=[1920, 1080]],
                screen=0, winType='pyglet',
                allowGUI=False,  # show mouse
                fullscr=True,
                monitor=screen
                color='grey',
                colorSpace='rgb',
                units='pix',
                useFBO=True
                )
                
# init trigger function
def send_trigger(triggerID):
    logging.debug('trigger: %i' % triggerID)
    
# load movies
video_paths = []
movies = []
for iStim, stim in enumerate(stimlist):
    video_paths.append(os.path.join(dir_stim, stim))
    movies.append(
        visual.MovieStim3(win, video_paths[iStim],
                          size=(1280, 720),
                          pos=[0, 0], flipVert=False,
                          flipHoriz=False, loop=False,
                          noAudio=True)
    )

presentation loop (frame-by-frame)

frames_pres = time_pres * framerate

for iStim in range(nStim):
    # let's draw a dynamic stimulus for exactly n frames
    for iFrame in range(frames_pres):
        movies[iStim].draw()
        # flip buffer
        win.flip()
        # send trigger after first flip and start recording flip-duration
        if iFrame == 0:
            win.recordFrameIntervals = True  # check for framedrops
            send_trigger(trig_start)
    # send end-of-presentation trigger
    win.flip()
    # send trigger
    send_trigger(trig_end)
    # stop frame-flip recording
    win.recordFrameIntervals = False
    # log info about presentation
    logging.info('STIMULUS: %i' % iStim)
    logging.info('FRAMES DROPPED: %i' % win.nDroppedFrames)
    logging.info('MEAN FRAMEINTERVAL: %f s' % np.mean(win.frameIntervals))
    logging.info('EFF. FRAMERATE: %f Hz' % (1./np.mean(win.frameIntervals)))
    logging.info('PRESENTED FRAMES: %i' % len(win.frameIntervals))

alternative presentation loop

for iStim in range(nStim):
    timer_dyn = core.CountdownTimer(time_pres)
    trig = True
    while timer_dyn.getTime() > 0:
        # draw movie
        movies[iStim].draw()
        # flip
        win.flip()
        # send trigger at first flip and start recording flip-duration
        if trig:
            win.recordFrameIntervals = True  # check for framedrops
            send_trigger(trig_start)
            trig = False
    # send end-of-presentation trigger
    win.flip()
    send_trigger(trig_end)
    # stop frame-flip recording
    win.recordFrameIntervals = False
    # log info about presentation
    logging.info('STIMULUS: %i' % iStim)
    logging.info('FRAMES DROPPED: %i' % win.nDroppedFrames)
    logging.info('MEAN FRAMEINTERVAL: %f s' % np.mean(win.frameIntervals))
    logging.info('EFF. FRAMERATE: %f Hz' % (1./np.mean(win.frameIntervals)))
    logging.info('PRESENTED FRAMES: %i' % len(win.frameIntervals))

#2

First thing to know is that MovieStim3 is basically an implementation of MoviePy, https://zulko.github.io/moviepy/ . MoviePy is far from perfect and is also kind of opaque, but you may find it helpful to dig around its documentation.

I’m actually very interested in the behavior you’ve found because I recently had an issue where loading, as separate objects, a 30 fps movie and a 60 fps movie led to the 60fps movie playing at 30fps and taking twice as long. This may be related. When I have time I’ll do some poking around and see what I can figure out.

Also, have you tried different encodings? I’ve noticed a lot of variation by codec in how MovieStim behaves.


#3

thanks for the hint on moviepy and on trying different codecs…

I observed two other interesting things:

  1. if i present a moviestim-object repetedly using my code, it continues playback (as expected), yet with increased speed (roughly double?). the presentation framerate (as calculated from win.frameIntervals) is not affected.

  2. I didnt check whether this is psychopy in general, or just moviestim: however, on my presentation machines moviestim does not take into account which refresh-rate is set via the graphic card driver. it displays using the highest possible rate (in my case 144hz, yet not very accurately ranging between 120-140hz)

the first point might be related to your problem…
the second (esp. the inconsitent framerate) might explain my timing issues above. just that I dont see a solution


#4

Hmmm. Some of this behavior is surprising. I’ve never had an issue with videos changing speed when they loop, and I would have noticed because I have some designs that loop movies several times. You’re getting this on both the Windows machines and the Macs? Can you share the system specs on each?

Another thing that radically affects performance is whether you are playing a movie file with audio or not, but given the framerate variability I’m assuming either not or that it’s massively desynchronized.

Finally, are you using 1.90.3 or one of the PsycoPy3 betas? I’ve never had behavior quite like you’re describing on 1.90.3, so I’m wondering if this is an issue with the beta.


#6

yes, I get it on win7 and macos. see reply below

I present videos without sound, so I can’t comment on audio/video sync

I use psychopy 1.84 with python 2.7. can’t change this: our sys-admins want to keep this version


#7

after some hours of testing and modifications I worked around the problems.
no solutions, not really satisfactory, but it works…

in fact there were two different, and probably unrelated issues:

timing issues:

those occur system specific. apparently psychopy and the graphic-card driver on my presentation machine (win7) do not go together too well. vsynch does not work reliably and the graphic card feedbacks far too fast flip-intervals. those are sensitive to changing opengl settings in my graphic-driver, but never behave fully predictable.

in the end I implemented timing via a system-clock timer.
the problem seems to be low-level, and moviestim loads movies according to the false framerates. so oddly the movie-playback looks fine. the problem remains, that i have no idea about the actual framerate thats presented.

increased presentation speed in reusing moviestim-objects:

presenting one and the same movieobject several times - not via loop argument, but calling it again, after another movie was played - results in increased presentation speed during the second playback (and scales up presenting it 3, 4, or more times). framerate does not increase (in its unreliable boundaries, see above)

this issue appears on all my machines, on win7 and macos. i surpassed it by loadind each trial I want to present as an individual movie-object. not very memory efficient…


#8

That’s interesting. I did set mine up so that it loads a separate movie for every trial. So, it loops a movie within a trial just fine, but for whatever reason if you play another movie and then go back to the first movie it acts weird. Good to know.