psychopy.org | Reference | Downloads | Github

MovieStim3 / moviepy issues - large initial frame time, "catching up" after seek

Hi, first poster here. I would like to report some issues with movie playback using moviestim3/moviepy. More specifically:

  1. The first frame (containing movie.draw() and win.flip()) takes very long to display, possibly messing up timing of video presentation.
  2. After seek(), the movie does not start from the new position, instead, a few frames from before the seek are displayed, after which the video jumps to the new position.
  3. Also after seek(), the video is played at a higher frame rate, as if to “catch up” with some kind of mistaken timer.

My video was playing choppy sometimes, playing too fast other times, so I did some testing in which I measured frame draw and total cycle timings.

Disclaimer: I’m fairly new to psychopy and python itself, so my methods, thinking, conclusions may be inherently mistaken. I’m happy to learn new things so please be patient and explain.

My setup: I’m using psychopy 3.2.4 library in python (not the standalone); moviepy 1.0.1, pygame 1.9.6 on win10 x64, GTX 1080Ti, i7-8700K.

I used the following code for the tests:

import psychopy.core, psychopy.visual, psychopy.event, psychopy.monitors
import time
import numpy as np
import matplotlib.pyplot as plt

# setting up monitor, opening window
monitor = psychopy.monitors.Monitor('LG32UD99W', width=70, distance=40)
monitor.setSizePix([3840, 2160])
win = psychopy.visual.Window(size=[2560, 1440], monitor=monitor, units="deg", screen=0, color=-1., colorSpace='rgb', fullscr=False, allowGUI=False)

# 2 movies: 720p or 1440p, both have no audio track
movie = psychopy.visual.MovieStim3(win, '720p.mkv', size=(1280, 720), noAudio=True)
# movie = psychopy.visual.MovieStim3(win, '1440p.mkv', size=(2560, 1440), noAudio=True)

# The 3rd test included this seek command:
# movie.seek(150)

movie.play()

# loop 300 win.flip() cycles, measure movie.draw() time and cycle time
n = 300
frametimes = np.zeros((2, n), dtype=float)
for i in range(n):
    a = time.perf_counter_ns()
    movie.draw()
    b = time.perf_counter_ns()
    win.flip()
    c = time.perf_counter_ns()
    frametimes[:, i] = (b - a, c - a)

# cleanup
movie.stop()
win.close()

# display frame time plot
ft_ms = frametimes / 1e06
fig = plt.figure(figsize=[14, 4.8])
plt.suptitle('720p')
plt.plot(ft_ms[0, :], label='draw')
plt.plot(ft_ms[1, :], label='flip')
plt.xlabel('iteration #')
plt.ylabel('time (ms)')
plt.legend()
plt.show()

Testing results (only 3 shown here, did many more with similar results):

  1. 1280x720, 25 fps movie

Most of this image is as it should be: cycle total times (labelled as flip) correspond with the monitor’s refresh rate (1/60); draw times zig-zag because the movie frame rate is 25 so in about half the cycles the draw() function reuses the cached last frame (moviepy keeps track of the elapsed time and only provides new frames when the time is right); decoding new frames takes some time, but when the last cached frame is used, the draw delay is very low. However, the first frame takes very long (red circle). It is not always the first frame, though; in some tests it comes a few cycles later. Why does it happen? Is there some kind of initialisation going on? Is my testing method faulty? I haven’t digged very deep into the code but I’ll try to find out. I’m also not 100% sure that this affects actual display timing, but it could.

  1. 2560x1440p, 25fps movie
    << I had an image here too but new users are limited to 2 / post so this one had to go. >>
    Same test with a larger-resolution movie. Basically the same result, with the first frame delaying long, the rest as expected. Frame decoding times take up nearly all the available time in a 1/60 s cycle, though. The total decoding time is not prohibitive for this resolution at this refresh rate, so it would be nice if there was a way to spread out the decoding over frames - something like a frame-server that takes care of pre-loading and decoding images, and serving them as needed. That would also allow even faster refresh times at this resolution (a topic brought up in other threads in this forum).

  2. Same 1440p movie, but adding a seek(150) before play()

The large initial frame delay is there, but the main point of this figure is the part after the initial peak (red ellipsis). There is no zig-zagging during the first ~15 cycles. This, I believe, is some kind of timing problem, perhaps the movie thinks it’s lagging behind so the frames are served at every cycle to catch up? A brief look into the code tells me that both psychopy and moviepy have their own way of calculating frame timings, but I haven’t spent time on understanding why this happens. Why is this a problem? Because if I ever do a seek() in my experiments (for example to randomise which part of a longer video would play during a trial), the video would play very fast initially, which is unacceptable.

My conclusions, please let me know if you agree:

  1. Does the initial long frame delay affect trial display times in a bad way? The duration is not constant, so if actual display times vary like this, that’s not very nice. I know that for best timings one should use a photodiode to check actual display times, which I will do, but if this delay is there, it should be eliminated nevertheless.

  2. As it is now, I think it’s better to avoid any use of seek(). It was not part of my test but I mentioned before that frames from before the seek leak into the video, and then the video image jumps to the new position. Plus, the video can play at a faster rate to “catch up”. In a streaming application this faster playback might be desirable, but in an experiment it is not acceptable. Delays can be measured with a photodiode, but the video rate should be reliable. Obviously, a 30 fps video would be better for a 60 Hz display, but the small rate difference is tolerable.

I was thinking, while I’m at it, why not post a few more movie playback timing-related figures?

BtW, I figured that movie.play() in my code above was unnecessary. Removing it doesn’t change a bit, though.

  1. Pause/Play (720p 25fps movie)


    This figure shows draw/flip delays during a playback in which the movie was paused for 1 sec in the middle. In the code, I added .movie.pause(), time.sleep(1.), movie.play() in the loop when i==n/2. The upper half shows the times spent in draw() and total cycles times; the lower part shows timings sampled right after win.flip(). Pause doesn’t seem to introduce any bad effect.

  2. Seek revisited


    I tried to include 5 secs of sleep after seeking (time.sleep(5.)) to give the machine a little respite to prepare, but it didn’t improve anything. The above figure is interesting because it shows a big delay at the beginning of playback (a few frames in). At that point, even the draw() time is increased by a lot - btw this is also shown in the figure above. Furthermore, at the beginning of the playback, a few frames were displayed from the beginning of the movie, that is from the position before the seek. Those frames should not have been displayed at all. I have no way to check, but I suspect the few frames before the delay in my figure might correspond to those out-of-place frames. TLDR, don’t use seek. Sadly, it’s inconvenient for me because I used it to display various parts of a nature video during auditory stimulation.

First, try a different encoding format. Especially on windows, MovieStim3 can be highly variable depending on the kind of movie file. I’ve had good results with .mp4 files.

Second, try loading, calling core.wait(.1), and then draw/flip/playing. It might just need longer to load everything into memory initially.

Third, try calling pause and then seeking and then playing. I’ve had some better behavior using “seek” when I pause the movie first.

Hi and thanks for your reply!

  1. I’ll have a look at MP4. Do you mean it’s better for seeking or faster for decoding? I’ve tried RAW video file as well, and the total presentation (incl. loading, decoding etc.) did not change much.

  2. I’ve figured out that a simple flip() is enough to get rid of the initial long frame time. Just load the movie, flip once before starting to play, and it’s all good. I guess the case I was looking at was unrealistic in the sense that experiments rarely start by displaying a movie; usually there’s some kind of welcome screen or something else before trials. Basically, nothing anyone has to worry about unless starting the movie immediately.

  3. Tried many different things to improve seek, none helped. Pause and play may work in your case, sadly, it didn’t help in mine. Could depend on the type (resolution, encoding etc) of the movie, not sure. I haven’t spent the time to delve in the code to understand what’s going on there.

Happy New Year soon, everyone!