psychopy.org | Reference | Downloads | Github

Rare and strange conflict between sound file and movie file with ffmpeg

movies
sound

#1

In short, I’ve built a script for running studies with babies that plays an attention getter followed by a movie file. The attention getter consists of a looming geometric shape and, at the same time, plays a short .wav sound file which is loaded at the start of the script. Immediately after the attention getter, the script displays the first frame of the movie, paused. The movies are all .movs with mpeg-4 encoding and 48000Hz mono audio.

I built in the capability to re-play the attention-getter if the infant does not immediately look at the screen, with the first frame of the movie frozen on the screen. So, the movie disappears, the attention getter plays again, first frame reappears paused. All fine.

The issue is, if the movie has sound, sometimes (and only sometimes, which is maddening) after re-playing the attention-getter several times, when I start trying to play the movie, I get the following error (picking up from the draw command in my script):

    dispMovie.draw()
  File "/Applications/PsychoPy2.app/Contents/Resources/lib/python2.7/psychopy/visual/movie3.py", line 383, in draw
    self._updateFrameTexture()  # will check if it's needed
  File "/Applications/PsychoPy2.app/Contents/Resources/lib/python2.7/psychopy/visual/movie3.py", line 303, in _updateFrameTexture
    self._numpyFrame = self._mov.get_frame(self._nextFrameT) 
  File "<decorator-gen-14>", line 2, in get_frame
  File "/Applications/PsychoPy2.app/Contents/Resources/lib/python2.7/moviepy/decorators.py", line 89, in wrapper
    return f(*new_a, **new_kw)
  File "/Applications/PsychoPy2.app/Contents/Resources/lib/python2.7/moviepy/Clip.py", line 95, in get_frame
    return self.make_frame(t)
  File "/Applications/PsychoPy2.app/Contents/Resources/lib/python2.7/moviepy/video/io/VideoFileClip.py", line 74, in <lambda>
    self.make_frame = lambda t: reader.get_frame(t)
  File "/Applications/PsychoPy2.app/Contents/Resources/lib/python2.7/moviepy/video/io/ffmpeg_reader.py", line 163, in get_frame
    result = self.read_frame()
  File "/Applications/PsychoPy2.app/Contents/Resources/lib/python2.7/moviepy/video/io/ffmpeg_reader.py", line 106, in read_frame
    s = self.proc.stdout.read(nbytes)
AttributeError: FFMPEG_VideoReader instance has no attribute 'proc'

My best guess is some kind of conflict caused by the sound.play() of the attention-getter using the same FFMPEG resources as the movie’s sound.play (because moviestim uses psychopy.sound for its sound), so in theory this could happen if you were alternating sound files as well.

Has anyone else encountered this, or have any ideas about how to stop it from happening?


#2

Hi,

Note that the error comes from the _FFMPEG_VideoReader object, not from the FFMPEG_AudioReader:

FFMPEG_VideoReader instance has no attribute ‘proc’

The audio and the video streams of a movie are treated as two separate streams (see movie3.py).

Probably the close() method of the FFMPEG_VideoReader object has been called somewhere before (provided that the subprocess.popen which creates the proc object never returns None).
Try to insert some debug print in FFMPEG_VideoReader.close() and see what happens.

Hard to guess more without having a look to your code.

Cheers,
L.


#3

Hi Luca,

Sorry for the slow response. Haven’t had a chance to try the debug code in FFMPEG yet, but I’ll be able to do that probably Monday. In the interim, you can see the code for yourself here: https://github.com/jfkominsky/PyHab


#4

@luca.filippin I added some info to the ffmpeg_reader code in the close function, as you suggested. That has been very helpful. That function is being called all kinds of times that it should not be, but it is inconsistent about whether it actually crashes the program.

On further investigation the relevant call occurs whenever ffmpeg_reader’s self.initialize(t) function is called. It calls self.close before initializing. It should then re-create self.proc at the end of this function, and the crash seems to occur if it fails to do so. When does it call self.initialize? In this specific case, as part of get_frame(), which I have pasted below in full, with the relevant line marked

    def get_frame(self, t):
        """ Read a file video frame at time t.

        Note for coders: getting an arbitrary frame in the video with
        ffmpeg can be painfully slow if some decoding has to be done.
        This function tries to avoid fectching arbitrary frames
        whenever possible, by moving between adjacent frames.
        """

        # these definitely need to be rechecked sometime. Seems to work.
        
        # I use that horrible '+0.00001' hack because sometimes due to numerical
        # imprecisions a 3.0 can become a 2.99999999... which makes the int()
        # go to the previous integer. This makes the fetching more robust in the
        # case where you get the nth frame by writing get_frame(n/fps).
        
        pos = int(self.fps*t + 0.00001)+1

        if pos == self.pos:
            return self.lastread
        else:
            if(pos < self.pos) or (pos > self.pos+100):
                self.initialize(t) #The troublemaker
                self.pos = pos
            else:
                self.skip_frames(pos-self.pos-1)
            result = self.read_frame()
            self.pos = pos
            return result

The initialize function is typically called, as intended, when I tell it to seek to an earlier point in the video file. That doesn’t seem to generate a crash, ever. The crash seems to occur under a very specific set of circumstances that requires going into how PyHab works, a little bit, but basically when starting to play a movie initially.

Also, it only crashes on some computers and not others. The crashes seem to occur specifically on a mid-2015 iMac, but this anomalous call to initialize() still occurs on my very new MacBook pro and seems to cause some weird frame skipping issues, in that it won’t render frames until a sound occurs in the movie file. Both are running 1.85.6 and high sierra, but the new computer has more RAM and an SSD. However, this initialize() call shouldn’t be happening at all, even if it doesn’t always cause a crash.

Here’s a more detailed breakdown of the specific circumstances:

PyHab plays an attention-getter to get the baby to look at the screen, which consists of a .wav file and a rotating visual.Rect object. After the attention-getter it waits 100ms and then shows the first frame of the movie file for that trial, paused, for a minimum of 200ms and until the experimenter presses the “B” key to indicate that the infant is looking at the screen, at which point the movie begins playing. The experimenter can also get the attention-getter to play a second time by pressing the “A” key.

The crash occurs if the experimenter starts holding down the B key while the attention-getter is still playing, regardless of the number of times the attention-getter has played in the past, and regardless of whether that very frame of the movie has been drawn before. It calls this initialize function under those circumstances and those circumstances only.

Below, I’m going to just provide a whole bunch of the relevant code from one particular script where I reliably run into this issue. First, the call to play the attention-getter.

if playAttnGetter:
                attnGetter() #plays the attention-getter
                core.wait(.1) #this wait is important to make the attentiongetter not look like it is turning into the stimulus
                frameCount=0
                dispTrial(0,disMovie)
                core.wait(freezeFrame) #this delay ensures that the trial only starts after the images have appeared on the screen, static, for 200ms
                waitStart = True

I’ve put attnGetter’s full code here, click to expand.

attnGetter() function
def attnGetter(): #an animation and sound to be called whenever an attentiongetter is needed
     HeyListen.play() #add ability to customize?
     x=0
     attnGetterSquare.ori=0
     attnGetterSquare.fillColor='yellow'
     for i in range(0, 60): #a 1-second animation
         attnGetterSquare.ori+=5
         x +=.1
         attnGetterSquare.height=sin(x)*120
         attnGetterSquare.width=tan(.25*x)*120
         attnGetterSquare.draw()
         win.flip()
     statusSquareA.fillColor='blue'
     if blindPres < 2:
         statusTextA.text="RDY"
     statusSquareA.draw()
     statusTextA.draw()
     if blindPres <2:
         trialText.draw()
         if blindPres < 1:
            readyText.draw()
     if verbose:
         statusSquareB.fillColor='blue'
         if blindPres < 2:
             statusTextB.text="RDY"
         statusSquareB.draw()
         statusTextB.draw()
     win2.flip()
     if stimPres:
         win.flip() #clear screen

You’ll note I have two windows going simultaneously here. That’s because one is the display screen, and the other is an experimenter interface.

Anyways, at this point there are no problems. After the above code, there is a while loop that hinges on “waitStart”. The relevant part of the loop is here:

While loop
while waitStart and not AA:
            if keyboard[key.Y]: #End experiment right there and then.
                endExperiment([[0,0,0,0,0]],[[0,0,0,0,0]],trialNum,trialType,[],[],stimName) 
                core.quit()
            elif keyboard[key.A]:
                if stimPres:
                    if playAttnGetter:
                        attnGetter()
                        core.wait(.1)
                    dispTrial(0,disMovie)
                    core.wait(freezeFrame)
                else:
                    core.wait(.2 + freezeFrame)
            elif keyboard[key.B]:
                waitStart = False
                if blindPres < 2:
                    statusSquareA.fillColor='green'
                    statusTextA.text="ON"
                statusSquareA.draw()
                statusTextA.draw()
                if blindPres < 2:
                    trialText.draw()
                    if blindPres < 1:
                        readyText.draw()
                if verbose:
                    if keyboard[key.L] and blindPres < 2: 
                        statusSquareB.fillColor='green'
                        statusTextB.text="ON"
                    elif blindPres < 2:
                        statusSquareB.fillColor='red'
                        statusTextB.text="OFF"
                    statusSquareB.draw()
                    statusTextB.draw()
                win2.flip()

Again, a bunch going on that’s not hugely relevant, but included for context (AA is set at the very start and does nothing in the study that keeps crashing).

As I said, the crash occurs when I am holding down the B key while the attention-getter is still playing. So, on its first pass through the loop, the key.B criterion is met, and it’s off to the races. That’s very simple, at the end of the loop, a doTrial() function is called, which itself contains a while loop with a complex set of conditions (not relevant), but what seems to trip the crash is the very first call to my dispTrial() function, which is basically my “draw everything” function. Here’s the whole thing, with the crash line marked. I know, my code is sloppy as all get-out (this is also an older version, my current iteration is a little better).

dispTrial
def dispTrial(trialType,dispMovie=stimPres): #if stimPres = false, so too dispMovie.
    global frameCount
    global pauseCount
    global locAx
    global locBx
    #first, let's just get the status squares out of the way.
    if keyboard[key.B] and blindPres < 2:
        statusSquareA.fillColor='green'
        statusTextA.text="ON"
    elif trialType==0 and blindPres < 2:
        statusSquareA.fillColor='blue'
        statusTextA.text="RDY"
    elif blindPres < 2:
        statusSquareA.fillColor='red'
        statusTextA.text="OFF"
    else:
        statusSquareA.fillColor='blue'
        statusTextA.text=""
    statusSquareA.draw()
    statusTextA.draw()
    if blindPres<2:
        trialText.draw()
        if blindPres < 1:
            readyText.draw()
    if verbose: 
        if keyboard[key.L] and blindPres < 2:
            statusSquareB.fillColor='green'
            statusTextB.text="ON"
        elif trialType==0 and blindPres < 2:
            statusSquareB.fillColor='blue'
            statusTextB.text="RDY"
        elif blindPres < 2:
            statusSquareB.fillColor='red'
            statusTextB.text="OFF"
        else:
            statusSquareB.fillColor='blue'
            statusTextB.text=""
        statusSquareB.draw()
        statusTextB.draw()
    win2.flip() #flips the status screen without delaying the stimulus onset.
    #now for the test trial display
    if stimPres:
        if frameCount == 0: #initial setup
            dispMovie.draw() #***THIS IS THE LINE THAT GENERATES THE CRASH***
            frameCount+=1
            if trialType == 0:
                frameCount=0 # for attn-getter (i.e., the frozen first frame after the attention-getter)
                dispMovie.pause()
            win.flip()
        elif frameCount == 1:
            #print('playing')
            dispMovie.play()
            dispMovie.draw()
            frameCount+=1
            win.flip()
        elif dispMovie.getCurrentFrameTime() >= dispMovie.duration-.05 and pauseCount< ISI*60: #pause, check for ISI.
            dispMovie.pause()
            dispMovie.draw() #might want to have it vanish rather than leave it on the screen for the ISI, in which case comment out this line.
            frameCount += 1
            pauseCount += 1
            win.flip()
        elif dispMovie.getCurrentFrameTime() >= dispMovie.duration-.05 and pauseCount >= ISI*60: #MovieStim's Loop functionality can't do an ISI
            #print('repeating at ' + str(dispMovie.getCurrentFrameTime()))  
            frameCount = 0 #changed to 0 to better enable studies that want to blank between trials
            pauseCount = 0
            dispMovie.draw() # Comment this out as well to blank between loops.
            win.flip() 
            dispMovie.seek(0) 
        else:
            dispMovie.draw()
            frameCount+= 1
            win.flip()

I know it’s the first call because it specifically crashes on a call where the case that the frame counter I use to track where I am in the movie is set to 0 (i.e., the start of the movie). This occurs even when this frame has been drawn previously without the movie’s play function being called.

I’m pretty well stumped at this point. Having narrowed down the circumstances of the crash I can at least avoid it, but I would much rather fix it. I’m open to suggestions.


#5

Hi Johnatan,
if the initialize() function is called from get_frame() then

either (pos < self.pos) or (pos > self.pos+100)

where 100 is the pipe’s buffer size in terms if movie frames (roughly 4 secs for 25 frame/secs movie).
So the initialize(), which instantiates the ffmpeg reader process, occurs either when you jump backward in the movie stream or when you jump forward of more than 4 secs.

I’d try to understand why the value of the time t passed to get_frame() and used to calculate pos is such to cause a call to initialise().

Then I’d also have a look to the parameters passed to ffmpeg when the call to popen return None.

Cheers,
Luca


#6

Thanks for this. I figured out when the anomalous initialize call is getting tripped. It’s a real hum-dinger I tell you what. The ultimate culprit turns out to be my old nemesis, movie3.py, particularly in _updateFrameTexture(). This function is what calls get_frame and all that. The tricky thing is, whenever _updateFrameTexture() is called, the last two lines are as follows:

        if not self.status == PAUSED:
            self._nextFrameT += self._frameInterval

However, _updateFrameTexture() is called when the movie is loaded, meaning that even if you immediately call “pause”, if it attempts to redraw the first frame again it will trip ffmpeg_reader’s get_frame() thinking that it has advanced a frame to 2 (it sort of has), and then when you call “play”, movie3.py calls:

  self._videoClock.reset(-self.getCurrentFrameTime())

Setting the ffmpeg_reader get_frame() t back to 0, setting pos to 1 when self.pos had previously been set to 2, tripping the re-initialize function. My guess is that the crash is a result of the re-initialize function being called to frequently or too soon after a different pyo sound (because this specifically happens on movies that have sound, and I’m using pyo on Mac because pygame is even slower and sounddevice doesn’t work), and this is causing some kind of memory glitch or something. I get enough segfaults when everything else is working properly that I strongly suspect pyo is at the heart of this (I hope we get sounddevice working soon).

So I changed that line of _updateFrameTexture() to:

        if self.status == PLAYING:
            self._nextFrameT += self._frameInterval

This stops the anomalous skip call and the anomalous re-initialization, without messing up which frame it draws when I call movieStim3.draw(), so I think this should fix it. I’ll make sure this fix works on the computer that had been crashing and if so put in a pull request later today. If so, I’ll mark this solved.

EDIT: Yup, that seems to have done the trick. Hooray!


#7

Well done you guys! Isn’t it always the case that the most bizarre errors are caused by such small and benign-looking code? Hats off to you both for your perserverance