Frames dropping when displaying many images

Hello,
We are trying to display a series of images (1000+) for a short amount of time (~0.2 seconds per image) but are running into issues where frames drop at the beginning of each image’s display period. We think that this may be due to the fact that the cache on the graphics card is too small to store all of the data for the images because there are no frames dropped for the first 100 images, which corresponds with the size of the cache. In order to ensure that the data is in the cache before we try to display the image, we need to pull the image into the cache before it is displayed on the window . This pulling process takes a non-negligible amount of time which results in dropped frames. The only way we could think of to get around this issue would be to use parallel programming in order to pull an image into the cache while the other is displaying, but this is not possible in python due to the GIL. Do you have any recommendations on how to avoid dropping these frames? Any help would be appreciated.

You’ve got 200 ms during each image display. What about doing your cache operation immediately after the previous image is drawn, figuring out what time has elapsed, and then get back into the win.flip() cycle by reducing the number of screen refreshes as required prior to the next stimulus? Or if you graphics card supports Free Sync/G Sync, you could use that to avoid needing to keep up with the screen refresh cycle altogether, just presenting new images on a non-slip 200 ms schedule?

Hi Michael, thanks for the suggestion!

I tried to figure out the elapsed time of pulling the next image in cache, but it always seemed to output a cost that’s less than a frame interval. which indicates no dropping frames. However, I know it’s not true since the end result shows I have dropped frames at these intervals.
Is there anything wrong with my timing method? This is how I time:

        clock = core.MonotonicClock()
        self.pull_in_cache(self.current)  #pull the next stim in cache
        self.list[self.current].stim.draw(MyWindow.win)   #draw the stim i want to display  
        time = clock.getTime()
    
        global actual_framerate 
        cutoff = 1. / float(actual_framerate)  + 0.005 # 5 ms range
        frames_dropped = ceil((time - cutoff)/cutoff)  <= always results in 0

How do you know you’re dropping frames? Perhaps the cause of that is actually something else?

I’m not sure I really understand your “frame dropping” code. I guess I’d rather be trying to actually draw frames on every refresh and count how many happen, and compared them to their expected times. i.e. an actual direct measure of frame dropping.

Having said that, if frame dropping is an issue, you don’t actually need to be trying to redraw on every screen refresh. Just .draw() and .flip() once at the start of every 200 ms period. Maintain an overall time schedule (0, 0.2, 0.4, 0.6 s, etc) rather than reset the timer on each stimulus, to avoid temporal slippage.

e.g.

  • set the stimulus counter to -1
  • draw the first stimulus
  • do the first win.flip()
  • reset the timer to 0 (only on this first frame)
  • enter a loop through all of your remaining stimuli:
    • do your cache stuff and draw the new stimulus to the back buffer
    • increment the stimulus counter
    • enter a timing loop. When the timer is > stimulus counter * 0.2 + 0.19, win.flip()

This allows you to sync back up with the screen refresh cycle towards the end of the 200 ms period. You are only drawing one update per stimulus, so frame dropping is not an issue.
On Mac or Linux at least, in the timing loop, you should do time.sleep(0.001) on each iteration to allow other processes time to do their thing. Otherwise, you can get timing problems due to other processes jumping in and disrupting your control of the CPU. On Windows, you can’t sleep for such short periods, so maybe you want to do multiple flips, starting at, say, 100 ms. i.e. during the win.flip() pause period, you free up the CPU anyway.

Thank you for the response! Unfortunately, it still seems like flipping is taking too long. Around 10% of the time, flipping takes greater than .2 seconds (sometimes up to .7 seconds), meaning that by the time the image is shown, the next few images should also have been shown. This results in one image being shown for too long, and the successive images are flashed in order to catch up in terms of time. And trying to fill the cache in advance does not help, as we cannot find a way to force the images into the cache other than by using the win.flip() method. This is how we have our code structured:

for i in range(1, len(self.list)): #self.list is the list of imagestims
        if i + 2 < len(self.list): # pulling the next next image into cache in advance
            self.list[i+2].stim.draw(MyWindow.win)
            self.list[i-1].stim.draw(MyWindow.win) #cover cached stim with current stim
            c = core.MonotonicClock()
            MyWindow.win.flip() #flip to pull into cache - often takes >.2 seconds
            a = c.getTime()
            if a > 0.2:
                print(a)
                print(i)
        MyWindow.win.clearBuffer()
        self.list[i].stim.draw(MyWindow.win) #draw next stim to back buffer
        while clock.getTime() < (i * 0.2 + 0.19):
             pass
        MyWindow.win.flip()

Do you have any more insight into this problem? We are feeling very stuck.

700 ms seems a very long time for such a simple task…

Assuming everything is going OK, your code spends most of the time in many millions of iterations of this very tight loop:

while clock.getTime() < (i * 0.2 + 0.19):
    pass

There is an issue with loops like this, in that they exclusively consume the CPU, starving other processes (including the OS) of time they need. That can lead to a bottleneck, where perhaps they break into do what they need, disrupting your timing. There are pauses every 200 ms during your win.flip() periods, but maybe that isn’t enough?

If you are running on a Mac or Linux, I’d suggest doing this instead:

while clock.getTime() < (i * 0.2 + 0.19):
    time.sleep(0.001)

i.e. slow the loop to ~ 1 kHz, yielding 1 ms per iteration to competing processes, so they have time to breathe. See if that makes a difference?

Unfortunately, I think there are still issues on Windows with the sleep function, I don’t think it can be as granular as 1 ms, with a minimum more like 16.6 ms, so if this approach is used, it needs to be a bit more sophisticated (e.g. do it while there is still, say 170 ms to go, then avoid that for the remaining period of the interval).

Lastly, although I’m sure you’re aware of this, but are the source bitmap files already scaled to the resolution they will be displayed at on screen? e.g. if your source file is a multi-megapixel photo, but your screen is 1024 × 768, then all those extra pixels are just consuming time and memory for no purpose. i.e. the source image should be scaled to 1024 × 768 itself.

We tried changing the pass to time.sleep(.0166) since we are using windows, but it still is not effecting the fact that first flip is taking >200 ms to run. All of the images have already been cropped to the size of the window before the program is run, so there are no extra pixels to worry about.

We also did some testing and found that second flip never takes more than 200 ms, it only takes about 20 ms. This seems to indicate that when the first flip takes too long, it is due to memory access as the flip on the cached image does not have the same timing issue.

Additionally, different images are causing this issue on each run, so there is nothing specific to the image file which causes the slowdown.

Are we correct in assuming the flip method is what forces the image into the cache?

This is really a question for @jon, but my relatively uninformed belief is that the heavy lifting for getting data to the graphics card is actually during the .draw() method of each stimulus. I suspect you might as much about this as me, though.

Meanwhile, win.flip() is almost instantaneous, it’s just flipping the back buffer to the front. The delay you see around that line in your code is just the pause to synchronise with the refresh cycle of the display, i.e. while we wait for that new content to be physically shown on the screen. The flip process itself should be very very quick. The data already exists in the back buffer, you’re just telling the graphics card to use it rather than the current (front) buffer.

For image stimuli the image is sent to the graphics card when the image is set, not when draw() is called.

In your code it looks like you’re trying to create a massive list of ImageStim objects and then call each of them and indeed, that will probably be too much for your graphics card memory. Mike was right in saying you should load the next image while presenting the previous but, in your case, the key is that you should only create a single image stim and update its contents each time. You need to create a list of just the image filenames and load them from disk during the presentation of the previous. You could use your own timer to handle the cap after loading the image (taking account of the time it took to load) but you could also use PsychoPy’s StaticPeriod.

Something like this (untested):

# before starting the trial preload the first image
stim.image = filenames[0]

# start your stimulus sequence
for imageN in range(len(filenames)):
    stim.draw()  # the current image
    win.flip()  # get it onscreen

    # prepare stim for next presentation
    stimTimer = StaticPeriod(screenHz=60)
    stimTimer.start(0.2)  # start a period of 0.2s
    if imageN+1 < len(filenames):
        # take next image and load it to gfx card
        stim.image = filenames[imageN+1] 
    stimTimer.complete()
    # will now loop around to next image, which has already been loaded

Thank you so much for your suggestion, we no longer appear to be dropping frames! We really appreciate your help with this issue!