Interest in or need for full H/W-accelerated video presentation?

Development
4

I’m happy to see that there’s interest in this topic. Quite a lot of points to respond to.

my experience with gstreamer

My installed version is currently 1.20.1.1, compiled from their latest source a few weeks ago. Unlike what @jon wrote, I haven’t had serious issues either installing or using gstreamer, but my experience is limited to video playback on Windows. Around the time I started using it (2019), Linux seemed to be their main target with less focus on Windows, since then, however, their support for Windows has improved.

That said, their support for Python on Windows isn’t great; I’ve been told it’s possible to compile it via cygwin, but I didn’t want to bother with that. For movie playback on PsychoPy, I wrote a C library that interfaces with gstreamer development libraries, handles initialisation, and exports a few simple functions, such as fetching the next frame. For these few routines, I wrote bindings in ctypes, because I want to keep it simple as long as I’m just experimenting with it. Cython would certainly be a more performant way to do it. There’s plenty of work to do to get this code into a publishable state, and I think it wouldn’t be that difficult to omit the C library completely - the number of routines that need bindings isn’t that great. Linux has python bindings available so things should be even simpler. Mac, I don’t know.

what to expect from gstreamer

I haven’t done a lot of research about platforms other than Windows, so I cannot say how many platforms gstreamer supports or how it compares to ffmpeg, but I believe it’s worth supporting if only for the fact that it’s one of the largest and most well-known systems of its kind. What I can say based on my experience in Windows, and what also should work on Linux (and other platforms?) is this:

  • It can be used the same way as FFmpeg. With automatic modules like playbin, it selects the appropriate codecs needed to play back a movie. Frames can be fetched from the pipeline (I’m using appsink for this purpose).
  • Even better, the automatic codec selection process can be influenced by adjusting codec priorities. I’ve only started to look into this, but if it works well, it could pick h/w accelerated codecs automatically when available, and fall back to CPU-based ones when not. As far as I’m aware, this cannot be done with ffmpeg(?).
  • It’s also possible to manually pick codecs, if one knows that it’s available and matches the media stream. I believe this is the way to go when one wants h/w acceleration in ffmpeg - first inspect the media to detect the format, pick the appropriate codec (using your own knowledge/logic) and supply it as a parameter to ffmpeg. This has been possible to do in PyschoPy for ages (check out my post from 2019).
  • Gstreamer can even do a fully GPU-based pipeline (that is, after the file has been loaded and demuxed). PsychoPy uses OpenGL, and gstreamer can supply frames as OpenGL textures, which can be drawn to the window, with essentially the same drawing code that exists in moviestim modules. For now, the only pipeline that can do this is based on NVidia’s nvcodec, which is capable of outputting GLMemory textures in the same GL context that’s used by PsychoPy. There’s another similar pipeline based on d3d11 but it doesn’t yet support OpenGL output. I’ve seen code for DirectX → OpenGL on-GPU texture zero-copy sharing, but last I checked it wasn’t available in gstreamer.

Certainly, h/w acceleration is also available in FFmpeg, and perhaps it’s also capable of automatically picking an accelerated path if available. In my experience, however, the only consistent way to ensure that the accelerated codec is picked was to manually specify the video codec for FFmpeg. This isn’t that similar in gstreamer, but gst can use codec priorities, as I mentioned above.

However, I did some testing on high-res (4K) video some time ago, and the CPU-based decoding in PsychoPy simply couldn’t reliably do 60 Hz without often missing some frame deadlines. Not on the systems I’ve tested, anyhow. Same should apply for high-framerate media (imagine an experiment that needs 140Hz+ display and wants to use movies). Even with the h/w acceleration-enabling hack for ffmpeg, PsychoPy still copies the frame back from and back to the GPU, which adds too much overhead.

With the fully-h/w pipeline and 4K video playback, frame fetching takes very little time (1-2 ms on i7-8700k + rtx 3090 + win 10 x64), which isn’t surprising as gstreamer decoding runs in a separate process and on the GPU, and only a texture ID changes hands. Drawing that texture with Pyglet takes more time (code modified from moviestim), with occasional peaks around 7-8 ms. I wonder if that’s because those are calls from Python. With a shared GL context, however, gstreamer can in theory directly draw onto the window, which could speed things up (haven’t tested this). Even if this may not be needed for the majority of users, but wouldn’t it be great if PsychoPy had the ability to do this as well? I mean, in addition to supporting gstreamer and having a clean and stable movie playback function.

new moviestim code

I’ve spent a little time peeking into the new moviestim code in @mdc’s repo. I guess I’m not the only one who found having several moviestim modules confusing, so it’s good to see that things are going to be cleaned up. My thoughts (to be taken with a pinch of salt as I didn’t look very carefully):

  1. I think it wouldn’t be extremely difficult to code a player module for gstreamer, such as the one for ffpyplayer.
  2. However, the current Frame class design and the OpenGL rendering code may need to be modified to accommodate on-GPU textures (OpenGL texture ID) as an alternative to keeping all the frame data, if you think that that’s worth supporting.
  3. I didn’t notice any special code for enabling h/w acceleration in ffmpeg either - I wonder if that’s being left to ffmpeg to decide, and if so, how well does it work these days?
1 Like