Video Decoding using libav and ffmpeg

I spent the last month completely taking the libavg video decoding module apart and putting it together again. I’m finally convinced that the code is well-designed and readable – and it’s fast. It turns out that getting good video decoding is not as easy as it sounds, so I’ve written up a complete description of the insides for anyone that’s interested: VideoDecoding.

The weird thing is that from the outside, it looks like a solved problem, so every time I start telling someone about this, I get the same reaction. There’s libraries for that, right? You just plug in libav or ffmpeg or gstreamer or proprietary things like QuickTime or DirectShow. All these libraries have existed for years, so they’re stable and easy to use, right?

Well, yes and no. If you don’t need advanced features, high-level libraries like gstreamer might do what you want. But we want frame-accurate seeking and a low-latency mode, as well as color space conversion using shaders. Opening and closing video files shouldn’t cause any interface stutters, and so on. Also, libavg can’t work with proprietary libs – we need something that works cross-plattform. That leaves libav/ffmpeg, and this library exposes a pretty low-level interface. It does support every codec but the kitchen sink (pardon the wording) and gives you control over every knob that all of these codecs have to tune things. That’s really great, because you wanted control, right? Anyway, you can get everything done with libav/ffmpeg, but suddenly things get complicated. For starters, you’re suddenly juggling five threads: demuxer, video decoder, audio decoder, display and audio mixer. libav/ffmpeg leaves all the threading to the user, so you’re dealing with a pretty complicated real-time system where lots of things happen at the same time. Dranger’s tutorial helps, but it’s dated.

To make things worse, the interface of libav/ffmpeg changes with minor revision numbers, so to support a few years of operating systems, you find yourself adding a generous amount of #ifdefs to the code. I couldn’t find documentation that describes which changes happened in which minor revision, so you need to guess appropriate version numbers for the #ifdefs based on tests with multiple systems. Oh, and there’s actually several constituent libraries that each have their own version number. Of course, you need to query the correct one. All of that takes time; the resulting code is hard to read and test. In addition, since ffmpeg forked and the developers aren’t on speaking terms (see this and this if you really want to know more), you need to test with libav (the fork) and ffmpeg (the original) if you want maximum compatibility.

All of this is really a pity, because I think the libav/ffmpeg developers are insanely smart guys and the library does do a really admirable job of de- and encoding everything you can throw at it. Also, if I’m honest, most of the time spent was figuring out how to organize the different threads well – and that’s something I really can’t blame libav/ffmpeg for.

Anyway, we’re now ready to add Raspberry Pi (read: OpenMAX IL) and VA-API hardware decoding, seamless audio loops and other cool things to libavg.

5 thoughts on “Video Decoding using libav and ffmpeg

  1. Hello,
    any updates about video support on raspberry pi? I am implementing digital signage for our company with pi and libavg and would be nice to integrate video funcionality.

    • No updates yet – we’re working on it. VAAPI support got a bit of a priority boost because somebody is sponsoring the development on that end (I’ll post more on that when it’s done). Given finite development resources, that pushed OpenMAX support back.

  2. OpenMAX support got pushed way back I presume??? Please let us know when we can get atleast a beta version to test? People are still waiting on it.

    And not only video rendering but general rendering speedups on the RPi would be great.

    • Yeah, sorry this hasn’t happened yet, although ‘pushed way back’ isn’t really the right way to put it. In projects like these, it’s mostly about finding someone competent who is willing to put in the effort of adding a feature in his/her free time – and that someone isn’t there right now as far as RPi support is concerned.

      Of course, anyone willing to try is very welcome and I’ll give him/her all the support I can :-) .

Leave a Reply to hellbender Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>