FFmpeg and a thousand fixes

(Collaborative post by Mateusz “j00ru” Jurczyk and Gynvael Coldwind; a short version is available at the Google Online Security blog).

Following more than two years of work, the day has finally came – the FFmpeg project has incorporated more than a thousand fixes to bugs (including some security issues) we have discovered in the project thus far:

$ git log | grep Jurczyk | grep -c Coldwind
1120

As this event clearly marks an important day in our ongoing fuzzing effort, we decided to provide you with some background on one of the activities we are currently working on.

At Google, security is a top priority — not only for our own products, but across the entire Internet. That’s why members of the Google Security Team and other Googlers frequently perform audits of software and report the resulting findings to the respective vendors or maintainers, as shown in the official “Vulnerabilities – Application Security” list. We also try to employ the extensive computing power of our data centers in order to solve some of the security challenges by performing large-scale automated testing, commonly known as fuzzing.

Back in December 2011 we were really inspired by Tobias Klein, his “Bug Hunter’s Diary” book and specifically the “NULL POINTER FTW” section discussing the discovery and exploitation process of a write-what-where condition vulnerability identified by the author in one of the FFmpeg demuxers responsible for parsing 4X Media (“4xm” in short), with its source code residing in the “libavformat/4xm.c” source file. The security flaw was not difficult to find through manual analysis, since the affected code was contained within several continuous lines of text; while it was just a single example of a trivial programming error, it got us thinking. After all, if there was a simple vulnerability in a C module of less than 400 lines of code performing a relatively simple task, chances were there could have been more similar or less obvious problems in the entire FFmpeg codebase, currently at about 832,000 lines of code (and definitely with more than 0.5MLOC back at the time).

While reading about the 4xm demuxer vulnerability, we thought that we could help FFmpeg eliminate many potential low-hanging problems from the code by making use of the Google fleet and fuzzing infrastructure we already had in place. There were also several other reasons why we decided that taking the project as a fuzzing target would be a good idea:

• FFmpeg had a history of reliability and security issues prior to Tobias’ discovery, see FFmpeg Security website.

• Feeding input to the software and triggering relevant code paths was as trivial as using the standalone ffmpeg executable with appropriate command line options.

• The project was strictly about parsing complex, often proprietary file format structures in native C code – essentially, a paradise for any bug hunter. There were lots of dynamic allocations, arithmetic operations, indexing buffers based on input data, moving memory around and other operations known to be frequently prone to various types of programming mistakes. As a bonus, different parsing modules were developed by different contributors, typically with varying security awareness.

• Input data was readily available – the internet was full of audio/video files in a variety of formats and encoded with different codecs. There were also dedicated corpuses of files designed to be used for media decoder testing. Two examples of such data sets are samples.mplayerhq.hu and the FFmpeg FATE project.

• Roughly at the same time the Google-developed AddressSanitizer run-time memory error detector was gaining recognition. The utility offered instant and accurate detection of common classes of memory-related problems such as out-of-bounds read and write access to {stack, heap, static} arrays, use-after-free, invalid free, double free and more, at the cost of a 2-3x average slow down and some insignificant memory overhead. The utility seemed to be a perfect candidate for improving the detection rate of fuzzing-incurred errors which would otherwise not be detected at all or would manifest themselves in areas of code completely unrelated to the root cause location.

• As a bonus, the ASan team decided to make it compatible with FFmpeg at early stages of the development and later ran the ASan-instrumented FFmpeg over a set of valid input files (not malformed or mutated in any way). Only by doing this, they were able to identify four bugs (see “Found Bugs“), providing us with more evidence that the codebase might require further investigation in search of programming errors in dealing with incorrectly formatted input bitstreams.

All of the above arguments discuss how the nature of FFmpeg made it suitable for automated testing, but there is also the matter of whether finding and having bugs fixed in the product is worthwhile, or precisely, who would benefit from the improved security posture of the project. FFmpeg and its derivatives (such as the spin-off Libav project) are the foundation of many other media-processing programs used both by desktop PC users and companies alike. For a fairly comprehensive list of products built upon, relying on or using parts of FFmpeg, see http://ffmpeg.org/projects.html; notable examples include Google Chrome, MPlayer, VLC and xine. As a result, it was expected that any discovered and fixed bug would make millions of users directly or indirectly more secure, being enough of a justification to proceed and take the effort from idea to realization.

Before any fuzzing actually takes place, it is usually crucial for the success of the operation to gather a set of files with extensive code coverage, so that more (potentially unexpected) program states can be triggered during the fuzzing itself, spinned off the original coverage. We approached the problem by collecting around 7,000 sample media files from the aforementioned samples.mplayerhq.hu website and the FFmpeg FATE regression test suite, later adding more exotic files from the public web in order to further improve the subset of formats and codecs covered by the corpus. Once we were finally happy with the total number of basic blocks touched while processing the test cases (being a good measure of the total code coverage achieved), we made use of some 2,000 cores and relatively simple algorithms (such as bitflipping, swapping bytes, truncating the files and so forth) to mutate the input data, feed it to FFmpeg and save information about any resulting crashes.

The first fuzzing iteration ran for approximately one week and was able to uncover around 130 unique problems in the code, ranging from simple assertion failures to stack-based buffer overflows and other severe conditions:

• NULL pointer dereferences,

• Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access,

• Out-of-bounds reads and writes to stack, heap and static-based arrays,

• Invalid free() calls,

• Double free() calls over the same pointer,

• Division errors,

• Assertion failures,

• Use of uninitialized memory.

Our personal feeling is that between 10% and 20% of the problems could be considered easily exploitable security issues; however, the estimation has not been formally confirmed in any way.

We subsequently contacted the project maintainer – Michael Niedermayer – who submitted the first fix on the 24th of January, 2012 (see commit c77be3a35a0160d6af88056b0899f120f2eef38e). Since then, we have carried out several dozen fuzzing iterations (each typically resulting in less crashes than the previous ones) over the last two years using similar resources, occasionally improving our original corpus and tweaking the mutation configuration (e.g. fiddling with mutation ratios or getting them to match the internal structure of the tested files). Ever since we started the effort, we have been working closely with Michael, who has been extremely keen to work with us and fix all issues we would send his way. The numbers speak for themselves – out of over thousand commits submitted to FFmpeg as fixes to our findings, at least 650 were authored by Michael, which gives an outstanding average of one commit each single day for the last 23 months! We would like to thank him for all the work he has done and continues to do to improve the stability and security of the product; finding the bugs is just the start of a success.

The other ~350 commits in FFmpeg were mostly submitted by Libav project developers: Ronald S. Bultje, Luca Barbato, Alex Converse, Martin Storsjö and Anton Khirnov. We have been concurrently reporting issues in Libav during the last several months and similarly to FFmpeg, the maintainers are doing a great job writing and submitting patches, which FFmpeg is also cherry-picking to their own git repository (large chunks of the two projects are shared, as Libav started as a fork of FFmpeg). While the former project is doing their best to catch up with the latter, the figures speak for themselves again: there are “only” 413 commits tagged “Jurczyk” or “Coldwind” in Libav, so even though some of the FFmpeg bugs might not apply to Libav, there are still many unresolved issues there which are already fixed in FFmpeg. Consequently, we advise users to use the FFmpeg upstream code where possible, or the latest stable version (currently 2.1.1) otherwise. It is also a good idea to carefully consider which formats and codecs are necessary for your use case and disable all other parsers during compilation time, in order to reduce the attack surface to a minimum.

We are presently still improving our corpus and fuzzing methods and will continue to work with both FFmpeg and Libav to ensure the highest quality of the software as used by millions of users behind multiple media players. If interested in the effort, please keep an eye on the master branches for commits marked as “Found by Mateusz “j00ru” Jurczyk and Gynvael Coldwind” and watch out for new stable versions of the software packages. Hopefully, one day we will be able to declare both project “fuzz clean” against most publicly available samples and simple mutation algorithms. Until then, we recommend to refrain from using either of the two projects to process untrusted media files or alternatively to use privilege separation on your PC or production environment, where absolutely required.

Complete lists of developers who have ever submitted patches for bugs we identified in FFmpeg and Libav are shown below (sorted by the number of commits). They clearly illustrate that as of today, FFmpeg includes virtually all fixes developed for Libav, while Libav only has 50 out of a total of 750 Michael’s commits (as previously mentioned, not all FFmpeg bugs affect Libav in the first place, though).

FFmpeg:

   750 Michael Niedermayer
   108 Ronald S. Bultje
    91 Luca Barbato
    77 Martin Storsjö
    48 Anton Khirnov
    29 Alex Converse
     5 Kostya Shishkov
     4 Thilo Borgmann
     1 Vitor Sessak
     1 Reinhard Tartler
     1 Paul B Mahol
     1 Mashiat Sarker Shakkhar
     1 Mans Rullgard
     1 Justin Ruggles
     1 Janne Grunau
     1 Aurelien Jacobs

Libav:

   107 Ronald S. Bultje
    89 Luca Barbato
    77 Martin Storsjö
    50 Michael Niedermayer
    48 Anton Khirnov
    27 Alex Converse
     5 Kostya Shishkov
     2 Thilo Borgmann
     1 Vitor Sessak
     1 Reinhard Tartler
     1 Paul B Mahol
     1 Mashiat Sarker Shakkhar
     1 Mans Rullgard
     1 Justin Ruggles
     1 Janne Grunau
     1 Aurelien Jacobs

We would like to thank all of the above developers for their hard work on making both media libraries better with every single day.