PDF Fuzzing Fun Continued: Status Update

(Collaborative post by Mateusz “j00ru” Jurczyk and Gynvael Coldwind)

Almost five months ago, Gynvael Coldwind and I wrote about an effort to improve the security of popular PDF parsing and rendering software; back then, we were primarily focused on the Chrome PDF Renderer and latest Adobe Reader applications. In order to achieve our results, we used several hundred CPU cores to create a unique, minimal set of PDF documents aimed at optimal code coverage. That corpus, which we now consider a fundamental part of our bug hunting success, was used as fuzzing input to numerous mutation algorithms (basic bitflipping, undisclosed PDF-specific algorithms that respect the primary rules of a document’s structure, and things in between).

As a quick recap, we found more than 50 vulnerabilities ranging from low (unlikely exploitable) to high (serious memory errors) severity in the PDF-parsing component of Google Chrome with the help of AddressSanitizer. All of these were fixed by the Chrome Security Team by August 2012, mostly by Chris Evans – kudos! In addition to that, we also reported around 60 Adobe Reader crashes appearing to be unique in June last year. This consequently resulted in a total of 25 separate fixes addressed by 12 CVEs, as described by the Adobe in their APSB12-16 security bulletin and implemented in the new 9.5.2 and 10.1.4 versions of the software for Windows and Mac OS X platforms. Unfortunately, a few very important questions remained unanswered, such as “what about the remainder of the reported bugs?” and “what about the security of Reader for Linux users?”. With Adobe releasing new versions for all supported Reader branches and platforms today (9, X, XI for Windows, Mac OS X and Linux), we would like to take the chance to give you an update on where we stand with PDF fuzzing, and what thoughts we have around Reader and the many other pieces of software people use in their daily work with documents.

Let’s start with Google Chrome – has anything changed since our last posting? Well, we’ve kept playing with different fuzzing configurations and algorithmic approaches, and discovered 20 new security issues over the last six months, summing up to a total of 78 unique bugs found and fixed in the renderer in 2012. As of now, we are not aware of any unfixed non-DoS issues in the PDF Chrome component.

[134955] [135488] [137106] [137288] [137302] [137547] [137556] [137606] [137635] [137880] [137928] [144579] [145079] [145121] [145163] [146462] Medium CVE-2012-2875: Various lower severity issues in the PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

[143798] [144072] [147402] High CVE-2012-2890: Use-after-free in PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

[145029] [145157] [146460] High CVE-2012-2895: Out-of-bounds writes in PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

Coincidentally, the Chrome Security Team just doubled the rewards for all significant vulnerabilities found in the PDF renderer, see a complete list of changes to the VRP rules at http://blog.chromium.org/2012/08/chromium-vulnerability-rewards-program.html. For anyone interested, Chrome Security issues eligible for a reward (all other, too) can be filed under http://crbug.com/. Good luck!

We used our corpus to fuzz two other PDF projects: poppler, an actively-maintained open-source library having its roots in xpdf (which it was originally a fork of) and ghostscript, which includes its own lightweight PDF parsing library called MuPDF. Both libraries can be downloaded and built manually, enabling the usage of AddressSanitizer, and thus, improving the memory error detection ratio. As you might expect, a number of serious problems have been localized in these projects, and they are currently being worked on by their respective development teams, together with some security teams associated with Linux distributions. We would especially like to thank Huzaifa Sidhpurwala from RedHat Security here for his continued interactions with project maintainers on our behalf, passing along reports and being an all-around helpful guy. The general progress in fixing bugs in both projects can be observed in respective upstream git browers: http://cgit.freedesktop.org/poppler/poppler and http://git.ghostscript.com/?p=mupdf.git;a=summary. We estimate that it might take another few weeks or months until the libraries reach a state where no low-hanging fruit vulnerabilities can be easily fuzzed out. Until then, we recommend any potential direct or software wrapped use of the software (such as Okular) be done with extreme care where sensitive data is at risk, either by running them in a properly sandboxed environment, or only against fully trusted input data.

Following the release of Adobe Reader 9.5.2 and 10.1.4 and publication of our previous post, we have continued working closely with Adobe and specifically their Product Security Incident Response Team (PSIRT) to ensure 16 pending crashes from previous iteration and new crashes from further iterations can be resolved as soon as possible. In late August, we created a new set of PDF files in the hopes of getting better coverage. There are many features in the standard that are only fully functional in Reader, such as 3D models or certain Javascript features amongst many others, and we wanted to fuzz test these. Generating the corpus took enormous amounts of resources and time; terabytes of public documents has to be run through extremely slow dynamic binary instrumentation-driven parsing. Having a new, greatly improved, and several times larger set of input files at our disposal, we tested its capabilities by running it through the very same algorithms that we used with the previous files, and the results exceeded our wildest expectations: we managed to trigger exactly 568 crashes with unique stack traces during less than a week of running the fuzzing engines (Note: the number of unique stack traces is usually much higher than the number of actual bugs).

All reproducing PDF files were provided to Adobe in a single report on August 27, 2012. The vendor immediately acknowledged reception of the files and sorted through them as soon as possible. Adobe’s PSIRT sent us regular updates regarding their progress on resolving the reported issues, but said that updates for Windows / Mac OS X platforms would not occur until January 2013. Although we suspected that many of the reported issues could represent critical vulnerabilities in the software and the proposed timeline was far beyond our regular 60-day policy, we refrained from making public announcements until today.

In the meanwhile, on October 15, Adobe released a completely new version of Reader marked as “XI”. Although no public security bulletin coincided with this new version’s release, the company incorporated ~50 new security fixes derived from our previous reports – fixes that would only be included in the latest application branch and not found in either the currently supported branches 9 or X. If we look back at the situation at the time, it could be summarized in the following three points:

• Adobe Reader 9 and X for Windows and Mac OS X were subject to ~16 old vulnerabilities from June 2012 which were missed in the August 2012 update.

• Adobe Reader 9 for Linux was vulnerable to all ~50 security bugs from the first iteration of testing.

• Releasing Adobe Reader XI with new patches resulted in the potential disclosure of ~50 new vulnerabilities from the second iteration of testing in both Adobe Reader 9 and X for all platforms.

The above scenario is a good illustration of what happens if certain supported versions of software are provided with security patches, while others are not. In the era of wide availability of professional binary diffing tools, we find it very possible that many of the bugs fixed on the Windows / Mac OS X platforms but not on Linux and/or in version XI but not 9/X could have been successfully found by third-parties by performing comparison between the patched and unpatched versions of software. In fact, this practise was the sole subject of Nikita Tarakanov’s ZeroNights 2012 presentation called “The Art of Binary Diffing or how to find 0-dayz for free”, which only shows that provoking such situations of security inconsistency between equally-supported versions of software can pose a real threat these days.

As of now, the overall security of the Adobe Reader software family has greatly improved compared to 2012. Today’s security bulletin addresses around 80 bugs in Adobe Reader 9 and X, and up to 19 unique bugs in Adobe Reader XI. According to Adobe, only 19 of these were of critical severity and thus the bulletin contains that many CVE identifiers for the most severe problems. Unfortunately, resolving some of the problems have still been deferred to next versions: 20 bugs are still pending in Reader 9, 14 in Reader X and 9 in Reader XI. It should be noted that these deferred bugs have all been investigated by Adobe and either partially fixed (up to a point where they are no longer exploitable) or considered non-exploitable (e.g. infinite recursions).

All in all, we think that Adobe did a lot of solid work in terms of dealing with our reports, fixing the bugs, and providing timely updates. On the other hand, we believe that they could still do better in how security updates are released for the various versions of their products. We are hoping that releasing a collective update for all Reader products today will become a standard followed by the vendor in the future.

Of course, we are planning to run more fuzz testing against the latest Reader (and other software) within the next few days, so you can probably expect more news on the PDF security front to appear on our blogs this quarter :-)

Timeline

• 14th of August 2012: new version for Windows and OS X released, old post is published.
• 15-26th of August 2012: Adobe Reader PDF corpus is created and used in fuzzing.
• 27th of August 2012: a total of 568 crashes are reported to Adobe.
• 27th of August 2012: vendor confirms reception.
• 7th of September 2012: vendor informs about 50 unfixed unique bugs known so far, pushes Reader for Linux release until January.
• 12nd of October 2012: vendor informs about 61 unfixed unique bugs known so far and Reader XI release in mid-November.
• 21st of December 2012: vendor gets back to us with a complete summary of fixes released in January.
• 8th of January 2013: new Adobe Reader versions for all platforms are released, this post is published.