Much Ado About Nothing: A Response in Text and Code

[spender]

Last Friday at the HES conference in France, a presentation entitled "Stackjacking Your Way to grsecurity/PaX Bypass" was given. Soon after, an image was retweeted frequently on Twitter, supposedly of the presenters getting "root on a grsecurity/PaX kernel on-stage". No other details were provided to those retweeting the image (except for those in attendance at HES) and it was mentioned that the slides/code for the attack wouldn't be published until after a repeat presentation at Infiltrate in Miami this Sunday.

I have a number of issues with how this was handled, which I will elucidate here.

For starters, the PaX Team and myself got about 30 minutes advance notice of the slides for the presentation. Though this in itself would not be out of the ordinary for people we don't know at all, the presenters in this case have used my research in their own presentations and created patches "loosely based" on grsecurity features and submitted them to the Linux Kernel.

Up to the presentation (and even still on the Infiltrate website), the presentation was entitled "TBA Kernel Fun", setting itself apart from all other presentations as being completely unnecessarily secretive. The presentation either went through a secret approval process through the HES committee (of which I was a member) or underwent no form of committee approval with no submitted abstract. The reason for such unusual secrecy is puzzling.

We knew from seeing the slides 30 minutes before the presentation that the attack demonstrated was apparently against a "2.6.36.3-grsec" kernel. We didn't learn until after the presentation was given that the kernel was modified to add a fake arbitrary-write vulnerability. I understand the case where one would do this to demonstrate a technique alone without having to kill any valuable bugs, but as the presenters themselves brag at the beginning of the presentation, they've discovered plenty of bugs in the classes required for their attack, arbitrary-write and arbitrary-read or stack infoleak. Yet not only was a 6 month old kernel used (to take advantage of the only core-kernel stack infoleak published in the past year, out of the dozens applicable to vanilla Linux kernels), but the arbitrary-write bug had to be fabricated. This seems disingenuous, since at least from a reading of the slides themselves, as I have no knowledge of the way in which the presentation was given, the introductory slides set up a picture of how buggy the Linux kernel is (which is of course true) and mentions how prevalent the infoleaks, etc are, creating the suggestion that the conditions for a "bypass" of a grsecurity kernel are anything but rare. The slides don't mention at all how specific features of grsecurity can drastically reduce the attack surface of the kernel and limit the possible damage from rarely-used modules and certain classes of vulnerabilities.

I object to the use of "bypass" when referring to a security system with dozens of features. An example of a bypass would be the various mmap_min_addr bypasses that existed: in these cases it was always possible to achieve the same goal and abuse the same vulnerabilities. In the case of this presentation, what is being bypassed exactly? The features with defined protections worked as intended, preventing them from abusing certain vulnerabilities that would have been usable on vanilla kernels. The exploit was done within a known attack space mentioned specifically in the PaX documentation (nothing can currently stop an arbitrary read+write). Because it's very difficult to do anything about this class (especially when still meeting usability and performance requirements) I've been focusing on, and I believe have been successful at, reducing the reachability of vulns in general (including those of this class) and in reducing their scope (turning a linear overflow into a write bounded by the heap object itself, for instance). Bugs of this class are getting more rare as they're eventually weeded out. Arbitrary read/write bugs often arise out of the non-existence of bounds/sanity checking, which in many cases are easier to spot than the more prevalent bugs of some form of read/write with added constraints, due to the deceiving nature of any existing (flawed) bounds/sanity checking.

So I'm left wondering why an attack with so little real-life application compared to the other things we deal with in grsecurity required so much hype-inducing secrecy. Did they know the technique could be killed and wanted to make sure they could give both presentations with a still-relevant attack?

For people I consider friends and colleagues, this was a slap in the face. I don't find it to be acceptable and have received no apology. So today, prior to their repeat presentation, I'm announcing the death of every single technique they presented at HES. I hope the message will be clear for others in the future that working with us will be much less painful than the alternative. It should also be clear to those interested in secrecy and hype-at-any-cost that a subsequent "reality adjustment" will be swift. This release time wasn't chosen intentionally: the PaX Team and I have been using all of our spare time since last Friday to write this up, test it, and port it to the kernels we support. In fact, we've taken great pains to make sure it was complete long before I return home on Sunday, ironically about 30 minutes before their repeat presentation would begin.

Enough talk. Here's what we've done:

Moved thread_info off the kernel stack completely for both i386 and amd64 -- it's now located in the task struct, which is located in its own slab cache.

Implemented PAX_RANDKSTACK for amd64 (without requiring MSR access, performance hit is the same as the i386 version since it calls the same function). This is applied per-syscall, making it essentially immune to infoleaks against the stack pointer itself, the same as on the i386 version.

Implemented additional protection in PAX_USERCOPY by identifying which slab caches required direct reads/writes from the userland accessor functions. Slab cache information is free within the current PAX_USERCOPY framework. We implemented a whitelist-based approach, and rewrote several areas of code to remove the need for direct reads/writes to certain structures so that the associated slab caches can be protected against direct reads/writes to/from userland. The technique allows for individual allocations within the whitelisted caches to be marked in the future, so that certain sensitive structures can be better protected. Of particular note is that task structs are protected under this mechanism.

Implemented active response against kernel exploitation. Attacks by unprivileged users will result in a permanent ban of the user (all processes killed, no new processes allowed) until system reboot. Attacks by root (either by real uid-0 or as fallout from buggy post-exploitation cleanup) or while in interrupt context result in a system panic.

Extended automatic bruteforce deterrence to apply to suid/sgid binaries -- detected PaX terminations or crashes will result in a 15 minute user ban (the amount of time intended to help offset entropy reduction attacks).

Improved MODHARDEN -- removed the fallback in netdev code to allow auto-loading any module when CAP_SYS_MODULE is present. The PaX Team came up with the beautiful idea, and I implemented a system by which we can control the entire asynchronous procedure of module auto-loading, along with correlating it with the original requester. Through this system, we can ensure (for instance) that mount -t can only cause filesystem modules to be loaded (verified through symbol inspection at load time). This system also has the side-effect of removing unnecessary reports for non-existent modules, as was the case in the previous system (since at request time we couldn't know whether the module existed or not). All these changes allow us to make stronger guarantees about the feature, regardless of any buggy privileged user code that performs certain actions on behalf of unprivileged users through dbus or the like.

TL;DR: Lack of coordination works both ways. Enjoy presenting a dead technique at Infiltrate; I hope the 15 minutes of fame from last week was worth it. If your path to infosec famedom involves screwing over friends for a free plane ride and hotel, you picked the wrong people. In case you'd like to continue this game, we're more than capable and willing to kill anything else you come up with Thanks for playing.

-Brad

[drosenbe]

Brad,

It saddens me that you felt the need to write this post in such an inflammatory manner, as I continue to have the utmost respect for your and the PaX Team's work.

First and foremost, I'd like to offer a public apology for the aspects of our presentation and its handling that led you to believe this was a personal attack - I certainly never intended it to be interpreted as such. As for the secrecy, the lack of advance coordination, and the (relatively minor) attention the talk has received, I have little to offer in the way of excuses except that you of all people should know how easy it is to get sucked up in the hype machine of the security circus. You have directly contributed to this hype by making public challenges about continued conference requirements for grsecurity/PaX exploitation talks and the lack of researcher response. That being said, all this was most likely unnecessary in this case, but hindsight is 20/20.

If the contents of our talk posed any immediate danger to users of grsecurity or represented a fundamental breach of a piece of implemented functionality, you would have been the first to hear about it. However, we have no illusions about the fact that our techniques do not pose this threat. Instead, we chose to present a study of how difficult it is to exploit a grsecurity kernel - the answer turns out to be "very". We chose to leverage basic exploitation primitives whose existence are admittedly infrequent but are certainly more common than most kernel vulnerabilities, which tend to be rather esoteric. Rather than being a cop-out, as you've suggested, this was done in an attempt to make our presentation more general, rather than a specific case study. We were explicitly clear in our presentation that the existence of an arbitrary write and a kernel stack leak were prerequisite assumptions for our attack, and I don't think anyone was deceived into thinking we were leveraging undisclosed vulnerabilities. Just in case, we'll be sure to clarify that our arbitrary write was artificially introduced during our talk tomorrow.

You seem to think that your introduction of new grsecurity features and strengthening of existing ones will somehow ruin our day - quite the contrary. I'm very pleased that these new improvements have been introduced, as they represent the continued evolution of defensive security in the face of new offensive threats (regardless of your opinion of the real-life utility of these threats). We'll be sure to mention these new features in Miami, and we're glad to have played at least a minor role in their creation - I only wish they had been presented in friendlier circumstances, and I again apologize for my role in the lack of communication that led to this disconnect.

I don't think there's any reason to feel as though our talk was a critique of grsec. The whole point is that grsecurity/PaX continues to represent the cutting edge in proactive security, and that many hoops need to be jumped through in order to leverage even powerful vulnerabilities into getting root. Keep up the great work, and I'm looking forward to continued improvements.

Regards,
Dan

[jono]

What Dan said++

[Dwokfur]

I dare to comment...
What capabilites a process should have to utilize the stackjack technique? I suspect an arbitrary process won't be able to start playing if RBAC is enabled.

[PaX Team]

What capabilites a process should have to utilize the stackjack technique? I suspect an arbitrary process won't be able to start playing if RBAC is enabled.

this is not about process capabilities but exploitable kernel bugs (one arbitrary write and one kernel stack leak). some bugs can be reached by privileged userland only, some are available to all, but all that is orthogonal to the presented exploit techniques.