Why a 'frozen' distribution Linux kernel isn't the safest choice for security

“Born of code and git commits and CVE's combining. This frozen kernel has many bugs and a history worth data-mining."

We all know the kind of advertisement:

“These are the git mines. Working here we see the talented engineers of [INSERT YOUR DISTRO HERE] carefully selecting only the most polished and pristine open source patches from the raw upstream open source Linux kernel in order to create the secure distribution kernel you depend on in your business.”

It's a compelling story and on the surface makes a lot of sense. Carefully curated software patches applied to a known Linux kernel, frozen at a specific release, would obviously seem to be preferable to the random walk of an upstream open source Linux project. But is it true ? Is there data to support this ?

After a lot of hard work and data analysis by my CIQ kernel engineering colleagues Ronnie Sahlberg and Jonathan Maple, we finally have an answer to this question. It’s no. The data shows that “frozen” vendor Linux kernels, created by branching off a release point and then using a team of engineers to select specific patches to back-port to that branch, are buggier than the upstream “stable” Linux kernel created by Greg Kroah-Hartman.

How can this be? If you want the full details the link to the white paper is here.

But the results of the analysis couldn’t be clearer.

• A “frozen” vendor kernel is an insecure kernel. A vendor kernel released later in the release schedule is doubly so.
• The number of known bugs in a “frozen” vendor kernel grows over time. The growth in the number of bugs even accelerates over time.
• There are too many open bugs in these kernels for it to be feasible to analyze or even classify them.

There are still reasons you might still select a “frozen” vendor kernel. One of them being a vendor defined internal kernel application binary interface (ABI) that doesn’t change over the lifetime of the release. If you are using hardware where the device driver hasn’t (or won’t, due to the attitude of the manufacturer) been submitted to the upstream Linux code tree then you may have no choice but to use a vendor kernel.

Having said that, the Linux kernel used by Android devices is based on the upstream kernel and also has a stable internal kernel ABI, so this isn’t an insurmountable problem.

However, it’s important to note that this only matters for internal kernel drivers. The user-space ABI used by all Linux applications is guaranteed to be compatible in later kernel versions with earlier versions. If that ever gets broken Linus Torvalds gets quite upset 😀.

This means that applications that work on a “frozen” vendor kernel will always keep working on the upstream “stable” kernel.

But thinking that you’re making a more secure choice by using a “frozen” vendor kernel isn’t a luxury we can still afford to believe. As Greg Kroah-Hartman explicitly said in his talk “Demystifying the Linux Kernel Security Process”:

“If you are not using the latest stable / longterm kernel, your system is insecure.”

“Let it go, let it go, Don't patch it up anymore.

Let it go, let it go, Use upstream and be secure!"