New Research: The Red Hat Linux Kernel Model Is Broken and Can’t Be Fixed

The older a Linux vendor kernel gets, the more unfixed bugs and security vulnerabilities it is likely to have. New research by CIQ engineers Ronnie Sahlberg, Jonathan Maple, and Jeremy Allison shows definitive proof that bug and vulnerability fixes are not being back-ported into Red Hat Enterprise Linux (RHEL) kernels. As of May 2024, the RHEL kernels have the following bug counts for which there are known fixes upstream:

• RHEL 8.6: 5,034
• RHEL 8.7: 4,767
• RHEL 8.8: 4,594

The following graph shows bug counts increasing over time, despite the availability of fixes:

As upstream kernel development continues over time, the number of available-but-not-applied bug fixes increases substantially in vendor kernels

CIQ’s new white paper, Vendor Kernels, Bugs and Stability, dispels the belief that older kernels are more secure than newer ones. The research follows on the “open secret” that all kernel bugs are security bugs by way of Jonathan Corbet, Linux kernel developer:

“In the kernel, just about any bug, if you're clever enough, can be exploitable to compromise the system. The kernel is in a unique spot in the system ... it turns a lot of ordinary bugs into vulnerabilities.”

Older kernels are not only less secure, but potentially much less secure than companies may wish to believe. Aging kernel code becomes increasingly static, making it more vulnerable to attack. Furthermore, if a CVE has never been generated for a bug, it will likely never be fixed, meaning the vulnerability is likely to be permanently exploitable.

Some companies and organizations, like SUSE and AWS, do proactively manage kernel bugs over time by rebasing off a newer upstream kernel version mid major release. While not immune to this problem, they are doing the necessary and hard work of back-porting fixes for bugs and vulnerabilities for as long as their fresher base kernels are supported.

Why vendor kernels become less secure over time

The problems of aging kernels stem from a process that began nearly 25 years ago. Linux vendor kernels are created by taking snapshots of specific Linux kernel releases from The Linux Kernel Archives. The snapshot is associated with a git reference or git tag and bug fixes are back-ported from newer releases into the older releases. Furthermore, what’s back-ported is often feature- or enhancement-related to the point where the kernel no longer represents the original snapshot, which Red Hat explicitly calls out in their documentation.

Upstream kernel development often involves tens of thousands of code changes and fixes every year. The following chart shows the RHEL 8.8 kernel commits over time with a notable decrease in back-ported fixes beginning in November 2021, with a substantial decrease in November 2022. Kernel development does not slow down in November 2021 — but the back-ports for the RHEL 8.8 kernel do.

How companies can counteract aging kernel problems and vulnerabilities

Linux admins should no longer assume that upstream bugs are found and fixed quickly. In fact, that is far from the truth. As noted in the white paper:

“​​The Linux kernel CVE team recommends that you update to the latest stable kernel version for this, and many other bugfixes. Individual changes are never tested alone, but rather are part of a larger kernel release. Cherry-picking individual commits is not recommended or supported by the Linux kernel community at all.”

However, the data shows a large overlap between new CVEs and the increasing number of unresolved RHEL kernel bugs.

The authors of the white paper recommend that CVE-concerned companies “subscribe to and use a stable kernel instead of a vendor kernel,” assuming they do not need third-party supplied drivers. They see it as the only realistic way to ensure kernel security. An upstream stable kernel provides much greater protection from security vulnerabilities and general bugs in the kernel code.

For assistance with kernel security, management, and drivers, contact info@ciq.com.

Questions about the white paper and its findings can be sent to info@ciq.com.