A Deep Dive into Linux Kernel Runtime Guard (LKRG)

Security in Linux is typically centered around identifying and patching vulnerabilities, configuring firewalls, and managing user access. While these are crucial, there's another layer of defense that operates at the very core of your system: the Linux Kernel Runtime Guard (LKRG).

Unlike restrictive security measures that aim to prevent attacks from ever reaching the kernel, LKRG is a vigilant observer, continuously monitoring kernel integrity and behavior in real-time. It's not so much about preventing a potential attack, but rather about detecting and stopping a confirmed attack as it unfolds within the kernel.

The Linux kernel manages everything from your hardware to the applications you run. If an attacker gains control over the kernel, they effectively control the entire system. LKRG can identify and block this behavior. Specifically, LKRG performs:

• Runtime Integrity Checking: It constantly verifies critical data structures within the kernel. If these structures are altered in an unexpected way—a common tactic for attackers trying to achieve persistence and hide their presence or elevate their privileges—LKRG can detect it.
• Exploit Detection: LKRG identifies patterns of behavior that are characteristic of kernel exploits. This can include attempts to manipulate memory or redirect program execution flow in ways that indicate an attack is underway. For instance, an attacker might try to gain "root" access (the highest level of administrative privilege) by exploiting a flaw in the kernel. LKRG aims to catch these attempts as they happen.

The key distinction for LKRG is its post-detection mechanism. Many security tools are preventative, trying to block known threats. LKRG, however, focuses on discovering and stopping almost-successful intrusions at the kernel level. This makes it a valuable safety net, especially when new, unpatched vulnerabilities (often called "zero-days") are being exploited.

How it works deep dive

At its heart, LKRG is a loadable kernel module. This means it can be added to a running Linux system without requiring a kernel recompile or a system reboot, offering flexibility for deployment. Once loaded, LKRG snapshots the current state of the CPUs, the kernel, and some of its most critical data. Then it implements a series of checks and integrity validations designed to identify signs of compromise.

To validate integrity of the kernel, LKRG keeps track of:

• Critical CPU metadata, such as which CPU cores are “online” and that they have the expected set of security features such as SMEP and SMAP enabled at all times
• Critical kernel global variables, such as whether SELinux is “enabled” and “enforcing”
• Keyed cryptographic hashes of the kernel image and modules as they’re seen in memory

When legitimate changes are made through documented means - such as a CPU brought offline or back online, SELinux state changed, or a kernel module loaded or unloaded by legitimate root - LKRG transparently updates its snapshot of the state.

However, when unexpected changes are detected, LKRG’s default response is to log the event to a remote server (if configured) and panic the kernel. That’s because a milder response would generally be ineffective at stopping these attacks.

To validate integrity of the kernel’s view of the running tasks and detect exploits, LKRG keeps track of:

• Credentials, such as user and group IDs and container namespaces, of each process and thread
• Control flow, or rather kernel function call chains as seen from stack frames in critical places
• Use of “usermodehelper”, limiting it to running a handful of programs such as “modprobe”

Once again, when legitimate changes are made through documented means - such as a process having its user ID changed through the setuid(2) system call - LKRG transparently updates its copy of this data.

When unexpected changes to task credentials or to control flow, or misuse of “usermodehelper”, are detected, LKRG’s default response is to log the event (including to a remote server if configured) and kill the task.

LKRG’s behavior, including what it validates and its response to detected violations, is configurable via the “sysctl” interface and module parameters.

The optional remote logging is performed right from the kernel, with traffic asymmetrically encrypted to the log server’s public key. The log server daemon and related tools are provided along with LKRG.

Performance impact

LKRG checks the current state of its monitored code and data against its snapshots of the pristine state once in a while - at least every 15 seconds by default - or in case of task credentials, just before those would be used to grant access e.g. to a file. The relatively infrequent kernel hash checks are somewhat time-consuming, whereas the task credentials and control flow checks are relatively quick but frequent. LKRG also keeps track of legitimate changes. All of this adds up to some performance impact.

Numerous benchmarks were run to estimate LKRG’s performance impact, which was found to vary greatly between different workloads. Some are not impacted at all (such as compute-only userspace workloads), while others are impacted significantly. A certain run of the Phoronix Test Suite against LKRG 0.8 shows overall performance impact at 2.5%, as the geometric mean of 58 individual test results. Some extra optimizations contributed by CIQ to LKRG since then, towards the upcoming 1.0 release, may have reduced the performance impact slightly, as seen in individual tests we ran.

The value of LKRG

Why would you need LKRG when you already have firewalls and regular patching?

• A Crucial Layer of Defense: LKRG provides an essential additional layer of security. Even if an attacker manages to bypass your initial defenses, LKRG can detect their presence within the kernel, acting as a final tripwire.
• Protection Against Zero-Days: For vulnerabilities that are newly discovered and for which patches aren't yet available, LKRG offers a critical window of protection. By focusing on the behavior of an attack rather than just its signature, it can potentially detect exploits for vulnerabilities that no one even knows about yet.
• Increased Attacker Effort: While no security tool is foolproof, LKRG significantly raises the bar for attackers. Bypassing LKRG often requires more sophisticated and less reliable exploit techniques, making successful intrusions more difficult and costly for adversaries.
• Resilience for Challenging Environments: In environments where immediate patching or system reboots aren't always feasible, LKRG offers continuous monitoring and detection capabilities, providing some peace of mind while systems await updates.

In essence, LKRG isn't a silver bullet, but it is a powerful component in a comprehensive security strategy. By actively scrutinizing the Linux kernel during operation, it adds a robust layer of detection that helps safeguard your systems against even the most advanced threats.

RLC-H and LKRG

LKRG is an Open Source project developed independently from the Linux kernel (but in sync with it). As such, it may or may not be readily available in your favorite Linux distribution.

However, in Rocky Linux from CIQ - Hardened (RLC-H), it is not only available, but is enabled out of the box, and is supported by CIQ. Moreover, it is cryptographically signed to be part of the UEFI Secure Boot chain starting with keys pre-installed in off-the-shelf hardware.