How RLC-Hardened brings active defense to the operating system

Uh and so I'm just I'm just going to let you run the whole thing. So just introduce yourself and then cover the topic. >> Brady Dibble, director of product for Linux at CIQ. I can't speak uh with the eloquence that Brian Dawson would have, but I can definitely shed some light on components of our Linux product, Rocky Linux from CIQ hardened and a lot of the fundamentals uh that RLCH builds upon. Uh definitely some of the harden packages, FIPS, STIG, and many of those acronyms you mentioned are things that we deal with on a day-to-day basis. So, looking forward to the conversation. >> Awesome. Well, thank you Brady for joining us.

And last but not least, uh our our technical resident expert here, Sultan. Would you uh would you like to introduce yourself? >> Hi everybody. I'm Sultan Alof. I'm CIQ's distinguished Linux kernel engineer. I've got over a decade of experience working on Linux kernel internals with my background mainly centered on Android and embedded systems, but I've also got uh a lot of experience on stuff running on large production systems like servers and stuff. I've I've been at CIQ for a year now and I spent a lot of my time working on um some security security related stuff we're going to talk about later in this webinar and um just improving the kernel in general at CIQ.

>> Thanks. >> Awesome. >> Well, thank you all three of you for joining me. This this is this is an important topic. Uh, and I I want to borrow this from Brian, but when we're looking at security, when we're talking about the Linux operating system, when we talk about security, there's there's a couple of different camps, and we'll we'll dive into both of these uh as we dive into today's topic, but there's proactive security and there's reactive security. We're all very very familiar with reactive security. CDE comes out, it gets a patch, there's a new version of your RPM, you run the update, uh or maybe you disable a service, uh remove a library, all that kind of stuff.

Um that's reactive security, and we're all very familiar with that. The problem is how do you get ahead of these problems? Um and so Brian, uh as we were prepping for the for today's session, Brian shared a a very wonderful metaphor of you know, how do you build Linux? how do you build your Linux implementation in such a way that it it's more secure to begin with? And he likes to uh he he likes to liken it to building a house. The best way to the most secure way to build a house is to build it without any doors and without any windows. The problem is that becomes unusable because either you can't get into the house when it's done or you can't ever get out of the house because it's there's no ingress or egress.

So much like much like a Linux server that doesn't work because apparently people don't install the operating system just to install Linux. I mean I do all the time but apparently people like to have workloads, databases, websites, whatnot. >> Crazy that you run OS for the OS, >> right? I mean that's that's my opinion, but you know, hey, what do I know? Um so let let me open this up to the floor here. Um over the past few years we've seen an exponential increase in the number of kernel CVEs. Let's not worry about applications. Let's not worry about uh user space. Let's just look at the kernel for a minute.

So what what does that increase in velocity mean for security teams trying to maintain patch cycles. >> So let's talk a little bit about that just for a second to think about. So what happened is a few years ago um the colonel decided that they wanted to become the CNA uh which is the CVE register for all of the colonel CVEes. And so what they started doing is analyzing each of the commits that went into it and deciding whether they were security relevant or not and assigning CVE associated with them. And so um that is where um the the increase kind of comes from and so Linux vendors and Linux OS vendors have had to deal with this.

And what it means is there's an increase of CVEEs that they have to worry about for patching on it. Now have been parting of a Linux OS vendor for a long period of time. I understand the stress that comes as you kind of deal um with um to deal with all of these security vulnerabilities and make sure that all of the packages that you ship are the have the vulnerability solved as it comes for for the customers on the back end of it. It's not only is OS vendors spending more time patching, testing, making sure these products grow out. um all the clients and all the customers of all these OS vendors have to keep up with that patching.

Now, depending on what industry you're in, depending on what your patch um policy is for a given co company on it, oftent times this means you're patching multiple times a week inside of the OS, whether it's a critical security vulnerability that comes up, whether it just happens that you're rotating through all your host and you just have to patch. And so it almost becomes a full-time job for patching and keeping on top of it and becomes more than one person on it. And just like um as the number of servers grows and the number of concerns kind of grow, you just constantly have to be on top of it.

Um, and so it it causes a lot of stress and money and uh manpower to keep this up to date. No matter what industry you are, especially as we're growing compute power and we're growing the number of servers and increasing the business, you just are increasing the load that it requires for you to patch those servers. >> Not to mention, working on the kernel is typically not easy or not a very common area of expertise for a lot of engineers. Uh, and so before we dive into that, I I uh I neglected my duties as host. I'd jump right into the introductions. But uh hello to all of you watching us live.

This is actually a live event. So anything we say or do can eventually be used against us. Uh and I say that because uh there is a chat function in the uh in the interface. Feel free to throw your questions or comments in there and we'll do our best to address those live on the air. Uh so as we get into this topic if you have some thoughts or if you have something you would like to ask my panel of brilliant people uh except we just lost uh just lost Nate. So welcome to our world today. This has been fun but we're just going to roll with it.

Uh so so Brady, do you have any any thoughts to to add on to Colonel CVES? >> Sure. U as to your house metaphor I mean it's very intentional that enterprise Linux and Linux in general is there with the doors open. It's intended to be do DIY and even a distribution rather than doing you know Linux from scratch which is the ideal way or Arch which is of course the preferred way for an enterprise even the enterprise optimized uh builds distributions they still have to be flexible because they don't know what your use case is. So all those doors are open from day one intentionally. Uh it's intentionally not overly opinionated.

Um at that point, you know, you're having to do a lot of work to bring that generic build up to your particular compliance needs. And that's really what it's about is you have to bring it up to compliance code because everybody has some compliance threshold or some CISO they're answering to to ensure that that Linux deployment is checking off the boxes, crossing the tees, dotting the eyes, and that when you go to audit um you're going to pass. Uh but really mission trumps all with a lot of these things. It has to function. Um, I think we we've joked before that the safest Linux server is one that's in a concrete box, not connected to the internet on the bottom of the ocean.

Aka the most secure distribution uh is one that's not usable. Usability has to be the function because if it's not usable, well then what's the point? Um, that usability can leave some of those doors open. Uh, there is no such thing as a secure deployment. There is a more secure deployment. Um and when mission is the priority and then compliance has to be met, security can kind of fall behind. For example, with CVE patching, uh it can be 50 250 days, quarters before a particular CVE gets patched. That's just the the working reality because even though those patches are coming out very quickly, there are so many steps you have to go through before you can make those updates.

Uh package versions have to be reviewed depending on your process. You've got to go through a life cycle process and make sure they all work together for your particular stack. If it's the kernel, even if it's a very critical CVE, can you afford to reboot? Well, well, we think we can get to it next week, maybe next week. Hey, there's some customers using it. We got to do node drain. So, there's this constant fight between keeping the mission going, uh, meeting compliance thresholds, which are you're getting constant pressure to make sure all those check boxes are checked, and then security comes into play.

Um and you know it's really a problem because I think on 2023 there was about 97 uh unique zero days that were exploited like KV uh and there's the average enterprise has like 200 plus open source packages uh in their given deployment and that's like 50 CVES a day. So you're getting bombarded by all these CVS. With the advent of AI, as much fun as it is, it's not just for enterprise and business and chat GBT, it's also about how quickly can, you know, threat actors spin up dozens, hundreds, thousands of new attempts to find new exploits. And so, just as fast as uh AI and other tools are improving our ability to patch things, they're finding exploits faster.

So, reactive patching, you know, reactive patching just isn't enough. uh if it's especially when it's coming third against compliance and you know mission >> well and it's tough Brady as you start talking about it you talk about how the mission trumps right um it means that people sometimes are delayed in patching to be able to do it and we have seen some massive breaches and massive security issues that have happened over the last couple of years where big corporations have been down for days or you know uh information has been leaked on it to various different actors. And so there's a real cost to having your exploit being exploited, right?

And having a hacker kind of get into your system. It it is a very tough balance between do I patch now, do I reboot now, do I take the downtime? And it's tough. And like I feel for customers across the board. I'm glad I'm on the OS side where I'm just producing and delivering the patches and I don't have to make that tradeoff because it's so it's difficult these days to go like when do I patch, how often do I patch and you know how do I balance that risk between staying up and solving the mission and being more secure? So, how can we better position the operating system so that out of the box it meets certain criteria?

It's it's one thing to be up on patches, but that's not what a lot of uh systems administrators are facing. More and more we're seeing industries go to some sort of regulation, whether that's uh PCIDSS or whether that's HIPPA or uh FIPS. We're seeing more and more organizations requiring that as as just kind of a standard. So what is is there a way to become compliance so to speak on day one? >> It is it's a great question Eric and and it it drives a lot of the thinking that I've had over the last you know year and a half or so we've been thinking through it like STIG CIS these are all really great standards.

Um, now there are things inside of them that don't always make sense from a security standpoint, but they give you a good basis to provide a secure operating system, but it's all very reactive. you're building, especially when it starts talking about CVE and vulnerabilities, you're building your system in a way that you're providing config to provide the least amount of attack surface. So the attack surface that you have um is really tough to kind of protect because Stig and CIS just as Brady was talking about mission trumps, right? You have to make sure that it's operational, which means you still present an attack service. And so Stig is great.

I love stick. I love CIS. I love the standards of trying to make sure that you have a minimal attack surface, that you have the right configuration to make your operating system as secure as possible. But in the world today, it's not quite enough. And so we t you talked early on about the difference between proactive and reactive security. And as you do that, um, you've got to think about the operating system a little differently. And so this is what we did in terms of how what goes into the thinking of RLCH. We thought about what are some of those things that we can proactively prevent um vulnerabilities from happening.

And we've seen this across the industry. The industry has been moving this way in certain things on it. One of the things that been standard for about 10 or 15 years is inside of GCC most operating system vendors will build with stack perfection protection on and this provides protection when a stack gets overrun and other stuff that prevents uh you being able to run code and it provides crashes on it. And so as we were thinking through this and our our talented um solar designer who works with us and has built uh a lot of the tools or worked with a lot of the tools in Rocky Linux hardened um has provided a way so that we're focused a little bit more on pro proactive security.

How do we shut down certain classes of vulnerabilities before they can interact in your system? or if we can't shut them down, how do we limit the blast radius and limit the exposure for it? And that's really what we're trying to pitch as we start talking about our LCH and Rocky Linux hardened. Do you want me to go into details of what's all included or or or Brady, do you want to give some of the details there? >> Yeah, let's see. Yeah, we at CQ have seen this pattern repeated um over and over with the similar customers, similar situations, very different customers, very similar situations. It all comes down to the same patterns of here are the windows I need to lock, the doors I need to close, the boards I need to put up to protect my environment.

And that's why RLCH is intentionally opinionated. It's making trade-offs for not just compliance, but keeping you functional while adding security. And that security is not just the hardening. When people say hardening, oh, this is hardened. Well, okay, I'm I have hardened the configurations so that now I am STIG compliant and I run open scap and I have my percentage and great, I'm safe, right? Well, to Nathan's point, it goes much deeper than that because the the guidelines, the security profiles are not designed to bring everything uh to the level of security that thread actors are actually targeting. It can't keep pace and it also can't lock down everything because it can't consider every possible scenario and implementation that a customer might have.

So there are still doors and windows that are open that are not closed by stig that are not closed by PCIDSS and that's where we move from to proactive hardening actual code level changes that shut those windows shut those doors. Uh there are configurations for example um I think about what 60 70% of high uh severity vulnerabilities come from memory corruption. That's why some of the things we're doing are hardening gibb c uh providing hardened malo and those this is attacking entire you know cws common weakness and uh enumerations and to Nathan's point if not outright mitig uh you know eliminating those CWEs it is at minimum mitigating them significantly to buy customers time there are about uh seven packages that are um added or modified in RLC Harden Um as mentioned we have a hardened build of gibbc.

Uh there's hardened open SSH turns off a number of services which are not commonly used. Um there is changes to password encryption uh and password security. Yes scrypt and password QC. Uh uh in addition to that um we you know we've eliminated some uh well there is tuning of all of this of course a tool called control uh which gives you the ability to ensure that your settings don't get lost upon reboot and to tune um some of the levels of security because there are trade-offs as well. How secure do you want to be? the more secure you are, uh, again, there's a trade-off. You're there are potential, uh, impacts to performance, and you're turning off things that you may didn't have not known that you were using or some of your users were using.

So, there is dialing back and forth to get it right. U, but control allows you to keep those settings um, and deploy those settings across multiple nodes. The LKRG, however, that's the real star of the show. Um, I think Sultan probably is best uh suited to talk about how cool LKRG is. >> Yeah, I don't know, Eric, if we want to jump into that just yet. I I can talk if you'd like. >> I think uh I think Brady set you up really well to introduce what LKRG is. So, let's let's dive into that. >> Okay. LKRG is a Linux kernel module that does proactive security.

So instead of doing something that makes the operating system hardened in advance like when a program is compiled, it's actually checking stuff in the operating system at runtime inside the Linux kernel to protect the Linux kernel from exploitation. And there is nothing else of its kind that does this proactively. So it will it will run in the Linux kernel and check what's going on in the Linux kernel to see if there are signs of an exploit taking place. And when it detects that, it will shut it down immediately. >> And I think something that might help people understand what LKRG is is to also decide also define what it is not.

this. We're not talking about antivirus and we're not talking about a rootkit detector here in a in a pure form, but we're talking much much deeper down the technology stack. Is that right? >> Oh, yeah. This this goes a lot deeper. And pretty much the the way that it works is that it looks for the pot of gold at the end of the rainbow that all exploits tried to get to in their long complicated chain of operations. What do you want to get at the end of the day? For example, with a local privilege escalation, you want to get root access. That's what LKRG looks for.

It doesn't look for the complicated chain of what are you doing? It looks for did someone get to that pot of gold at the end of the rainbow. So, it doesn't matter how someone tried to exploit the system. It looks for what the goal is when those exploits take place and it detects those irregularities and uses that to shut down exploits regardless of how the exploit take like takes place. >> Yeah. I mean, one of >> Okay, go ahead. Go ahead, Nick. Sorry. >> One of the things to think about it and and I'm not as deep down the stack as Sultan. um he still runs circles around me from a technical set.

But the way I think about it is like LKRG takes a snapshot of the picture that should exist inside of the kernel and then detects when it's changed. And when there's valid changes, it updates its picture. When there's invalid changes, it flags it. it and it will either report on there's a bad actor, it will crash the given pro process or just block it from accessing whatever it needs to do. So that picture that is supposed to be the pot of gold at the end, the the the right set on it is always correct and it knows when it's correct and knows when somebody's trying to tickle it and change it and throw in the false gold rather than, you know, the real aspect of it.

At least that's how I understand um LKRG. Sultan, correct me if I'm wrong, but that's my >> that's that's exactly what it does. it it looks for what's supposed to happen normally and correctly and if it sees something that happened without the correct steps happening before that it'll say oh no no that's not right. It's like it finds someone in your house who didn't enter through the front door and the only thing that is normal is for someone to enter through the front door. If there's someone in your house and you know they didn't enter through the front door, they're not supposed to be there. That's what LPRG does.

>> And uh we we we come equipped today, don't we? We've got a uh we've got a live well, not a live demo, we have a pre-recorded demo uh that we we put together. Um so, let me pull that up real quick. So, hold on. You ready to to kind of talk through that? >> Sure. >> Awesome. All right. Well, here here goes our uh here goes our LKRG demo. So this references the the attack vector used by a specific CVE, but it's used by a ton of local privilege escalation CVEEs related to the kernel. It's talking about the pot of gold at the end of the rainbow, the commit creds, where the exploit goes for that function inside of the kernel to change the credentials of a process or thread without going through the front door of the house.

And before without LKRG, you can see that it gains root access just fine. UID0 root, it got it. It's in. There was nothing that stopped it from taking place. And then now with uh LQRG loaded, the exploit is going to be attempted again. And we will see what happens. And oh, it got killed. What did LKRG say? LKRG spotted that something took place that was not supposed to happen. The credentials for that task were changed without going through the proper channels to do it. It saw someone claiming to have root without having gone through the front door as expected to get root. And what LKRG does that is is most powerful is that it doesn't it doesn't stop a task or a thread or a process from changing its credentials on the fly.

It stops it when it attempts to use those credentials. it it stops it from trying to use the fake driver's license at the store when they try to buy something instead of the moment they create the fake driver's license. And when it sees that they're trying to use that, it kills the process immediately. And the the program no longer executes any more code in user space. It is killed without having any more chance to even react to LKRG taking an action. Now, some of the cool features about LKRG and and I'm going to put Salt on and give him some credit here in a second um is that like uh LKRG a couple months ago went 1.0 And one of the big aspects of that is operationalizing it and making it ready for production environments which is wonderful.

Um and we we celebrate that it's in Rocky Linux hardened. Um but I want to talk about what's going on right now. Um in LKRG a little bit there is uh um some press you can go out and find it that basically said they can get around LKRG. um Ensultan and Solar and others have been working on how do we improve LKRG not to just block this workaround that they're doing but how do we detect when people are doing similar things on it and they're building additional features into LKRG that will show up in Rocky Linux hard and soon on it and it will provide just a better net around the kernel around that pot of gold to detect when there's problems and people are doing it and detecting certain things and making it happen.

And so it's it it's awesome as you know a director here as as the head of Linux engineering to see like some of the technical improvements that are going and that it's not just one and done like we're continuing to try and expand the scope of where proactive security happens inside of Rocky Linux hardend. Um, so stay tuned as we kind of continue to make more and more changes to to LKRG and have it protect much more of your system and and different aspects of vulnerabilities. >> So, one of the questions that always comes up in security conversations and I've I've seen it already in the chat is, okay, so this is running on my system.

What kind of performance impact can I expect? >> That's a great question, Eric. And I'm sure many people are scarred by the the days of anti virus on Windows and their computer slowing down to a crawl when it tries to do something. LKRG is not like that luckily. Um the performance impact is about two and a half% overall and that is not even factoring in additional performance improvements that we at CIQ have made and contributed to LKRG as an open source project. So even even before those improvements, the overall performance impact is just two and a half percent. >> And part of that let's let's put this in practice, right?

If you look at the standard usage of cross systems, right? You're using about 50 or 60% on a really busy system. So is it going to cause additional time to complete um certain actions? Yes, but it will be minimal. I mean, depending on how often your tasks are run, will it use a little bit more CPU? Yes. Will it use a little bit more memory? Yes. But usually you have those buffered in your system anyways. Um, and it's not like it's 10% where you're seeing a noticeable slowdown. Most people won't notice that that impact. There will be some and every workload's a little different. And so depending on your workload, you'll see an average different.

Um, one of the things I've been in the industry a long time, especially on on operating systems, and everybody's all like, "Oh, benchmark it for me. What's the benchmark? What am I going to see on performance?" And it all depends on your workload. Um, now the two and a half% that we have measured is based off of a synthetic workload, based off of something that we actually could measure the performance with on it. Um, and we think that's about where the maximum uh percent is that you're going to kind of see. It might be less, it might be a little bit more. Um, but it it's probably not going to be as noticeable depending on what workload you're running and where you're running it.

>> And so a followup to my earlier question, uh, Mark in the chat actually brings up a really good point. Uh, with LKRG, is there a danger of false positives, i.e. blocking legitimate usage? >> LKRG strives very hard to not generate false positives. At present, I am unaware of any false positives that take place within LKRG. And the only false positive that I have seen in my months of uh working on it was actually not even an issue because it would it would show up when uh a process was dying. Anyway, you would see a message from LKRG saying, "Oh no, this is this is someone trying to do something nasty, but it was a process that was already on its way out the door." LKRG did not do anything differently that impacted how the system operated.

So are there are there false positives? If there are, I don't know of any. And I I don't believe Solar knows of any either. So it is um it is very much something that is a a big goal of LKRG is to not generate false positives. >> And so along that there used to be false positives in LKRG. So you can go out on and and look through forums and other sort of stuff and you'll see some um and Solar Sultan and uh um Adam who is one of the other uh developers on it that's not at CIQ um on LKRG have worked hard to try and reduce those false positives and we'll continue to do so.

So if you find some report it and we can go fix it. CIQ is very much committed to the success of LKRG and all of the open source work that goes into it. It's the the first and only company that has basically sponsored development on LKRG that I know of and we're the only ones shipping it in a product. So, we stand by it >> which which is great. And going back to the compliance question that you asked earlier, one of the big things in compliance today is secure boot and making sure that you're able to do secure boot, which makes it really hard when you deal with these third-party um kods for it because secure boot requires that a a kod be signed.

And this is one of the critical things about rocky Linux hardened is it's the only operating system where LKRG is signed by the operating systems secure boot key. So it auto gets load you can load it in secure boot you can run it in secure boot mode with LKRG and it will work just fine. >> Awesome. So one thing I'm really excited about with with RLCH and that's Rocky Linux from CIQ hardened um for for those playing uh acronym bingo at home uh is that LKRG is just one part of that. It's just one aspect of RLCH. Um, so I I want to zoom out a little bit and kind of look at what are what are some of the other features that come included with RLCH uh like LKRG.

>> So Brady mentioned a bunch of these uh before on it, but let me walk through just a little bit and and give you a little bit more details on it. Um so hard and malo is one of the other big preventative measure measures and what harden malo is is it creates buffers around memory allocations and it creates um detection pages so that when something gets written outside of the boundary of the buffer it gets detected and handled on the kernel and usually causes a crash of a program. So buffer um uh buffer over overflows overruns no longer cause um code execution or potential code execution or finding you know sensitive information in the memory but just get stopped um early on in the process.

Uh we add additional password security to it. So Yesry is an encryption algorithm um that is uh changes the SHA 512 on it. So it adds memory time complexity as well as um CPU compute complexity on it which helps protect against AS6 coming and trying to crack passwords on it. It uses password QC that determines the strength of the password um and makes sure that you have not just a password that is cryptographically sound but also um takes in factors like entropy and other things into the complexity of the password to make sure that systems or that people use complex password that are harder to guess and harder to compute.

um when it comes to using things like John the Ripper or other things where you're trying to crack password um in addition to some of those on it we do some surface reduction on it. So open SSH um as Brady mentioned reduces the number of libraries linked to it. One of the cool features of it is because of how open SSH is written on it is that uh there was a exploit for open or XZ um a year or two ago on it that affected a lot of open SSH clients because open SSH links against a program that links against XZ. So it gets loaded as part of the open SSH program and libraries associated with it.

um with the open SSH version in Rocky Linux uh hardened uh it only loads XZ and the the requirement for it for a short period of time during startup and then removes it from the linking. So it's no longer in memory and so the exploit doesn't have an opportunity to use it to exploit into open SSH. We do the same sort of thing with some of uh Gibbc hardening where we break down and and try and stop um your uh environment variables from crossing security boundaries. It's not perfect um but it does reduce a lot of of the scheduling on it. Now those are some of the active and proactive features.

The other aspect that comes with RLCH is the ability to do some of those hardening foundations that talked about earlier. We have stick versions. We have CIS versions and we have scripts to be able to take uh and validate that you are stick compliant and CIS compliant up to the point that an OS can do. There's a number of things inside of CIS and STIG that you have to do about the client machine on it. So you can't get 100% compliance out of the OS. Um but we get as close as we can with as much as we can to be able to make that happen.

So whether you find yourself in a in a government environment, in a you know adjacent environment, dealing with financial services, whatever the case may be, you can come out with some of these established standards and say at install I am 95% of the way there. Um and I I I h I can show it through um hostap um the the host cap reports um to be able to prove that it's there and everything like that. And 95% that's not just a wild number. That's the actual number that the gold image is out of the box. >> Yes, >> that's really cool. 95% of the way there uh on on day zero really.

Uh that's really cool to hear. Uh so we we've had a few questions come up in the audience and I've been doing my best to kind of weave them into the conversation. Um, but I want to address a few of these specifically. Uh, so I'll open open these up to the whole group. Um, question we had, is there an SBOM software bill of materials for RLCH packages? >> Go ahead. >> Yes, every image uh that is built has an SBOM created. >> Awesome. Uh and then um so one of the questions uh that might merit some conversation is are the changes we're making at the kernel level in the kernel code itself or is this something that we're adding on to it?

The the heart of the question was is this being released by kernel.org >> you want to talk about kernel? >> Yeah, it's um it's not part of something from kernel.org. It is an external module that is loaded into the kernel and uh it is its own project um unaffiliated with kernel.org. >> And I I think it's important to point out that while LKRG is not part of the kernel itself, LKRG is open source. >> It it is open source and you can take LKRG and you go build it for any of the kernels and and the team the open source team checks compatibility of new kernels from kernel.org.

Um, Rocky Linux hardened is based off of Enterprise Linux and so it uses the Enterprise Linux kernel. Um, and so when you get Rocky Linux hardened, you get a kernel that has been through all of the Enterprise Linux process for it. Um, and it's the same kernel that you would use if you were to take Rocky Linux 9, for example, um, and install it yourself. You get that same kernel with the LKRG added into it, um, as a KOD, as an external KOD to be able to do it. So all the applications where you rely on certain levels of the kernel and you know that they work with Rocky Linux 9 should work with Rocky Linux hardened.

Now I'll I'll always do the caveat of you always want to test all of your applications on it because while I use the word should there are some applications that may you know cause some issues as we talked about before. We're not aware of any that would cause issues but you always want to test before you put something into production. test before production. That's that would have saved the uh the sanity of a great number of systems administrators over over the years. Um so turning our attention a little bit to FIPS uh question from the audience was is Rocky Linux 9 FIPS compliant? >> Great question.

So Rocky Linux the community edition itself is not uh the every month every day NIST the organization behind FIPS uh compliance uh raises the bar essentially. So Rocky Linux as it stands today being a rebuild of RE some could argue that there is some inheritance of whatever RE's um FIPS compliance is but the code delta ages quickly and every time you go to a new minor version the delta becomes effectively too wide to even be considered compliant if you look at it from the side or at an angle. And generally when you're looking at a FIPS compliant or certified module uh you need to stay on that module but you also need to get CVE patches.

So for Rocky Linux from CIQ we pursued FIPS certification completely independent of re and from Rocky Linux. So we have pursued FIPS compliance uh certification for Rocky Linux from CIQ8 which covers both 8 and 8.6 versions. Uh we also have pursued 9.2 and 9.6. 6 and we will continue to pursue FIP certification for every 2.6 and10 version. This covers the five main packages uh that is kernel open SSL lib crypts and NSS which are used in uh enterprise Linux for the vast majority of your encryption. We have our CMVP meaning the packages are actually certified for FIPS 140-3 uh for off top of my head I think ligrypt uh and our 8 and 9.2 2 kernels and the rest are coming in the very near future for uh complete coverage of RLC 8, 8.6 and 9.2.

We've also got our 9.6 packages in the MIP list which means they've been approved by the lab and it's just waiting for NIST and the government to look at the paperwork and give it the stamp. So these are we had to do more. However, um when we went through the certification, uh the bar had been raised um because the labs learned more about what the requirements are. So there are multiple code level changes that CIQ had to make to bring these packages up to the bar for FIPS compliance. And that as a result, we don't have any uh caveats associated with our packages that maybe were there during previous packages that went through this process.

Uh and then you know as a result um all of our stuff I mentioning that we're a very open source first company. All of these changes are available through our GitHub and we encourage you to go take a look if you have any questions. Uh but the actual packages the versions that were certified by the lab, you know, approved by the lab and have been certified by NIST are all available through RLCH. >> Beautifully put. I love it. Uh and then uh we're we're kind of coming to the end of of our time together. So I've got one last question from the audience that I wanted to uh bring to the surface and that is how does RLCH impact traditional HPC or high performance computing systems?

Well, I mean it depends on what your implementation is. Um RLCH out of the box is going to be somewhat agnostic uh to RLC, you know, HPC systems. you bring in your open HBC and uh but otherwise there are certain nuances that you need to take into account for like a werewolf uh deployment where a werewolf is has some conflicts um with you know stig and certain configurations. So for example the gold image for rlc that's based on stig is is not a werewolf gold image.

Um we do have uh other webinars we can go through from our webinar uh the werewolf pro product which takes these things into account but um you know for HPC deployments uh for ephemeral loads um you know deployments you're going to have to make some additional tweaks on that gold image but you're already so you know close to where you need to get to that uh it does you know gets a lot of the work uh done upfront for you. >> This is why I love Brady on the call. See, I'm not an HPC guy at all, and I'm learning it the more I stay at CIQ, but having Brady, who just understands that world a lot better than I do, is wonderful.

>> I I could spell HPC at this point. So, I'm getting better. >> Well, I really want to thank you both. I get my uh get my numerical uh right. I want I want want to thank all three of you for joining me today. Brady especially uh for for jumping in last minute. And Sultan, you this this was your first webinar and uh I think you did a fantastic job. Really glad that you joined us. And you know this means that I'm going to drag you on this more often. >> Oh no. >> That's the problem. You show up, you do some you do a good job and then you get volunteered in the future.

So >> but uh that that pretty well uh covers the the questions and our topic for today. But that is not all. We are producing more and more content out on our YouTube channel. Uh this webinar and others like it are uh will will be available within a few hours uh if not a day or so of the live airing. So you can head over to the control IQ or CIQ YouTube channel and catch the replay of this webinar. Uh we're also starting to produce tech tips, uh thought leadership videos, uh a little bit of everything over on our YouTube channel. So, if you haven't dropped by there recently, make sure to uh make sure to drop a like for this content and subscribe.

Also, leave a um also leave a uh a comment in the comment section if uh if there's any topic that you'd like to hear. Um Brady, and is there anywhere you want to send folks, LinkedIn or or blog or anything? >> Please check out our LinkedIn profile. Um but also ciq.com has a whole list of blogs, webinars, white papers, references. There's some very interesting uh blogs in particular about RLCH. Check out the systemd core dump which shows how LKRG uh prevented um some exploits that were in enterprise Linux unpatched for a number of months and how it got up that that proactive hardening outright prevented it and mitigated it for months before it was actually patched.

>> Awesome. Um, and then, uh, so we we did have one last minute question come in. If you want to learn more about LKRG, you can go to lkrg.org. Uh, Nate, is there any any closing thought or anywhere you'd like to send folks to learn more? >> No, I think you guys have covered a a lot of it. Uh, I one of the things is it's we really try and take a lot of open source. Go look at the CIQ um GitHub page on it. We are putting more and more out there. Um, one of the things as the director of Linux engineering is I'm I'm always trying to look at ways of how do we get better and and release more into the open source world and more is coming.

I'm excited for some of the things that's on it that we weren't able to talk about that's on the road map for us to come. So, um, stay tuned. >> Well, Nate just volunteered to be on a future webinar as well. So, uh, Sultan, any any closing thoughts or anything, uh, anywhere you'd like to send folks? Uh if you'd like to to look at like what's been done on LKRG, you can go to GitHub and see the the commit history just to see like how much stuff I did, which was sponsored by CIQ. So you can see that CIQ has put a lot of work into LKRG.

It's not like we just made one or two changes and said, "Oh yeah, we're we're shipping LKRG to the world." No, it was um a lot of extensive work on it that that we've invested and continue to invest on it. >> Yeah. just the the most recent release actually added like 1,400 lines of code but removed like 2700. I may have those numbers transposed but there's there's more than uh there's more than 2,000 lines of code removed from LKRG before it went to version 1.0. So definitely check that out. Uh there'll be links in the show notes below. But until uh until next time, uh I didn't do my due diligence.

We're talking about uh RLC. We're talking about RLC again uh in two weeks. Uh I forget what the topic was off offhand, but we are talking about RLC. I'm doing great as a host today. So with that, I'm going to just end the call. We're done. But on behalf of of my guests today, Brady, Sultan, and Nathan, uh thank you all for joining us, and we look forward to seeing you again in a future webinar. Thank you so much, and we'll see you next time.