Announcing Rocky Linux from CIQ - Hardened

Uh we are building upon all of these to create the harden Linux distra project here at CIQ. uh but uh uh we are moving forward. It will not be just reuse of what we had although we are reusing stuff obviously. uh but uh uh this is the modern age things have changed and uh also uh the distribution we are building is very different because uh uh here we are focusing on um creating the unique combination of uh AI stability uh and API obviously as well u uh uh that is customary for enterprise Linux distress and we are combining this with um fast a fast pace of uh security hardening enhancements during lifetime of uh this long-term supported distribution.

So uh to uh reiterate it's uh uh on one hand the stability as a as a stable uh software versions maintained over multiple years like it's it's expected for enterprise Linux distros and its compatibility with uh enterprise Linux standards but it's also um rapidly evolving uh security hardening and uh Oh, so so you're stealing my fire. I'm going to pause you there for a minute. So, I have some stuff, but thank you for the introduction. And I want to say um I can't ask you to brag about yourself, but I do want to call out to the audience here um that um you have for a long time not only been an expert in Linux security, but a thought leader in Linux security.

And as uh Solar said uh he created and founded the openwall Linux uh a security focused hardened Linux distribution or Linux project within that developed a number of modules one that is very known well known called John the Ripper. He's discovered some major vulnerabilities. he uh co-created and as a co-maintainer of uh the Linux kernel runtime card or runtime guard or LKRG which we'll talk a bit uh about more later. So without embarrassing you solar I do want to make sure that the audience understands um that you represent the type of deep security expertise that went into Rocky Linux from CIQ Harden u experience and knowledge but also uh you represent the Linux expertise that we have here at CIQ.

So um I'm going to move on and then Solar I'm going to want to get back to the detail that you were getting into shortly. Before I do, Brady, I want to go to you as our product guy um and ask you to explain uh what is RLC and and why was it created? Why do why do we need it? Absolutely. So, Rocky Linux is already great. Um especially with the end of life of CentOS and all the changes that have been happening in the Linux ecosystem. Um many C you know enterprises and customers have naturally landed on Rocky Linux or other EL distros as the kind of spiritual successor and technical successor to CentOS.

So Rocky Linux is awesome. We love it. CIQ has been a principal sponsor. the CEO of CIQ, Greg Kurtzer, was one of the core founders of uh Rocky Linux. So, we're deeply entrenched in Rocky Linux itself and we think it's a great distro that people should be using. Now, that is at the end of the day still a community project and it is a standard enterprise Linux distribution. Its goal is to be flexible. Its goal is to be powerful and stable and reliable and it's great out of the box. Some people, some customers and especially enterprises need more. Uh so the first thing that CIQ did was we launched Rocky Linux from CIQ.

It is Rocky Linux, but we added the commercialization around it. The accountability that you can really only get from having it hosted and delivered by a commercial organization, things like open source indemnification, the ability to communicate with someone directly um if you need faster support than just going straight to the community. So there's a lot of commercialization and support around RL Rocky Linux in RLC that you're already getting. However, it's still out of the, you know, offtheshelf general purpose enterprise Linux. Okay. What we see in kind of our vision for Linux is we believe the open the operating system can do so much more. Rather than leaning in towards standardize, standardize, standardize and one sizefits-all.

We see that there are many different use cases and workload specific optimizations that our customers absolutely need. Uh they're already taking Rocky Linux and then having to bring it in internally and do a considerable amount of maintenance and modification and optimization. Sometimes pulling a lot of open- source tools and kind of cram them all together and make them work themselves. And you know this is something that's there are tools out there. There's products out there but not everybody is a deep Linux expert. uh and not everybody or every enterprise wants to maintain um such a complicated amalgamation.

I understand I get I'm assuming lots of time sorry to interrupt I'll hand it back is uh either people aren't experts or you know they have other important things to do so if they could save time uh and and possibly the risk with manual hardening then why not right and sorry to interrupt uh back over to you Rocky Linux hardened is is our bring uh CIQ um looking at all the work we've done for some of our customers and the expertise the deep uh security expertise CIQ has internally and broader our opinion of what hardening looks like.

This is we know that with um with the the threat actors out there today and some of our customers who are constantly under attack that just a standard hardening you're going to get out of the box is not enough and that we believe that there is much more that can be done to get out ahead of this standard uh standard hardening standard security and start actively preventing attacks. Uh, and that's where Rocky Linux harden came from is we're saying, "Hey, here is our opinion of what hardening should look like and here's something you can effectively turn key use to start protecting yourself and your interests in production." Awesome.

So, Brady, thank you. I'm going to want to go to solar in a minute, but before I do, uh, uh, you know, for his take on the what and why. Um but before I do a if we're looking at this from a business level uh right a CISO level or an IT operations management level what are the expected benefits or gains um from deploying uh uh Rocky Linux from CIQ hardened or RLC Pro Hardened. So you're making a balance between costs and risk and productivity. You don't want to impact your performance and you don't want to impact your functionality. The OS is there bridging the gap between all the hardware you're invested in and the expensive software stack you've built out.

So it has to work. You want to maximize performance. It has to have all the functionality. Uh but in the meantime, it can be very expensive to build and maintain your own hardened operating system. Companies do it. It is a very time inensive and you have to have the right skill sets internally. Okay. Even then you have to maintain it release over release. Making sure it's always stable. Make sure you have stable upgrades. Make sure that the uh you're always keeping par with the industry and thread actors don't wait around for um things to go through bureaucracy before they launch it into uh and start train.

Make sure that you wait for you to get all your patches in place like we'll go through they're not going through sprint cycles and to get permission to start using the latest cutting edge technology. So it's a cost factor from the perspective of do you have of do you want to maintain a team that large it's a factor of can you get and build a team of the security experts to do it and it's a factor of mitigating risk uh there like what are you going as far as you feel you need or have to to mitigate the risk of the increasing risk of all the cyber security threats that are out there.

Awesome. Thank you. And I want to go with that solar. Do you have uh uh and I'm sure you do some more to add uh as to uh you know at a top level what is RLC Pro Hardened and why is it important? Why did you guys decide to build it? Uh yeah uh thank you. Uh I liked a lot what Brady said. Everything is just right and very well said. Thank you Brady. Um I have a few things to add only a few because it was so well said. um uh specifically uh in uh the community rocky Linux project and LC uh Rocky Linux from CIQ. We currently maintain bug compatibility with Redhead Enterprise Linux.

It would take some extreme uh reason for us to deviate from that. uh uh and uh uh what we are doing with uh LLC harden is uh we are actually deviating deliberately deviating at least from bug compatibility. So we do fix bugs uh and uh we do make changes for hardening obviously uh and uh then another topic that was brought up uh is reasons to use CH business outcomes u uh Brady spoke very well but I can add maybe a slightly different perspective a bit more technical uh what's using the system actually means for for CIS admin uh for example Uh so our goal of the hardening uh is that fewer systems would actually get compromised.

Uh those numbers are small anyhow luckily uh or because our upstream our baseline uh Red Hat Enterprise Linux is so good. They do a lot of hardening already but our upstream and our baseline. So we will always be adding more on top of that. We already have some additions on top of uh uh Rocky Linux 9. So on top of real 9 as well and uh we will be having more. So uh returning to my uh previous line of thought uh fewer systems compromised uh and lower impact of successful security compromises as well and a better detection of successful security compromises. uh and uh then uh in certain cases uh when a new CV is revealed uh we will be able to find out that our systems as a hardened distra uh is not affected or affected to a less extent.

We will communicate this to customers and there will be maybe less of a panic uh to address those CVS. uh they will be addressed obviously if anything still needs to be addressed on specific systems they will be updates they will be timely uh but maybe less of a panic and uh uh then for specific CVS we will occasionally page them ahead of our upstream uh so uh if we identify some CV that is important to page and uh that uh it's low risk to page uh maybe it can sound paradoxically to some but fixing a bug always actually carries a risk of breaking something introducing a new bug or some unexpected interaction.

Uh so uh when we consider fixing something ahead of our upstream it's not only about criticality of this uh uh particular issue we are fixing but also um potential risks associated with it. So when the balance is good we will fix issues ahead of our upstream. Uh and uh one final thought I had uh is that our hardened systems may be uh good uh for not literally for compliance. This project focuses on real world security and not compliance. This is important distinction but uh for certain cases like um uh M&A when security of a company being acquired is evaluated by a potential buyer or uh uh for due diligence uh then the very fact that uh all the fleet of service are running preh hardened operating systems it is a plus.

That is awesome. And then so to play back you set me up for a good segue as well. Um so you know I hear rough summary is um if you run RLC hardened um you should have fewer systems exposed smaller attack service within um those systems better detection faster remediation timely updates sometimes ahead of upstream and then also what we kind of I think in marketing materials called um an your audit ready um not necessarily in the terms of compliance but you are in the cases of M&A for instance where people are doing due diligence. A company had an issue with this when it was acquired.

So, you were saying running um, preh hardened or hardened Linux systems should help with that due diligence. Um, um, now I'm going to use this. Brady, it looks like you may have some comments. So I'm going to actually first let me see did you have anything to add to that? Nope. Uh there's a there's a lot of interesting details to get into but it's with respect to time. I think maybe some of them will be covered. Okay. Awesome. So um I want to move into a little deeper. You brought up compliance solar and uh and I'm out there and I'm trying to learn how to harden my Linux systems.

I come across these these these uh these benchmarks such as STIG and CIS uh CIS, right? Um and so I was under the belief that if I just go uh uh ensure my system is CIS compliant, I am good. Why um is that not sufficient or why did we not just align with CIS Ben CIS benchmarks for our hard? What is the difference between um rocking Linux from CIQ hardened and uh applying CIS benchmarks? Uh yeah, this is obviously something we discussed when starting the project. Uh and uh I would say those things are orthogonal. So uh what we are making in our hardend product is changes that are pre-baked into the system and this includes code changes and code additions extra packages for security.

Uh what those hardening guidelines say is how to adjust settings that were meant to be adjusted and some that were not uh on u pre-existing Linux distributions that are not specifically intended as hardened. This doesn't mean there is zero hardening. You cannot find a Linux distra that has no hardening today because some hardening went into ultimate upstream projects. Uh but um uh most distress uh do not add hardening as the last layer like we are doing here.

uh so uh those uh guidelines about cening those distros and specific installs of them or maybe there may be uh may exist some I know do some do ex exist like on AWS marketplace preh hardened images uh but still it's a modification a fork of uh uh as a distress for specific install uh that is poorly maintained as a fork not or not maintained at all in those images where you install then you update and then you may have some of those settings lost on on other update you have to reapply and uh what we are doing here is a product actual product that is

preh hardened uh with like I said primarily with code changes and additions which also add additional configuration settings that didn't even exist before and we'll be providing documentation on those but most importantly we provide reasonable defaults so a guide is not really necessary uh to have a system secured for real world usage. If needed for compliance uh then they can be combined both what preh hardening we have applied and most security hardening guides can be applied on top of our systems. Phenomenal. Phenomenal. And I think there are plans actually to uh to uh add CIS compliance to Rocky Linux from CIQ harden, but that's in addition into or over and above the other modifications we've made.

Is that correct, Brady? And do you have any comments to add? No comments. I mean we do intend to provide CIS uh guides for CIS and STIG but that is in addition to the core value is really coming from those code changes that Solar Designer's talked about um where we're going above and beyond what you'll get from the STIG or CAS uh you know guidelines the code changes which are going out and reducing the attack surface it's adding additional prevent prevention for entire classifications of vulnerabilities, but it's also adding detection because in the end, you're never going to prevent every single penetration event. But what you can do is greatly increase your ability to detect when those things could happen or almost happened.

And that's part of that whole trifecta of reacting to security events by patching when vulnerabilities are detected, outright preventing new vulnerabilities or at least mitigating them until you have time to get those patches applied and then detection to ensure that when things do penetrate, you know about it or you can catch them very early in the process. Awesome. Thank you. Thank you. And for the audience and for you guys, I am going to want to get into the how, right? the details of the hardening in the modules momentarily here. Uh but last sort of set of questions before we do that I'll throw out to both of you is um how is RLC Pro Hardened more preh hardened than rail and other enterprise distros.

Uh and I'm going to add another question you guys figure out how to parse this and and why why don't I just do this myself? Why don't I just go um get an upstream distro and uh you know um uh go find some guides and do the hardening myself. Um anybody can pick up that question. I'll go ahead and start. Um part of it is many of the guides do focus on I got my STIG or I've got my CIS. I'm I'm good to go.

uh and that's not necessarily with code changes or you're bringing in you know even LKRG as an example is an open source project and many of the the items we're bringing in are open source projects some of this work has been done as part of uh the special interest group the security special interest group at thefra Linux uh enterprise software foundation so some of these things are out there integrating them with your system with your specific setup ensuring they're maintained build over build, minor version over minor version, update over update, not even accounting for all the enhancements that we have on the project roadmap. That's already a lot of work to incorporate, to ensure stability, to maintain it, to make sure it's configured correctly.

Um, this is a can be a large lift. Can you do it? Yes. And everything CIQ like CIQ's work is in the open. So those who are particularly savvy can go and pull some and put it together. However, uh this is not always feasible for an enterprise from the perspective of do you have capacity and and bandwidth to do that. Do you have the headcount and just time to do it? Uh does your you know your procurement or your auditors allow that to be exclusively done internally or do you need some sort of commercial indemnification uh alongside this security work for risk mitigation purposes? So you may not necessarily be uh you know your business side might necessarily want that to all be inhouse.

So there's uh there's definitely the ability to do it but there's a lot of value added by having it come from CIQ both for the convenience and reliability and stability um for the headcount and capacity and also for the the business liability. Okay. And and thank you Brady. um solar. Do you have anything to add in particularly around um around uh you know what's the what how how is RLC preh hardened different than uh RE and the other hardened enterprise uh districts? Uh yes. Uh so uh I inadvertently answer that part of this question earlier. Uh uh real is our upstream and our baseline.

Uh so uh we already have all the hardening they have done uh in the uh minor release that uh our LCH product is based on and I have a lot of respect for Redhead and their engineers for the hardening they do but uh uh just because uh how uh we are positioned as a downstream we inherit all of this and we add more to it. So that is how we are different. We have more hardening and how is it hardening is different from this? Uh I would say uh it is more progressive. It is evolving at a faster pace. We pay more uh uh uh more attention maybe or um at least we we are quick we are quicker to adopt uh new hardening measures and this is our intent.

Um and uh uh when vulnerabilities are found uh we uh review not only uh the specific bugs but we also learn from them. Uh we learn the vulnerability class if it's a new class or if it falls into existing we uh treat it as statistics basically this vulnerability class is common and uh we try and address the entire class. So we had cut in changes most just code fix to to individual bug. We had code changes like defensive programming I would say uh to prevent further bugs like it yet undiscovered or even not introduced yet uh from having full impact or from at all. Well, that that is actually a great statement on the on uh the accelerated risk metad mitigation um uh uh and the way we we enable you to proactively mitigate threats.

Um what I do want to do is dig a little deeper solar into the specific system level hardening. Um so maybe you can take a moment to uh cover what some of the key harden modules are and why you chose them. uh or actually maybe I'll set it up. Maybe we start with uh the deployment of LKRG. Um I know that's something that you co-founded and you're a contributor to um that we've chosen to deploy at RLC Pro Hardened. And by the way, David, real David and Christine, we are seeing your questions. Um if I don't get to them in the next five minutes or so, then we will get to them during uh questions and answers.

Um but sorry, back over to you, Solar. uh can you tell us a bit about uh what modules and why they were chosen? Maybe starting with LKRG. Uh thank you Brian. Uh but if you don't mind, I'd like to address one previous question a bit more fully first before we have moved on. Uh the question was why don't you do this yourself harden the distribution the system rock Linux or RC uh to create the equivalent of our harden system and this is related uh obviously to us doing things in the open contributing to the Linux community publishing everything as open source which we intend to uh so yes you can take the open source code even though a lot of effort many years of effort went into it.

For example, for LKRG which you have mentioned Brian uh it's a project we released publicly in 2018 before even started as a company and uh it was founded uh uh the project by my colleague Adam Zabroski. is currently director of offensive security at Nvidia and this is uh his pet favorite open source project. Uh uh anyway um this is a distraction. Sorry. But uh uh we do a lot of open source. That's my point. And yes, you can take this open source stuff and build it yourself and create a harden system and some uh CIS admin with advanced skills could do this and would have fun with it.

But uh at enterprise uh size it is a fleet of systems and uh maintaining each one individually is hard and would be a lot of effort and uh installing them from some repository we would essentially be duplicating the effort we are doing at CIQ in terms of productizing. Yes, you could productize this for your company's internal needs but it's a lot of effort really a lot. And uh one other point that I haven't mentioned yet uh is uh the secure boot chain.

Uh so we have signing of uh uh the boot chain components by CIQ and this includes LKRG uh something you wouldn't easily do yourself unless you introduce your own keys which you will need to set up on every system uh in uh firmware settings which is compass not something you would do uh at a company something you may do on as a hobby on your own computer but not not beyond that. Oh. Um, thank you for that answer. And then if we could uh uh then move to solar. What are and I'll reframe the question. What are um the modules that were hardened and why were those chosen and if you need a starting point I know there's interest in LKRG.

Uh so maybe start with LKRG. Yeah. So as a project that's have already been mentioned we have LKRG Linux Kernel Runtime Guard. um which uh uh validates integrity of the kernel itself and uh of some of its data structures specifically pertaining to information on running tasks. uh uh it is uh uh intended to catch various kinds of uh unauthorized system changes such as introduction of root kits and in a certain third party study there's a master stis written about on it uh caught eight out of nine kernel root kits tested no other kernel kit detector tool was anywhere close to this result even so This

isn't even primary focus of alkar g uh in g we are thinking high about uh high level about protecting the kernel and its data in general not specifically against road keys but it also does that and uh uh it also detects kernel vulnerability exploits specifically those that try and modify task credential so that you have a nonroot shell and uh uh an exploit would cause it to gain root privileges. We try and detect this so promptly that those privileges would not be used uh at all. For example, uh when there is an attempt to open a file using whatever privileges a test has, LKRG first checks that those privileges are authorized against it.

Shadow database of process credentials that keeps track of what is expected. Uh so an exploit would have a harder time. It would need to target both databases not only the kernel's normal one but also Jes and and and so and then uh other modules I'll brace. So I know uh we have a hard and gibb c uh can you tell us a bit about the hardening there? Um uh I believe we have custom code um or you've done some of that hardening yourself. Can you tell us a bit more about uh why the hardening and what hardening was it? Yes, thank you Brian and uh actually this is what I would have started with because LKRG while it is very important it's an add-on.

It is an extra package that we provide that we maintain that we commit to maintain over multiple years and that we sign with secure boot all of which is very important but it is an add-on and uh what's most important is changes to our core system packages. Uh they would be harder to describe individually. There will be a lot of them we already have too many to mention now. Uh but uh yes we have currently changes to Jalip C. Currently I would say they fall into seven categories to glibc alone and glibc is a core system package. Almost every other package uses it. So it affects how every system runs essentially.

Uh one change that maybe stands out is uh that we have our ellipsy and this dynamic link distrust uh most of its typical environment variables when it's run over u privilege change context privilege uh change so when the program uh runs uh by someone with different privileges and the program eventually gains when it is running uh uh upstream JFC already has some of these But we take it to a next level and uh we review the entire list of environment variables changing from version to version of gypsc and make our own decisions on what else we can harden there.

uh I wouldn't go into the rest of the six change categories for JIP alone it would be too much but yes but currently we uh have a variet packages for GIPC and for open SSH and this list will be growing a lot and and then um we've added the control framework uh what was the thinking behind that or can you provide some background on the control framework and the thinking behind bringing that in yeah uh control is uh something uh I came up with for open pen wall Linux project uh that I have mentioned uh like 20 20 years ago and um uh it is about uh changing uh system settings that are normally not uh officially configurable.

It's stuff uh that uh for example CIS guidelines would advise you to change. So you run schmot commands on some system binaries and you have a program with fi permissions on its files that are set differently from the way it's packaged and then uh on a uh system card and with those guidelines you when you upgrade this package you lose those permissions you need to reintroduce uh in control is different that on one hand we add something on top of how stuff stuff is packaged uh uh permissions config file settings the defaults they they remain in certain way packaged static uh but then through the control uh framework uh those settings may be adjusted and importantly they will persist our package upgrades uh we do this through using RPM triggers.

So when a package is upgraded a piece of code coming from control gets triggered um and we call those things control facilities. uh different packages may add extra facilities to control and um this pieces of code is triggered and it rate reduces whatever change was prior to package upgrade. So you first on package upgrade you automatically triggering dumping of previous state then the upgrade and then reintroduction of your new custom state on this system. Awesome. Awesome.

And before we move on to another topic so we can get into Q&A which by the way there's some great questions you guys can't see them there's some great questions coming through to the audience uh please keep them coming we will shift to Q&A and open discussion um open discussion in a few minutes um uh but uh solar and or uh Brady of course Brady jump in at any time if you have anything to add I'll direct this towards solar you've also done some work around password security uh around password policy as well as hashing. Can you uh tell us a bit more about what modules those are and uh what you actually I'm deeply into password security since mid9s and this means both offensive and defensive side of it.

Uh actually I'm into offensive security for Linux not only for in terms of passwords road exploits and stuff and this is important experience that I need to properly protect systems. Uh but in terms of passwords in particular uh we do have uh two pieces of code uh that I wrote or co- wrote with echo maintainer uh or different upstream original out uh in uh LC harden and this is password QC which is the password and passphrase importantly policy enforcement tool. uh which we suggest as a replacement for redheads uh uh PW quality library. Uh uh we have uh some data from a third party. There's a presentation showing how more efficient uh or actually mostly most more importantly effective I would say um password KC is.

Uh so it prevents more weak passwords without causing greatly user frustration. That is this is very important to balance uh and to frame it that way because it's a trivial task to ensure no weak password simply don't allow any password insist on password length like 30 plus we don't do that although you can configure a system to insist on plus if you but yeah oh go if I may interrupt quickly sorry um and I think you were starting to get to this so forgive me um how does password QC and its performance you talked about some of the comparisons compared to what is shipped um in upstream realm um for password policy and security.

Mhm. Uh so for typical default settings as analyzed by third party I mentioned uh uh password QC uh detected in this specific test uh four times more uh weak passwords than uh PW quality from Redhead. uh those numbers are small uh but uh uh any small numberless matters it's an account hundreds of systems especially right that small number really matter yeah so the difference is four times in terms of detection of passwords at pretty much the same effect on user experience or or actually even slightly better user experience with password QC because uh there are no unexpected rejections of for example simple generated passwords. Awesome. And um that uh is very helpful.

Brady, I want to go to one last question then last comments and we'll shift to Q&A. Um Brady, I want you to start with this, but solar I absolutely want to hear uh your background and that is um what is the performance impact? Brady, early on you talked about the fact that there was always a trade-off, right? We could have the ultimate ultimately secure system but it may be um unusable in the real world solar. You talked about this addressing real world security. Um underpinning those um comments are the consideration of performance. Um have we considered performance and uh and and and if so um how how have we mitigated performance concerns?

And I think that's a that's very important. I'll try to keep it concise from a functionality standpoint. If it you can't use the system, then users are either going to circumvent it in ways that reduce security or you're just not going to be able to use it. The most secure system, the most truly secure system is one that doesn't actually function in real life. So, we're constantly aware of the balance between functionality and hardening. Uh, but there is trade-offs and there are levels of trade-offs both in functionality and performance implications. from a out of the box RLC Pro Hardened targets no performance impacts. We are absolutely measuring the performance impacts for everything we turn on by default.

At the same time, we do provide and will continue to provide optional um increases in security with additional like hardened tools like uh harden malik as an example where with maximum security you will see a performance increase and we're always looking at how we can reduce that over time but in the short term that will not be uh by default. So if performance is a higher priority for you immediately then absolute maximum security uh you'll be able to turn those things on if you desire but from our default out of the box hardening we're every single thing we add we are benchmarking and aiming to not have an impact to your performance and being cognizant of the functionality impacts of any hardening we add to your end users and your use cases.

Awesome. Um and then uh and again all uh let me just we're going to get ready to move to Q&A but solar I did jump in as you were asked um commenting on password um quality. So I want to give you an opportunity to add any additional insights or information on password policy not quality um andor the performance discussions that we've had and then after this there's some questions waiting for you that I'll ask you. Yeah. Thank you Brian. uh I would actually like uh to uh make this flow from um password security to performance because those topics are related and related they are for offline password cracking and defense from it.

Um uh so I wrote co-maintain John the Reaper password crackers since mid90s and this is a lot about performance optimization because uh you are working offline with password hashes and encrypted files. It also supports lots of things by now. So cryptocurrency wallets for example but returning to password hashes uh uh and uh uh this is about optimization for performance. Uh so I have a lot of experience with this and this is very useful also uh to minimize performance impact of security hardening changes to know what will have impact and what will not. Uh and uh uh still on the passwords topic another change we are making inc is we enable yescript password hashing as a new default instead of sh 512 script that is used by upstream enterprise Linux 9.

Uh yes script is something I designed using u building on top of the excellent design of escript by Colin Perival who was uh in in one of his many roles freebd security officer for a while. Uh and uh uh this script is the next evolution of uh this concept uh sequential memory card um uh password caching. uh it is used already by many other distributions but not yet by enterprise Linux. Here we obviously make it enable it by default and this maintains compatibility with standard enterprise Linux 9 because all the code is already in there. I contributed to upstreaming it as well and others did contribute obviously it's open source.

uh uh so we we enable it now and uh it does provide advantages against the offline password h c h c h c h c h c h c h c h c h c h caching in particular on GPUs it is currently unimplemented on GPUs and that's for good reasons because GPUs are not expected to provide a lot of performance benefit over CPUs for attacking those hashes so there is little motivation for people to implement it that way but they will implement hopefully at some point uh and I do expect like I mentioned uh little performance improvement going to GPUs. So we provide this kind of defense.

Uh and uh on performance of the hardening changes uh there are two kinds of hardening changes we make. Most of those have no performance impact at all. So those changes to core system packages like I mentioned C and open SSH right now there's no performance impact and then there are optional additional uh components. So maybe at the uh future point additional configuration settings that will have performance impact. Right now uh we have two additional packages the Linux kernel runtime guard uh and hardened malo for the kernel and for user space respectively protection that has performance cost. Uh Linux kernel runtime guard has low performance cost. Uh according to a set of fonx benchmarks it's about 100 tests.

Uh the geometric mean of slow system slow down with default settings of LKRG was 2 and a half%. Uh this is mean which means that uh certain tests became a lot slower and some not at all but the mean is 2 and a half%. Uh for hardened maloc unfortunately uh the performance impact is a lot higher. It's in t of%. uh but as I mentioned uh N has to be enabled. So uh customers can choose whether to enable the things on a particular system and for example uh they may be enable it on a system when there is still idle processing time left few time left on the system uh to have greater protection and uh if uh the system later starts to experience in load issues those protections can be disabled by the admin in order to gain better performance at expense of security.

So the tradeoff is left up to the assistant. Right on. Thank you. I'm glad we went back and you expanded on that answer. Thank you, Solar. Brady and Solar, thank you for answering my questions and participating in the discussion we've had thus far. We're running a bit behind my intended schedule, so I want to go ahead and jump over to Q&A. Um, I don't know if you guys are seeing it, but we've had some great questions come through from the audience. So, I am just going to jump right in. Uh I'm going to start out with a question from David uh that asks is LKRG implemented using EBPF?

If not, is it just user mode code? Uh thank you for the question. It's neither. Uh it is a kernel model. Uh there are also uh uh user mode helper tools but primary elkage is a kernel model. Uh so it does not use ebpf. It loads into the kernel's oldfashioned way uh with native code. No uh ABPFM uh and it intercepts uh some kernel functionality through K probes. So optimized to F trace by typical kernels actually uh in the same way that an EBPF based implementation would there is actually exists another project that does somewhat similar things through EBPF but we do it through a kernel model.

Awesome. Thank you. Moving on to the next another question from David is are you using Linux in MLS mode? uh uh as to SE Linux uh we retain what uh real uses. So what's our upstream enterprise Linux uses we retain uh all of it and we currently make no changes to that. Uh the protections we are adding are orthogonal to those provided by SA Linux. Well one uh related thing is that LKRG it adds a little bit of protection for SA Linux itself. So if it was enabled and it unexpectedly becomes disabled, LKRG will detect. But as I sat, we make no changes to say a Linux car.

And then Brady, I want maybe to direct this one towards you as I march down the list. Um, do you bring your code changes back to upstream and if not, why? And I know there's absolutely a component as being the product owner here that's on you to answer and then solar maybe maybe add any clarification. Yeah, absolutely. It's our policy that any work done on open source code is done in the open first of all. So uh everything that CIQ works on will have an open source of public GitHub repository where anyone can visit. Uh we make this you know free and open. Our FIPS work is done in the open.

Our LTS work is done in the open. It is we make it as easy as possible to get access to it. As for contributions upstream, we absolutely do make contributions upstream and push contributions upstream where it makes sense. Not all of them make sense to contribute upstream. Um, but when it is something we feel that the upstream would want or should be part of the upstream package, we do push it. Uh, in some cases, it may be pulled from our open source and um, in the open development. Awesome. Thanks. anything to add to that solar? Uh yes uh Brady thank you for the excellent and entirely correct answer and I can add some specifics uh and u uh not only regarding open sourcing of our work directly but actual contributions to steam projects where actively contribute or at least encourage similar changes being made there.

uh uh I have mentioned seven categories of changes to glibc we currently have and uh one of those changes equivalent of it recently was implemented in upstream glibc I helped review those changes I encouraged this a lot and I think it wouldn't have helped wouldn't have happened if I didn't remind about this uh in particular for the a sprint function the semantics are finally changed to be uh safer Okay. So, um let's see going on to the uh next question. Uh is LKRG specific to LKRG is it different from IMA? And maybe if you guys are familiar with IMA, which I'm not, you can uh quickly explain what that is.

And we could table that one for a uh a minute uh uh and go to the next and then we can go back to Christelle's question uh if and when we get a chance. Um question here two the next two questions from uh Michael and Matthew are are really about deployment. So Michael asks what are the thoughts on RLC Pro Hardened on embedded devices as well as use in regulated and compliant environments? And I know those are kind of two different things. I know Brady will have some thoughts. Um uh but maybe why don't I we start with solar uh uh what you know and I think in particular solar.

How about RC RLC Pro Hardened on embedded devices? Uh well uh it should work where typical rock Linux works. We uh make no changes that should cause extra compatibility issues. So at least we will treat those as bugs and we'll try to fix and avoid. Uh so it's usable. Uh and uh then the question is which embedded devices and what's the purpose? Uh for example we hear that for alk independently from this project uh from Linux. LKRG is being used on some Android devices uh by uh some some companies making some derivative. Um so people are using this but uh whether you should it really depends on your use case and u what you want to achieve there you can have security handed device.

Yes this makes sense. Okay and then I think I want to direct the second part of the question to Brady to answer quickly and that is uh uh what about the use of RLC Pro Hardened in regulated and compliant environments? Absolutely. Um and just to s out there embedded in IoT definitely look at those uh distinctly we have heard asks about hardening for IoT and embedded devices distinct from onrem HBC and cloud use cases but we al want to respect that there are some nuances with embedded and IoT items that are not necessarily captured in the scope of the RLC roadmap today.

We're definitely um we work with a lot of uh large like on-prem bare metal and uh cloud-based use cases for RLC hardened but know that that embedded in IoT use case is something we're aware of for compliant uh want you to know for compliance for auditing for going through your atto for talking to your compliance team we have a separate product called Rocky Linux uh from CIQ compliant which is going to be announced in the near future and that's based uh a large part around our work we've done with customers around compliant Fed ramp ISLE type situations and it's going to be based around our FIPS 140-3 certified modules and additional compliancebased um products that we're packaging all into one individual item.

So when you're looking purely at compliance that the RLC compliant will be something that I would ask you to look out for in the very near future for RLC hardened compliance is not our first goal. It will by definition u meet some compliance needs but we're we split out RLC harden from our RLC compliant explicitly because with some of the compliance aspects you are focused on trying to meet the compliant requirements not so much trying to purely target the most cutting edge and as s mentioned progressive threats there is prevention and there is there are code changes that you can't necessarily do while staying compliant with certain things like FIPS 140-3.

And so we are splitting these into two separate items. Um pure progressive cutting edge security through RLC Pro Hardened and maximum convenient compliance through RLC compliant. Excellent. Great answer, uh Brady. And before we go into our last question and get ready to wrap up, I'm just going to report out that uh one of the questions from Christelle, I believe is how I pronounce it, uh was is LKRG different from IMA? Solar Designer um she came back and explained that IMA is integrity measurement architecture and Solar Designer said yes, LKRG is different from IMA. LKRG isn't an LSM. So, um, with that, we'll go into our last question because we're, uh, running out of time.

But guys, frankly, your answers were so good and this information was so valuable, I didn't want to stop. I wanted to cover as many questions as possible. Um, so we'll go ahead and go into uh, one last question and then wrap up. This question comes from Matthew Smith and it is would the advice be to replace all Rocky Linux servers with RLC Pro Hardened or use a highspec RLC Pro Hardened like a kind of firewall in front of the Rocky Linux estate? Very interesting question about where you deploy this in your overall um um um topology. Curious on the on the responses. I'll let solar go first but I have I can take this one.

Uh so uh we intend as a general purpose operating system just like rock Linux is and so we intended for usage on all systems. Uh I understand that in some cases uh it could be different uh u for for various reasons not all systems could be switched to LCH. Uh but we do recommend switching and it is most effective this way. It is not a firewall sync. So you cannot uh easily protect an entire network that way. Uh but uh one feature that uh for centralized management that we have is Linux runtime guard remote login. So you can dedicate a system where you would collect logs about kernel security events from other system.

Currently kernel but we will probably expand on this. Uh but for this you need LKRG on all those systems as well. So, okay. Okay. And then as we're tight for time, Brady, you said you had comment. Uh, I'll give you 60 seconds to add to that and then we'll wrap up. Twitter. Uh, not every use case. Again, it's general purpose but hardened. And then, over time, CIQ will be announcing additional workload specific optimized product builds. So, for HPC, Harden is not necessarily going to be the best use case. for running your AI hard might not be the most optimized use case. So there will be additional variants uh we'll talk about in the near future um where that variant is for use in a case where you would already be using explicitly a hardened build.

Uh but to Solar's point, it will run in cases where you would typically run general purpose um enterprise Linux, but there are also use cases where the tradeoffs from the hardening we do make it not the right fit because you are make having a trade-off from certain functionality or features that are turned on by default. And with harden, you're intentionally reducing the surface area, which does lead to some unexpected functions no longer being available because those are not typically used. Awesome. Brady, um, thank you for that answer. It was, uh, great. We are at time. Um, so real quick before I uh uh uh sort of give the audience some next steps, uh I'd like each of you guys just to take 60 seconds to provide final thoughts.

Uh Brady, let's go ahead and start with you. Any um final thoughts, closing statements before we move to solar? I'd say that this is a there has been so much work and this is the work of decades coming together into a commercialized project and we're very excited that Solar Designer is has come like brought us along on this journey and that we're together on this journey. Very excited about putting all the work that he's done and the community's done in a uh easy consumable package for all of our customers. and even more excited however on how we like the rest of this journey like where we're going from here.

Um this is it's been a long road to get here and where we go from here is just going to accelerate all the things we can do and all the enhancements we can make to hardening by going like thinking above and beyond and being very progressively focused on keeping up with uh the crazy new cyber security threats we're seeing in the news every day. Excellent. Thank you, Brady Solar. Any closing comments? Brief closing comments. I fully agree with what Brady said and I'd like to reinforce that it's a multi-year journey and we are committing uh to maintaining and improving and building this project over many years.

So, long-term support for the enterprise Linux district as usual plus long-term advances in hardening. This is what we are doing. And uh another point I'd like to make uh is that I'm really grateful to my colleagues at CIQ. I'm not doing this alone at all. Uh and a lot of work has gone into uh this product already by others and more will be in there. Uh we have really talented engineers here at CIQ. Uh so I'm hoping for a lot more and um I'm really happy to work on this team. Excellent. Thank you. Well, we heard from Solar and Brady about uh the launch or the announcement of the tech preview of Rocky Linux from CIQ Hardened as well as CIQ's view on workload optimized um Linux OS distributions to drive business performance.

Um they talked about the system level hardening and modules that were hardened. talked about how RLC Pro Hardened for short offers accelerated risk mitigation, enables advanced threat detection with uh hardening uh additions such as LKRG uh and how we make it simple for you to deploy hardened systems um which build on the already fairly secure upstream distro such as RE and then add specific and unique hardening on top of Rocky Linux. Um, if you have more questions, uh, reach out to us at info@ uh,ciq.com or even better, um, go to ciq.com products hardened. The link to be posted shortly here and, uh, sign up for our tech preview program.

If you'd like to learn how this hardening works and how it might apply to your company and even more so even have a voice in the future of hardening of Rocky Linux, um then go ahead and just uh go to uh the link that will be shared shortly, sign up and uh we will look forward uh to working with you um to improve the hardening of of enterprise Linux distributions. And I am sorry I am having trouble pulling the link. Uh, I will get it right here. Sorry, I thought I had it prepared. And uh, I will drop this here in the comments section. Oh, there we go.

Jess already beat me to it. Um, thank you. Thank you Brady. Thank you Solar for all the people in CIQ that helped contribute to this release and will be supporting this check preview and general availability. Thank you and have a great day.