How to stop config drift across a hardened Linux fleet

Okay. Um now uh at CIQ uh and me as director of product we figured one way to solve this problem is to just deliver compliant Linux images out of the box. Right? You lay them down, you boot it, and we have you at 95 plus% compliant out of the box, and that saves some time, right? Because then you just got to tweak and tune a few variables, right? But now we meet the reality of what modern hardening looks like. If you were an at an enterprise, you are managing fleets of a 100 to,000 servers. You have cattles and not pets. And uh the first thing you you run into is actually how do I harden all those systems, right?

And we give you an image. Great. But then deploying those images uh efficiently across the fleet is difficult. So enter the answer. Um, we deliver hardening playbooks that will allow you to take your generic fleet of uh, say Rocky community servers and run playbooks uh, across your fleet and get them up to uh, 95 plus% compliance and even more if you tune the controls. Right? So, that's where you start. Great. We have very efficient playbooks. We've grown across our fleet. We scan them. Our fleet is compliant. But as uh somebody from the DoD stated to me, you know, when does configuration drift start? Well, configuration drift starts as soon as you lay down that image.

Right? Now, one of the things that happens is at the end of the day, you're delivering servers to run workloads. Right? This is like, well, we developed something in a clean room. Now, we deploy it in the wild and we have application dev teams that are running into issues because the port's turned off. the cryptography standard um is high um and uh they will either start asking you to change those systems, they will start changing those systems themselves and before you know it your beautifully hardened fleet is you know a hundred different configurations of hardening um gradually moving out of compliance with your baseline presenting regulatory risk for you.

Now for a CIS admin both that hardening process manually as well as the whack-a-ole of managing configuration drift um consumes critical time right time that you might be chasing down um um CVEEs and insure ensuring your security posture time that you might be deploying modern infrastructure um so uh so that's the problem we have to solve right you are either balancing ree to close out um speed versus risk, right? And that means um I can decide to go fast and not maintain compliance on all my systems. I'm getting exposed to risk or I can error towards mitigating risk and it may crawl my infrastructure team I mean bring my infrastructure team to a sc to a crawl.

So what we need to do is enable a reasonable balance of speed versus risk. >> Got it. Got it. And so Larry, just uh you get to see this from like a sleeves rolled up kind of standpoint uh from the CIS admin kind of view standpoint. Anything you want to say to elaborate on what what uh Brian just said uh keeping the whole heart uh fleet hardened, that week-to-eek grind and what that looks like, what people like kind of experience from a painoint standpoint. Well, from my experience, of course, hardening is important, right? Um, you know, as an operations person, as a person who's been in HBC for years, um, we see a lot of issues like this where we have, um, you know, CVEes and whatnot causing risk to the environment.

So, from commercial HBC's perspective, it's really important that you have a rigorous process that you have people that are testing all the applications. In a HPC, in a commercial HPC setting, you could have, you know, 30, 40, 50 applications that all have to be vetted. You can have a container uh uh first environment and you have to, you know, you have to measure those compliance rules with what's in the container. I found examples just recently where certain applications just don't run because the the STIG or the CIS hardening basically locks down things that are foundational to running inside of containers. So it's really really important to have a rigorous process.

So you have a in in specifically in commercial HPC you have a CAD team that's vetting all their applications throughout the process of the hardening. Right? I'm assuming that we're bringing hardening into a an existing environment. Um and whether it is an existing envir a green field environment or a brownfield environment you still have to do the same checks and balances right because in the end in especially in a commercial HBC it's the product not the process but at the same time you have to protect the IP of the company and prevent risk and this is especially important in large HPCs because the surface area for running things like DDoSes internally or externally is really really large uh in those spaces.

So, I'll leave it at that. Um >> yeah, that gives us some good operations. >> Yeah, great context of kind of what people are kind of running into. Um before we get into the solution and the context on on what how that works, I just want to level set a lot of people who might be coming in here uh new to a little bit of stuff that we do at UCI um on the the players in in how we're approaching the solution. So, uh, Brian Dawson, can you give me, um, just a quick level set on what exactly RLC ProHardened is, uh, in pl in just in plain terms?

>> Yeah. So, so, um, RLC ProHardened, um, is a workload optimized version of Rocky Linux that aligns with the CIQ mission statement of we're delivering uh, the modern I performant and secure IT stack for the AI generation, right? And um with Prohan specifically um we realized this problem statement that we covered earlier and we built a system that delivers um both compliance hardening i.e. this pre-mediation to mitigate the thrash and risk with um with hardening your your systems for compliance reasons. And then as shown on the screen here, proactive harden which in addition to supporting disastig, CIS, NIS 800-171 and going forward PCIDSS and HIPPA as well as other other standards.

Um during that time when you can't patch your system or harden them as the slide showed, we deliver LKRG. We delivered harden modules that are uh u um um common attack vectors. We harden um your credentials. Um and then we deliver and manage these as overall hardened configurations. And what what ProHarden enables you to do again is to quickly achieve compliance um quickly um uh uh achieve protection when your system can't be hardened and patched i.e. in those periods of drift while enabling you to do that in a quick and performant way. >> Nice. Nice.

And then similarly, Larry, can you just give me a highlevel context as to what Ascender Pro is because I think this is the other part of the solution that kind of plays into it and and Brian kind of alluded to the whole idea of playbooks, but I want to just make sure that we level set not just for people who might not know what Ansible was, but even if you don't know what Ansible is, what is a sender pro? What does it look like? What how does it work? Well, Ascender Pro is really comes from the genesis of the whole Ansible project. Um, originally Redhead had come up with a product called tower.

And what a center has done, Ascender Pro has done is bring everything from the legacy what we call AWX environment uh into CIQ and we're doing things like patch remediation. We're keeping up to speed so that there's no CVES in the product. Um and then in addition we provide compliance reporting with a tool we call Ledger and Ledger gives us an idea of what's installed on every system whether it's compliant whether it's exposed to CVEes we have another product we we call uh registry and and registry is a way if you're familiar with um Anible uh of having a vetted galaxy um basically uh repo for your various collections.

So you decide what collections need to be there and then that way you able to vet the public domain before it makes it into your corporation. And then we have react which is a log uh processing module that you know talks about uh you know uh what mitigation we need to take. And then finally a Galaxy pro proxy product that that if you're not using registry we made that available to the open source for people to really kind of derisk outages in galaxy which happen from time to time. So it's really about maintaining the whole suite is about maintaining your fleet doing patching and compliance checking auditing.

You can do things like deploy machines virtual machines and tear them down. It's a really robust uh tool for basically running all your patching and and compliance operations and it's enterprisegrade and scales rather well >> and and if I can call out one of the things that is that is uh um extremely valuable about it uh is that people are used to executing their anible playbooks and anal commands often from um the command line um and to have a well organized guey in front of this that allows you to easily correlate your jobs, analyze runs, um, uh, uh, etc., etc., like with Ledger Pro. Um, we've gotten feedback that that makes a huge difference in terms of managing your fleet at scale.

>> Absolutely. And control the credentials. You know, you you don't have to let your credentials get out of control. They are in one place and that's incender. >> Yeah. Yeah. So, know that's good context on kind of both pieces and sides of this stuff and it's a good segue when you're kind of hinting at the interface and all that kind of stuff uh into let's get into how we solve this problem and what the problem that kind of looks and feels like and where that leads to. So, there's two kind of aspects at it and Brian you kind of hinted at it which is like how do I tackle that first part of hardening a fleet >> right >> to begin with.

Um, and then after that I want to talk into like how do you manage that drift afterwards as well? So, uh, Larry, let me push it to you and can you go through what that solution looks like on how do I manage hardening my fleet? What is, um, using Ascender Pro to kind of get that done for you? >> Well, yeah. So, um, really the the first part is to, um, set yourself up a, um, you know, the Anzible Pro fleet. um connect the um registry to our depot galaxy instance and then you can pull down the hardening playbook. Um once you've done that, you set up job templates in a sender that are going to set up a baseline um compliance check, a drift check uh and a hardening check.

And then um you test it on systems. You test it on a few systems. And you know, of course, you're going to vet that image, you know, in the testing process. So you're going to pick some pre-pror systems or some development systems and you're going to apply the hardening that you think is the right recipe for your organization and then you set your CAD team about testing all their applications verify and you get fun full functionality throughout that process. And as we see on the automation path here, we see the operators. Those could be APIs um or those could be um MCPs or any other um type of uh operator.

They access ascender. Ascender uh pulls what it needs relative to hardening from registry and it'll deploy tasks to this thing we call the ascender eee which then will patch and remediate your hosts. and they're pulling their data again from the global Galaxy server if you allow that or from your curated Galaxy server that comes through uh CIQ's Galaxy. Uh once once you've got the recipe, you just move that to production. Um and of course monitor it and monitor the drift. Uh in the examples we will show here shortly, you can see there's hardening for the various forms of hardening and drift detection as well. >> Yeah.

and and I'd like to um throw out or emphasize and if you have any uh additional comment Larry share um one of the things uh that is I think valuable with that setup in the solution the team has designed is the ability that I can go out and harden um say community RLC or rocking Linux from CIQ pro harden servers you know great I can reh harden them remediate drift but what is also interesting in running to a lot is I have Rocky deployed in my HPC environment. I have Rocky deployed across my nodes. I realize I have a compliance need. And that first pass of the playbooks that Larry will show you will enable you to effectively turn a community Rocky into a compliant hardened Rocky.

And hopefully Larry, you'll cover um the playbooks were designed to um not be opinionated and prescriptive, but to expose what we call bars and roles to allow you to quickly um just tailor what you want to cover via flags and setting of bars. >> So, so this isn't just uh hey, here's the process of by which you're going to be doing it. Um go out and build your own playbooks following this path. this is actually here are the playbooks that are going to help you run these types of things. Does that sound about right? >> That's correct. >> Yeah. And on the other end of that, I'd say that's correct, but it's also not here goes the one recipe for hardening because every organization and every team the infrastructure team support is different.

So, it's a mix of um we're not just giving you a set of tools, right, or parts for a car. We're giving you the car, but you can adjust the interior, change the tires, change the rim, and customize it to fit your needs. >> So, it's a good starting point, right? >> Yeah. Yeah. There's literally hundreds upon hundreds of hardening steps that these uh playbooks go through. And the beautiful thing about it is you can exclude any number of those in in your setup. So, let's say, for example, you have um two environments in your corporation that have two different compliance needs. you, if you're an HPC, you've got to have NFS.

If you just run CIS, it's going to disable NFS. That's not good in an HPC, but you may have another containerbased platform that needs to be fully hardened um with minimal threat surface. And so, you may run the full CIS on that. The beautiful thing about a sender is you can either separate that by organization or you can set up a whole set of um playbooks that that vary uh for the task. Um, so it's very it's very funible in that respect.

>> So do you mind if I bring up the screen just so you guys can talk about that interface because I think this is kind of a very sharp departure from what a lot of people know and think of as far as m managing all these things through uh an interface rather than just doing it all command line and you having to look through files and that kind of stuff. >> Awesome. >> Sure. Good. >> And uh Larry, you're you're going to talk to this, right? Yeah. >> Yeah. So, so in here we're just showing and it's it may be a little hard to read on the screen, but uh we're showing basically um five different uh hardening uh playbooks.

Uh so we have uh Dissa Stig, we have stock conversion to RLC hardened um which includes all the things that Brian had mentioned. Uh these other uh these other playbooks actually do the conversion to RLC or if it's already RLC it uh it doesn't have to do that conversion. Then we have CIS Stig and plus STIG which has been a request uh and Brian can talk to that a little bit from some of our customers who want a mixture of both CIS and STIG. We have a full-on just STIG and then we have this new uh NIST uh SP800-171. >> Um followed up by that we have um two drift detection playbooks that will drift detect a set of systems against CI CIS and DSIG.

And the beautiful thing about these is the playbooks are already there. The only thing you have to do is find the variable selection that you need to match your compliance requirements, right? And uh yeah, so that's that's how it works. It's fairly straightforward. Um you point it at a fleet and you press go and off it goes. It does its work. It's it's it's really it's really a beautiful thing to see. >> Yeah. And if I I'll you know add I think uh three additional comments. So you called out the CIS and disastig um and this is something that is rather um unique um because we've had a uh particular heavily regulated customer that said I have a varied fleet at any given time I don't know if my authority to operate is going to be based on compliance with CIS or compliance with disastic.

Um, so we were able to create a playbook, one of its kind, um, that will enable you to in under 20 minutes, uh, or roughly 20 minutes. It may vary bas based on certain factors, but as low as 13 minutes, run a playbook and not only harden a system for disastig compliance, but harden a system for CIS compliance. And some people may note that um, the cryptographic requirements for the two are different. Stig requires a stronger cryptographic requirement. So the playbook is intelligent enough that it will um have a default for um erroring towards um the stronger cryptographic compliant and managing overlap, right? Or allow you [snorts] to tune and choose uh uh otherwise.

Um, so that speaks to uh the fact that the playbooks alone are more performant than anything out there I can measure because part of where this process gets tedious is um you harden or you remediate drift. It's a 40inut run or more with standard Ocap guides per system and then you find out you've locked yourself out of the system and you have to start all over again. So the iteration time is really really costly. The other thing is what it takes to debug it, which Larry will show you more. When you're looking through these immense um um run reports, and I'm forgetting the name, Larry, right now.

Um so that UI that Larry just showed you, it gives you a notice of success, failure, time of runs, etc. Yeah, it's essentially the playbook run. Uh this next picture here is uh before we go to that though, I wanted to I wanted to stop for a minute. So someone says, "Oh gosh, it's 20 minutes per run. I've got a thousand systems. That's going to take forever. Another beautiful thing about Ascender is it supports this concept of fan out. And anyone who's been in HBC for a while knows what that means. That is the level of parallelism. And even with a smaller system, you can have a very wide fan out because all the work's being done uh by the execution engine as well as on the nodes themselves.

So the um so the amount of fan out the you know how quickly you can get this done is is pretty immense. um you know you can have parallelism above a 100 concurrent nodes at a time so it it goes rather quickly it's uh it's not necessarily a a drawn out process and then yeah >> yeah I was going to say so help me understand where people typically run it now how do they run this stuff now is it a lot of manual stuff that happens uh right now on an individual basis um and even for larger fleets you're maybe just running uh your own

roll your own playbooks on on probably command line anible or something like an AWX that's that's old and uh not updated or or I'm I'm getting to understand the speed factor here and how that is really really helping um and how that can can really kind of speed things up. I'm I'm having a hard time thinking about how as someone who's not into the the hardening stuff that much. How How does that work? >> This looks like what this looks like without the >> Yeah, because I I just I get to see all these nice screens and it makes complete sense to me from the beginning and I'm like, well then how was it done before?

How how are people tackling it right now? >> And I'm going to quickly jump in, but I think Larry has the more meaty part of the answer, right? and and one is as I've talked about um what it looks like without this visibility and without sort of the pre-baked turnkey hardening um is just a lot of churn and often times I'm seeing that you know it is automated it's you know managed through a GitHub repository and in mature organizations um but um but it is also managing that um uh what I call it manually Right? They're automated runs, but now we got to manage a whole collection of playbooks across different fleets, versioning and changing of those playbooks, etc.

But Larry, um, love to hear what you see out there. >> Well, [snorts] you're talking to the wrong guy because, you know, I'm a salesperson, right? Technical sales. So, I would first tell you that, well, regardless of what you're doing today, come and use Ascender Pro. But with that said, I mean, it's all over the place. There are other commercial tools that are available for this.

And uh you know I challenge you um to look at the versatility you have with a sender versus those commercial tools and there are people that roll their own because a lot of you know especially in universities potentially they may not have the budgets and so they you know they use shell tools they cobble things together maybe they have good control of their credentials maybe not you know there's it's going to be all over the place and there again there's when you go into you know into the supplier community and looking from my perspective it's mainly commercial HPC when you get into tier one, tier 2 suppliers, uh they're lucky and they need tools like uh Ascender Pro to really get them over the last mile and the tools that we provide through um you know, you know, through uh Greg Kurtzer's stewardship of the company, right?

Um you know, we're open source first. We do have some commercial offerings and that's part of the game, but uh you know, again, just encourage you to take a take a serious look at Ascender Pro. Come join us. >> Awesome. Awesome. So, talk to me a little bit about what it looks like uh afterwards after you've gotten through the hardening process. Um it's been six months. Uh Brian, you were talking about this earlier, how uh drift starts to happen. Um >> it might be six hours. >> Yeah. >> But Yeah. Right. Yeah. >> Yeah. uh six hours is something that you can maybe if you're on top of it kind of make these small moves, but if you're waiting maybe six days or six weeks or >> six months out, it just gets worse and worse and worse.

>> Um what does that look like and and and kind of how can this solution help you manage that drift? I'm let Larry take a lead on that one. I'll jump ahead and go. I'll say quite literally we have customers where this happens for six years, right? Or you know to that order of magnitude. >> Sure. >> Um um but Larry, I know you just went through that flow. Um and maybe you can talk about um how you solve for that configuration drift after 6 weeks, 6 months. >> Well, well, first of all, you know, uh measure twice, cut once, better long than short, which is my what my father always told me.

plan. Don't do things without change management. Agree to a process, have a feedback loop inside of your organization because the security team isn't necessarily going to be the same as the operations team. So, you need to have those connections. You need to follow good change management. You need to be a well functioning organization, especially in the enterprise world. And then you start doing drift remediation from day one and and drift uh validation from day one. Um and it is a continual process uh because people, you know, sometimes have credentials, they're working around customer tickets trying to make things work. You know, you've got to um you know, you got to run the business.

And so when things do creep up, and they will creep up, um you need to make sure that you're running that drift check frequently, right? We had a situation where a container was brought down from a third party and that container runs inside the operating system and a CIS rule basically fails immediately as soon as that container is cached on the system. So you're going to run into these things and then you know when you do you have to take the appropriate action. It could be getting with the the container uh vendor or whoever's maintaining the cont container to fix those problems. You know it's it's all about operations.

It's all about uh configuration management, change management, and again um in a container ecosystem, it's a little more broad. Uh you have a lot more players that you have to work with and those players don't necessarily um conform to the standards within a corporation. So it is a kind of a it's a it's a ongoing process. It never goes away. Um and here's the last thing I'll say on this is that the standards are going to change as Linux changes. The standards are going to follow with it, right? And so you know, so you know, invest in that as a corporation. >> And Larry, and probably what my response should have been practically, right, we talked about long spans.

You may have long spans where you might allow a system uh uh configuration to drift, like say military. Um sometimes they have a saying, mission trumps all, right? So you may have systems in the field where it'd be disruptive. But that said, what a sender pro and rocky Linux from CIQ pro hardened enable is continuous compliance, right? So this drift remediation job or drift detection job. You start with that runs in minutes and can be scheduled to be continuously scanning your system. Now to Larry's point, depending on the nature of the node, the server, when you detect drift, you have two options. you can flag it and figure out how to diagnose it and remediate it.

But for certain deployments, nodes, and systems, you actually can just automatically then trigger the remediation playbook, which is item potent, meaning it won't um uh it can safely rerun multiple times. So, it will only um remediate the things that have drifted. But this starts to look like, hey, I set it and forget it for those notes. I scan, I scan, I scan. Um, something is detected. Bam. I remediate it. We are achieving continuous compliance at that point. >> Yeah, [snorts] absolutely. That's uh that's not only, you know, there today, it's um, you know, it's it's prudent on a properly vetted um, you know, rule set, right, that you could do you should do these things, but you should always know what caused the drift.

And there's a whole another conversation, we don't have to have it here about that process operationally. Right. >> Great. Great. Um, anything else before we start to wrap up that we want to show or talk to uh that we've uh didn't mention? Um, I just want to make sure that we're covering everything and that we've hit all those highlights on that stuff for you guys. Well, I mean the the only as a sales guy, right, the only thing I'd like to uh show is how React works just at a high level so people can see this and have that as a going away present um you know for them to take back and think more about.

So if you could bring that uh slide up real quickly. Yeah. So this is basically event driven automation. On the left hand side you see an event coming off a host that's usually forwarded through SIS log to React. React gets a chance to inspect that. it goes against its rule set and if it finds that there's a problem and it has a known workaround, it can literally trigger a workflow inside of a sender pro that remediates that problem right away. It could be like cleaning up a temp drive. It could be reconfiguring DNS or we actually have an option to query an MCP and have the MCP come back with a response.

And if it's a known response, you already have a playbook to remediate that. You can then trigger that directly from React into Ascender Pro to do remediation. So um and this React is hypers scalable. you have a central React server and you can put listeners on the edge or wherever within your organization um to take this uh take this these remediation steps and it's so I would say it's an end tiered architecture that is really well thought out and I you know I know the architect personally and I've known him for a number of years and knowing that he's done a fantastic job on this react is a really good lightweight tool.

It seems like uh that takes uh a sender from just being run on a schedule to actually being um proactive on being some sort of monitoring and being able >> versus polling versus event driven effectively. Yeah. Which which means fast response time, right? Reduces um the lag. Um you know, a couple of things I want to add, right? What part of this step in um maintaining continuous compliance is also this phrase of being audit ready right um it's one thing to harden your systems it's another thing to keep them hardened and then it's another thing to be able to prove that you have kept them

hardened or kept them compliant right and one of the things that ascender pro um enables you alongside the RLC pro harden playbooks is that at every one of these runs we are capturing deep logging I Larry has a job there um where you're able to generate a clean um easy to identify report. Um our audit checks enable you to generate what they call OCAP scans um HTML reports with auditors look at to see your compliance level. Um so that's important. And another part of um retaining that compliance is as Larry said, you know what these guides roll. In fact, um, out of the blue last month, they rescended, um, uh, the CMC CMMC requirements right now.

One of the things that we do, why? So, I can go to what we call upstream configuration as code. I can go get various playbooks and run them somewhere. Um, why do you come to us? Because not only do we pre-bake them, pre-est them, validate them to save you from doing that time. It is vendorbacked and maintained. So every time that those um standards rerev um you're going to get a new version of the playbook that you can run, right? One less thing um that you have to worry about. So you know my um you know just think about is um the time that we're taking to do this worth the cost or is the time that we're not taking to do this worth the risk?

and um consider taking a look at um at what we offer with the combination of Ascender Pro and Pro Hart. >> Yeah, that's huge. That's huge. >> And Larry, by the way, you're much more than a salesman, my guy. >> No, no, no. I don't get on uh webinars without people who are smarter than me. And uh you two are some of the two smartest, and it's more than just being a salesman at that point. So, appreciate it. >> Thank you. >> Look, that's the solution at the high level. Uh RLC Pro Harden sets the baseline. Ascender Pro uh with the hardened playbooks uh gets your whole fleet there and holds it against that drift.

Um and then also leaves you that audit behind there. Um Brian and Larry, I want to thank you both for being here. Um and for all of you guys who are watching, if you want to see what this looks like in your kind of own environment, uh your own fleet and and what it can do for you, uh please reach out for us for a demo. Uh we'd love to be able to set up a full walkthrough of what the process looks like. Um, so you can dig into the specifics uh needs for your your setup itself. >> And if you're just starting to try to figure it out, we have a lot of those conversations.

Reach out to us. Right. >> Absolutely. We do it every day. Um, also a reminder that if you're watching this on demand, um, we monitor those comments. We would love to have any questions or feedback from you uh on on what you've seen here today and and how we can go going forward. So again, thank you guys for attending and uh we'll see you next time. >> Take care.