CIQ
HomeCIQ Webinars

Securing Your Infrastructure With Rocky Linux and CVE Tools

May 5, 2023

In this webinar, our experts will dive into the world of security and provide you with a detailed guide on ensuring your infrastructure's security using Rocky Linux and CVE tools. You'll learn about the latest security threats and vulnerabilities and how to identify and mitigate them. Our experts will also guide you through the online and in-server tools that make this not only possible but simple.

Webinar Synopsis:

  • How To Secure Your Infrastructure Using Rocky Linux and CVE's

  • Skip Grube's Rocky Linux Background With CIQ

  • Scott Shinn's Security Background With Atomicorp

  • What Are The Top Security Threats We Are Facing Today?

  • Security Depends On What You Are Running

  • What Is A CVE?

  • How Are CWE's Different From CVE's?

  • How NIST Cross-References CVE's Using Errata

  • Does Containerized Code Have Unique Vulnerabilities That The OS Does Not?

  • How Do You Mitigate Modern Software Threats?

  • Does Backporting Cause Applications To Break?

  • No Software Is Immune To Backporting Issues

  • Accessing And Viewing The Rocky Linux Errata

  • Are CVE's Different For Different Versions Of The Same Operating System?

  • Can CVE's Expire Or Die?

  • How To Query Fixed CVE's In The Rocky Linux Errata

  • False Positive CVE Testing Results On Backported Software

  • Live Rocky Linux Errata CVE Demo

  • Do CVE's Require Large Updates And Fixes?

  • How To Read And Understand Errata CVE Change Logs

  • How To Mitigate Application Failures After Updating Software

  • Problems With Using The Same Software Version On Different API's

  • How Are Backports Handled When A Security Fix Depends On Other Changes In Previous Versions?

  • Real World Security Concerns With Not Updating Software

  • When Is Updating Software Worth The Risk Of API Failure?

  • How Can You Help Maintain Software Packages?

Speakers:

  • Zane Hamilton, Sr. Vice President - Sales, CIQ

  • Rose Stein, Solutions Engineer, CIQ

  • Skip Grube, Senior Linux Engineer, CIQ

  • Scott Shinn, CTO, Atomicorp

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Good morning, good afternoon, and good evening wherever you are. Thank you for joining CIQ. We're focused on powering the next generation of software infrastructure, leveraging the capabilities of cloud, hyperscale and HPC. From research to the enterprise, our customers rely on us for the ultimate Rocky Linux, Warewulf, and Apptainer support escalation. We provide deep development capabilities and solutions, all delivered in the collaborative spirit of open source.

Hello everyone. Welcome back for another CIQ webinar. We appreciate you joining us. Welcome, Rose, how are you?

Rose Stein:

Hey, I'm doing great. How are you doing?

Zane Hamilton:

I'm doing very well. Awesome. What are you talking about this week, Rose?

How To Secure Your Infrastructure Using Rocky Linux and CVE's [4:46]

Rose Stein:

We are talking about something that is top of mind for a lot of people. This topic comes up a lot, and it's about security, specifically, how we can secure your infrastructure using Rocky Linux and CVE tools. So today we're going to be talking about, we're going to be really getting an in-depth understanding of enterprise Linux, CVEs, and the update cycle completely. Now we're going to touch on a couple different things in the supply chain of Rocky Linux packages, where they come from and what they do for your security posture.

Zane Hamilton:

That's great. Thank you, Rose. I think we hear more and more today about the term SBOM, supply bill of materials. So it's interesting that we're going to be talking about that. I'm glad to hear that we're going to dive into that. I know we have some others to bring in if we want to go ahead and have them join.

Rose Stein:

We do. We got a team.

Zane Hamilton:

Skip. Skip and Scott, welcome.

Skip Grube:

Hey everybody.

Zane Hamilton:

Skip, I know you've been here several times. I'll have you introduce yourself first though, and then we'll go to Scott.

Skip Grube's Rocky Linux Background With CIQ [5:52]

Skip Grube:

Okay. I am Skip Grube. Full disclosure, I am a CIQ employed engineer at large. I work on a lot of different things and I do the same stuff for the Rocky Linux project. I've been a volunteer since it started, and I work on the release team.

Zane Hamilton:

Let's give us a lot of great things. Thanks for coming back to Skip. Scott.

Scott Shinn's Security Background With AtomiCorp [6:13]

Scott Shinn:

Hey everybody. My name is Scott Shinn. I'm on the Rocky Linux security team. I am not a CIQ employee. I work for a company called Atomicorp, and I've been involved in open source security and security in general since the mid nineties. Across the board from intrusion development or intrusion detection development to offensive tools. And I've also been a security officer many times in my life both for the federal government and the commercial sector.

Zane Hamilton:

That's great. Thank you, Scott. Thank you for joining and thank you for being a part of the community. It's great to actually get to talk to people from the community. You guys do a ton of work that I think a lot of times goes unrecognized, so really appreciate it guys. It's like having multiple jobs. You spent a lot of time there.

Scott Shinn:

It's a weird thankless thing to be involved with and I can't tell you why I still do it after almost 30 years, but it's just a weird niche part of open source.

Zane Hamilton:

No, it's fantastic and I think we need more of it. It seems like that's always something that gets not left behind, but it seems to be easy to go fast if you're not worried about security, but then you sacrifice a lot if you don't worry about security. So I really appreciate it. Thank you. So, since we are talking about security today, let's go ahead and dive in and talk about from a security and threats perspective today. What are we facing right now? What are our top threats that we're facing?

What Are The Top Security Threats We Are Facing Today? [7:38]

Scott Shinn:

A couple of ways that we define this is big picture big picture stuff is what we call unstructured threats and structured threats. And, some people might go into something called a highly structured threat. But an unstructured threat is something, let's just call it the background noise of the internet. And that is just the cacophony of automated stuff that's going on, which might not, which may or may not be human directed at this point. It may just be, you've got botnets and stuff trolling for, for vulnerable systems. And then we have what we call structured threat, which is usually somebody running a campaign, right? Like they have a goal, they're there after you or maybe somebody who is working with you or maybe they're targeting one of your customers or one of your users.

And a highly structured threat is that, that category we throw in there where, because we needed a category to put nation-state type attackers into, and generally speaking you can defend against a non-structured threat or a structured threat, but you can't defend yourself from a highly structured threat, right? I mean, they've got tools and things they can do that you can't. In the context of Linux though, when you look at a Linux environment, we approach this from what we call a tax surface, right? So the tax surface is the forward facing part of your thing. Whether it's a web server, an application, or a cluster or whatever, that is exposed to that threat category, right?

So just , you know, putting that in the security guide talk. That's how we say it. But in general, when we're talking about, hey, you've got a thing, it does something, part of it is reachable and, in theory, hey, there's some nuance to that. But that's really how we categorize this stuff. I dunno how you see it, Skip, but that's been the riff I've always used to management in my space.

Security Depends On What You Are Running [9:49]

Skip Grube:

See, I think about it more specifically, and I'll give a general BS answer, which is that the worst the worst security threat is based on what you're running, right? Like if you're not running PHP, you don't care about PHP threats, whatever, right? It's what you have and what you use. That's why I've had people come to me and it's like, well, what's the worst threat on Linux? And I was like, well, PHP, all of that type , yeah. But again, Linux, when you say Linux, you're talking about Rocky Linux. Well, there's 3000 packages in Rocky Linux. Yeah. And no one has all of them installed, right? So what are you using? Are you a Java shop? Do you use Go Lang? Maybe Go Lang like, it's all relevant to what you are, what you're doing, right? And what you have exposed. So I guess that's drilling down further than structured and unstructured is.

Scott Shinn:

Then it speaks to all the things you can do, right? I mean, I'm using Linux, I'm a desktop user, Linux desktop user, certainly use it for all my servers, right? So there are going to be certain threats that are going to target me as a desktop user that are relevant. And you're right. I mean, depending on how you have the box setup you may have a ton of stuff on there that you're using or not using. But all of that is a tax service, right? Depending on not how, but what Right. If I'm running a browser. Right? Exactly. Right. So There are a lot of things you can do on a Rocky box or whatever you're using.

And it really comes down to that tax service, right? Like, what is the part that something can communicate with, whether it's probably largely going to talk about this in the context of internet usage, but I mean, you've got other things too, peripherals, USBs and things like that, that are a threat factor. I think practically speaking, I mean, there's no way, and I've been doing this for almost 30 years, or actually maybe I have been doing it 30 years, I can't tell anymore. To know the entire breadth of the security space, there's just too much to know these days, right? Whether we're talking about forensics or encryption or, attack or defense or whatever. But at least from the operating system level this is what is powerful about a platform like an RPM based platform like Rocky, is that there is structure to the way it's designed, right?

What Is A CVE? [12:30]

We have a package inventory, we can tell you how two pieces work with one another. And we have a, and I think we're probably going to get to this pretty soon, but Rocky Linux, we track threats, right? We track threats down to the package level. We categorize them. We Have a lexicon that we use to describe it, that is commonly known. It's called CVE. And you can query them, and you can query them, right? And you can loop them up. Do cross reference that. Now, if you think about an operating system like a rocket you've got two primary ways of attacking that thing. We're having a problem. You have launch problems, right? Good rocket bad launch blew up, right? No, bad on SpaceX.

They actually did an awesome job today. The rocket they were testing, they were testing the booster and it worked fine. Then you have design problems. It was a bad rocket. The launch was fine, but the rocket was bad, right? So if you thought to apply that to an operating system, you have vulnerabilities that are in the software, right? That's just going to happen. And then you have vulnerabilities caused by the way you've used or implemented the software, right? So that's just two buckets, two easy buckets to think about. They can be in both. And that's, that's, that's really terrible. But we're talking about the way that we classify vulnerabilities behind the scenes with something like CVE, we have the ability to go in and say and this is part of its standard. This is an issue that is in a piece of software caused by the way that it's implemented.

And you'll see this in our area. We'll say, okay, this vulnerability exists in this particular piece of software if it was configured this way, right? We do split that hair, right? Because it's important to understand that there are times you can have something that has a vulnerability in it, but the way we built it makes it not an issue. The way that we configure it makes it not an issue. And conversely, we try to capture when we can where that could create the vulnerability, right? And you'll see that in the edit. I'm probably skipping ahead of what we're talking about with the air out of stuff, but I'm going to ride around to it actually. Yeah. Yeah. This is a pretty key part of when we talk about vulnerability, vulnerability management and living with this is that you can have a vulnerability, but you can mitigate it, what we call compensating control.

How Are CWE's Different From CVE's? [15:00]

You can mitigate it just by the way you use it, right? Or the way you run it. Certain types of vulnerabilities, and there's a higher level above what we call a CVE. So CVE is like an index in the back of a book, right? It's categorizing these things across. There's an infinite number of cvs. There will, there will never be an end to CVEs, but we have a thing above this called a CWE. So CWE is a common weakness, right? And there's only about 900 of those. And that's how we categorized, how a vulnerability works or what makes something to be a vulnerability. So CWE is something like SQL injection, right?

SQL, I think it's scripted query language, Houston database servers. If you're running a web application, an SQL attack basically is a way of tricking the web application to parsing what you send it as raw SQL statements. So you can make the database do stuff that it doesn't want it to do. That's a single CWE. You can approach security in a way where you target the CWE and that in effect, you make the CWE go away. So if you think about a CVE, those probably don't come across very well as software, which it is. And that software has dependencies. And if you get rid of the dependency that that vulnerability depends on, it doesn't matter if it's vulnerable, right?

It's not exploitable, right? We've addressed a different problem. And there are places that have been done in the Linux kernel or in the programming language that will eliminate a particular vector. That is a more modern approach to creating an overarching security strategy. Attack the CWE if we can. Sometimes you can't. A lot of times you can't. But if you can address the CWE you can eliminate it, I mean, you can have vulnerable software, but it's not exploitable, right? It's not going to be as or as exploitable as it would normally. All of this falls under a group, an organization called NIST NIST.gov. And they are responsible for tracking these CVEs and CWEs and cross-referencing this information.

So we as Rocky are leveraging NIST, that's the National Institute for Standards of Technology, in order to and other operating systems do this too. But leverage NIST as part of the way that we track vulnerabilities in Rocky. And that includes at the NIST level when they will categorize the vulnerabilities that we have under a CVE with its weakness ID as well, so that that information is actually combined. So if you are using particular strategies like I'll throw a plug out here for a great tool called mod security to mitigate web vulnerabilities, right? I don't want to monopolize everybody's time here, but this is obviously something very near and dear to what I do. this is all I do. Yeah.

Skip Grube:

You're going well. You got it. I'm like, yep.

Zane Hamilton:

The one thing we say quite often is CVE and Scott, I don't know if you actually defined what CVE was. I mean, I know you told us a lot about it, but common vulnerabilities and exposure is typically what stands for, if I'm not mistaken.

How NIST Cross-References CVE's Using Errata [18:44]

Scott Shinn:

Right? That's correct. That's correct. Okay. And the NIST is where these are managed from, right? So when we put out errata we will reference the CVE in the single errata blob of information. We will also reference NIST, right? And so for the people who are doing this, you know on the security team side CVE may affect Rocky, but it also may affect a ton of other platforms too. So what's important to know about that is when you're looking in looking this up in NIST is they will reference all the information that they've collected on it. And it can be a mile wide and a mile deep.

It can just be scratching the surface. But in general, it's up to us to share it back and forth with them and us. So, like when we say, Hey, the NIST might say, this is the vulnerability, but it's on us, the operating system people to say, this is our fix. And they track it too. So we actually are tracking it from both sides. They will reference our errata in their CVE information. What that means to you though for the people who are watching this, I won't say it's the voice of truth, because it's not always a hundred percent accurate, but it's definitely the place where combine all this information together and it'll say things like, this is what the fix is, or this is more information about the vulnerability, and these are the other platforms it might affect.

Zane Hamilton:

Okay. Thank you. Yeah. So this is an interesting question. Thank you for the question, Dave. He is interested in the way that CVEs are published might not affect containerized code and vice versa, how containerized code may have unique vulnerabilities that don't affect the OS.

Does Containerized Code Have Unique Vulnerabilities That The OS Does Not? [20:43]

Skip Grube:

I will say it can't be containers all the way down, right? So your containerized application is running on top of something, right? Even if, so let's say I've got my Rocky Linux laptop, right? And I fire up a docker container. Yes. In the Docker environment, let's say I run my app and maybe, but maybe my app is launching from a Debian container, right? The CVEs in user space for Debian will be relevant in addition to the CVEs for my Rocky system to the application that you're running, right? So, I'm not sure where I was trying to go with this, but.

Scott Shinn:

Well, you're increasing the attack service, right? Yes. You're giving it more opportunities.

Skip Grube:

And yes, I see containerized code may have unique vulnerabilities that don't affect the OS. That's true. But in security, and I don't know about you, Scott, but I always think of it as additive, right? Now we have these Rocky vulnerabilities for the base system plus whatever container platform I'm running on, right? Like they're increasing. I don't like disregarding the base system just because I happen to be in Debian or whatever for my container app. And then there's itself, of course, right? Same sanitized inputs, right?

Scott Shinn:

And if you think so , when we are running a campaign to attack something, we may not be trying to root or something like that.

Skip Grube:

That's the white hats, by the way. They're not actually running. It's, it's okay. It's cool. It's above board.

Scott Shinn:

The end goal could be anything, right? But I mean, let's just say in general, I'm just trying to get to the data. The fact that it's in a container, it's going to give you some different options to sequester that data potentially. Yeah. But it doesn't necessarily make it better or worse, to be honest. It's just different. I think actually the strength of a container is that you are increasing your velocity to address a particular issue, right? You've got a very streamlined way of building and distributing applications.

Your software developers are now your infrastructure developers, right? So like, okay, great. So are there good tradeoffs for everything? Maybe it is, maybe it's not.

Rose Stein:

Okay, guys, so thank you for this background and helping me understand it more. But I think probably what is on my mind, and probably your mind and lots of people's mind is like, how do you then mitigate it, right? Like, how do you mitigate the threat?

How Do You Mitigate Modern Software Threats? [23:53]

Skip Grube:

Yeah. So I'm going to take this one, Scott, at least for starting, and then I'm sure you'll have extra stuff. So I will say that almost every IT person in the world will tell you, keep your software up to date, because people are always working on fixes and I'm, I'm here to tell you, as somebody who compiles a lot of those fixes, that people are always coming out with ways to solve individual software, right? So Apache, OpenSSL, whatever. Updates. Updates are great. And I'm going to segue that into the way that, as Scott mentioned, in Rocky Linux, we're an RPM based system. It's really great. We have a pre-made thing already. A way of doing updates that is not just oh, here's a rollup, here's a security rollup. You don't know what's in it, but you should install it. Apologies to windows, but. I hate that. This is a security rollup, for the month of March. What, what exactly does it do? Eh, just install it. It's cool. it'll work out.

Zane Hamilton:

It never breaks anything.

Skip Grube:

One of the ways, and you don't even need a Rocky system to do this, honestly, one of the ways, I'm going to share my screen here, and I will show you. So we're going to get into the practical part of the talk here. Going to segue okay. Click the share screen button. Skip. There you go. Move this over here.

Rose Stein:

That's actually a good point, Zane, because when you do updates and you update regularly, then things break.

Skip Grube:

Hopefully not if we, if we at the Rocky Enterprise Software Foundation are doing our jobs.

Rose Stein:

There you go.

Scott Shinn:

The design that we actually use with RPM based distributions is to make that risk go away, right? So we are our design, right? And this is really the Red Hat design also with enterprise software, is that we never break the API, right? So from the end user's perspective, updating is our intent. The update process is not a risky process. We track things like config files. We won't overwrite a config file. So at the end of the day, that is the power of that, right? Is that you can have a low risk way of updating the system. We're not talking about just, dot slash configure, install here. There's a lot more thought that goes into it than that.

Does Backporting Cause Applications To Break? [26:30]

Zane Hamilton:

I'm glad you bring that up, Scott. That does seem to be something that used to happen quite often, config files did get overwritten. So you would have people who would not update.

Scott Shinn:

Will the update kill my API? Will it break all these applications? And I think we'll probably talk about backporting here in a bit, but no, it won't, right? I mean, maybe 20, 25 years ago that was an issue. But this is a problem we've hammered flat decades ago.

No Software Is Immune To Backporting Issues [26:58]

Skip Grube:

Having said that, I will put a great big Skips addendum onto that asterisk, which is that people are not perfect. The Fedora project, our upstream Red Hat Enterprise, Sentinel Stream, and Rocky Linux itself, we're not perfect. And mistakes do happen. Like things, fortunately, especially if it's a very big noticeable mistake, I will say it impresses me how fast things can get fixed, right? So you can't just say that, oh, everything's fine. Nothing will ever break when you update no matter what. That's not, I'd be careful about saying that for anything, even enterprise Linux, although as if in the Linux world, if I had to make a bet on an update not breaking, I would bet on Rocky Linux though, because our deltas for our changes are generally quite small. If we can help it within a version like 8.7 or 8.6 or 9.1 or whatever, the only things we will change are very minor bug fixes or security patches. And the idea is to leave the rest of the software alone and make it continue working. Like we expect it to. So that's my big giant, Hey, by the way.

Scott Shinn:

We're going to get into the Backporting thing here, I think too.

Accessing And Viewing The Rocky Linux Errata [28:24]

Skip Grube:

Oh, yes. With that, yes. So that's a big concept, especially in enterprise Linux. Lots of folks, don't they? It would help them to grasp, right? And I've just got one of the key sites here, which is errata.rockylinux.org. This is a site that presents information that is pulled mostly from Red Hat Enterprise, Linux or upstream. It contains information about updates that Rocky Linux has released, produced, as well as you said Scott, all those important CVEs and links to them as well, right?

Scott Shinn:

So, another critical part of this errata is typically when we're talking about a vulnerability data that's being published by an upstream source, like NIST they're not going to look at it the way that the distributions are putting the packages together, right? So what's important about this area of data, this is where the operating systems get to have, basically articulate why an issue does not exist or how it's mitigated in our package. Right? And just a really cheesy example of that is there was a vulnerability in a particular piece of software NTP, and it's mitigated by the fact that it doesn't listen on anything other than localhost, right? So, You could do it.

We've gone EastWest attack, if you're on the box, you can attack it, but as an externally facing, it's not vulnerable. Well, if you look at this from the CVEs perspective, it would say it's vulnerable. If you have this version, you have a problem, right? But the errata is where we're saying, because we configure it this way, or because we compiled it this way, or because of the way that we modify the source code. Again, we're going to talk about what that means here in a second, but this is where we say that, right? So you have those artifacts in hand. If you're trying to close out an issue, what we call a POAM in the government or you're dealing with an auditor, that's where you've got that data to say, no that is a false positive. This is not relevant to whatever you've found here.

Skip Grube:

Yeah. The important thing is that, and this is again, previous jobs I've had this issue before, and one of the things Scott is touching on here is that security is way more complicated than just crushing the CVE. CVEs are bad. We need to get rid of it. Like the software we're using. I've had issues in the past where there's, what we call checkbox security where it's like, oh we scan the system and it's got a hundred CVEs listed in it, this is terrible or whatever. But it's all about context, though. You can't just go by volume of vulnerabilities or something. Like there's, some a hundred CVEs, could be terrible on a system. But it depends on what you have installed, what you're doing to mitigate things. It depends on how you are using what, how exposed the system is, like you said, listening on the local versus listening on the internet. All kinds of things are.

Scott Shinn:

The term is compensating control. Yeah. Right? If there may be something that you have that compensates for this particular or eliminates that weakness that the CVE needs to live, right?

Skip Grube:

Yeah. So I'm just going to play around here a little bit just because we got this open now, but so you can see, I'm just going to filter through some stuff. In Rocky Linux and in other operating systems really, we have something, I think Scott mentioned it, something called errata. Is that Latin? I don't even know.

Scott Shinn:

Errata

Skip Grube:

Errata? Yeah. Whatever. I don't pronounce it. And they are basically an update in the Linux world. It generally means an update of a single. So in this case I have GNUTLS, which is a security library that is used in a lot of things. A lot of things depend on this. And it looks like, oh, in Rocky 8 there was a security bug fix release. And we can click on this and drill down into it. And we see here our LSA, Rocky Linux Security Advisory. It's given a year 2023 and a number. These just go up in order and down here, you'll notice, like Scott was talking about, this is the CVE, the all important common vulnerability that is being tracked for this package.

And we updated it. And let's just look head over to Mir here. You can look at all of the cool information. This you can see here. This affects everybody running this package. Debian, and here's NetApp. Fedora, like, these are all of the distributions like Rocky Linux that have packaged this are affected by this. And I mean, heck, if you built this thing for Windows, you would also be affected by this. It's not just an OS specific type thing. But this errata here is specific to OS, right? So we track it like this. And again, this is all, most of this is pulled from our upstream Red Hat information. So we have links to their bugzilla here, a hat tip to them. They do a great job.

And this errata has told us that, Hey, we fixed it. Install this version. You can see the different architectures and so forth here. And what's even cooler is that if you're not inclined to use this site, you don't have to, you have your Rocky Linux system here. This is just an 8.7 system sitting on my desktop's old VM. I cheated already. I already gave it an update info. Info, new TLS, and let's see what we got here. Aha, aha. And let's just check it real quick. 1569, and we'll go back here. 1569. Awesome. And it's important to remember that, did you guys call it errata? I always call it errata, but now you're just messing with me. The errata and the CVEs, there's a one to many relationship there where a piece of errata is an update of a single package, right? And this excuse me, an update of a source package could affect multiple binary packages. We updated the source code for Rocky 8. That's an errata. There's an errata, excuse me, there's another one for Rocky 9.

Zane Hamilton:

Neil corrected me.

Skip Grube:

Thank you. Thank you. I knew it. It is errata. All those years of education.

Scott Shinn:

I wonder if you just updated the site at the same time. You know what I mean? I saw it.

Zane Hamilton:

It's a good call. Scott

Are CVE's Different For Different Versions Of The Same Operating System? [35:56]

Skip Grube:

Yeah. But you see there, they'll be different for each operating system because Rocky Linux 8 and 9 are quite different operating systems. They have the same name, but there's several years of development between them as we keep them both supported with updates as you can see. So even though each of them, in fact, we'll pull up these recent updates. Here's the Rocky 8 one, here's the Rocky 9 one. You'll see they fixed the same CVE here, and we'll have ones, in fact I know of one that gives you an example of one too many. I'm looking at you wanting it right now. Actually, let's check just a random, I'm

Zane Hamilton:

Pretty sure Scott said earlier, internet facing.

Skip Grube:

And we get all kinds of love for Firefox. It's one of our most fun updated packages ever. It's great, web browsers are just really complicated OSs these days, right? That's the mantra. But you see Firefox here, this particular update for Firefox covers these CVEs. And you can go and explore, see how bad they are, what exactly they do. It looks like this one's still new. Here is the reserved listing, huh? Interesting.

Can CVE's Expire Or Die? [37:19]

Scott Shinn:

The nature of the CVEs is that they are also dynamic on their side. They will be updated from time to time. They will also expire. Generally they're going to expire because of security research or whoever posted this on the other side, whoever came up with it may have asked to have it retracted, right? So in as much as a CVE is a dictionary or an index of security information, it is dynamic and it will change. And sometimes they'll go away, right? Somebody will say, put it this way, anybody can submit a CVE, right? They may be in a state where they're early right? Or they're not fully understood. And then other times, there's people who will vet those CDs and say, no, indeed, this is not a vulnerability.

So just because they're out there, they're not all at the same level of maturity. I'm sure this will be a good conversation for maybe another session, but there is a framework that does talk about maturity of the CVEs and the exploit and things like that. And that's called CVSS. That's Charlie Victor Victor Sierra. And that's a numerical score. And it does get into things like, does the exploit for this exist? Is it easy to use? Is it over the network? Is it overlooked? So just something to bear in mind with the CVE, just because a tool says you have it. You can't always trust the tool. Life is complicated.

I want to segue here into backporting because this is probably, I've been dealing with this since the nineties. And it's important to describe how the Red Hat mentality worked with this. So when we have a piece of software, say Apache, and we say it's version 2.4.16 or something, it's not, okay. It says it's 2.4.16, but it is not. What it is, is it's 2.4.16 originally. And then as time has gone by code has been backported into it. So we have taken fixes from newer versions or even invented our own fixes and built them into the package. And I know this can be very confusing if you're using a tool like OpenVAS, which I work on.

And you run it against a Rocky box or Red Hat box or what have you, and it'll say that it is vulnerable. But there's more to it than that. And we know how hard that is for you, the end user of Rocky or CentOS or Red Hat, how hard that is on your lives. So number one, I'm on the security team here. I am always prepared. And the other people on the security team, we're always prepared to help you respond to somebody who says there's a vulnerability in Rocky when we know there isn't, right? How do you know there isn't? Part of it is in the errata, but in the actual package itself, you have the ability to query it, and it will tell you all of the CVEs we fixed, right?

How To Query Fixed CVE's In The Rocky Linux Errata [40:47]

So you can use this directly through the RPM command. There's a change log in every single package that's built into the package. So you don't need to go to a website or anything. You can be in a totally out of band offline network and be able to look this up just with the command RPM. So you can use the "rpm--changelog" and the package, and then every single CVE we've addressed, as well as the links to our errata will be in it. And so what I always recommend is, when you're dealing with security tools or third party auditor and you've updated the system and they're like, this vulnerability exists, and you're stuck, we'll go, how do you prove it? How do you prove it? You fixed it, I updated it, but your tool is still scanning it and saying that I'm running a vulnerable version of Apache or SSH or what have you. The RPM database has this information, and it'll say, on this date, we added this fix for CVE 123. And I highly recommend checking that out. If you haven't done this before. The command is "rpm--changelog", and then the package name.

Skip Grube:

I don't know if we have my screen open right now, like on the broadcast, but we should share that.

Scott Shinn:

Yeah. It's pretty critical. I mean, I can't tell you how many times in my career I've had to explain this. I'm sure everybody on release engineering has done this a million times and you get sick of it. I am, I know. But why do we do that?

False Positive CVE Testing Results On Backported Software [42:18]

Okay. So the reason why we do this is because this is enterprise software, right? We do not break APIs. That's it. So a good example, let's not pick on PHP by telling everybody how PHP is handled in Rocky. PHP is a high velocity programming language. New versions come out every couple of months. And they end their lives with old ones. And if you look at the version of PHP that we ship in even Rocky 9 I'm sure it is already the end of life, right? It's probably, what, seven or something?

Skip Grube:

Maybe? No, no.

Scott Shinn:

It's the end of life for PHP, but it's not the end of life for Rocky.

Skip Grube:

If it's not, it's very close. And that's only because Rocky 9 is relatively new.

Scott Shinn:

And that's because we're backporting the fixes from the newer versions into the old one. So, you would look at something like PHP and say that, oh, it has all these particular issues in it, or Ruby, or Python or what have you. But the fixes are being backported, right? So it isn't vulnerable to all these things that a tool will test on it, but you're not going to be able to prove that. Ah, this is it. Yeah. The change log. This is gold. Yeah. This is where you can look at it right there and it will have the bug IDs here, in Bugzilla it will have the CVEs or in there. And there are even settings in YUM and DNF that allow you to query just by CVE.

Skip Grube:

If you guys want to see what Scott was talking about.

Scott Shinn:

This is gold

Skip Grube:

HTTPD is the Apache server software for Rocky Linux. You can see.

Scott Shinn:

There you go.

Live Rocky Linux Errata CVE Demo [44:07]

Skip Grube:

Here's this one you, you notice here. So apache.org released version 2.4.37, right? But you'll notice we have way more. In fact, you know what? It's probably going to be, I will bring it up on the web browser, because I think it's going to be easier to read there. I don't know, whatever you guys prefer, but I can pull it up here too. This is our Rocky build system. It's open to everybody. You can see when stuff was built, check it out.

Here we are. Yeah. So this has the same information here, but let's have a search for 2.4.37. Highlight it for everybody. So if we go down here, look at all the 2.4.37, look at all these changes that were made, right? And almost all of them, you'll see we'll be related to, oh, there's a CVE, gotta fix this. CVE gotta fix this one. Gotta put, so these are little itty bitty updates. In fact, I'm here to tell you many times I like security fixes because it's usually the only thing that changed, like, it's five lines of code. Yep. And it's just, there's like an if or something that was put in there and it fixes it, right? And as a release engineer, I love it because it's easy to build, right? Because there's no new wacky requirements.

Do CVE's Require Large Updates And Fixes? [45:29]

Scott Shinn:

No, I maintained a PHP branch for a few years of a legacy PHP branch. And it's the same thing. It's not like not like a kernel patch where it's like, three megabytes inside. Yeah. Yeah. It's honestly like two or three lines.

Skip Grube:

Like somebody noticed something. And it turns out that if we just add an if statement here, it fixes it. It's cool.

Scott Shinn:

So what you're highlighting here, you've got the version number highlighted, right? So an RPM actually comes in two pieces, right? The version number, which is the version that comes from what we call upstream. Okay? Upstream is when as RPM people, when we say upstream.

We mean apache.org or OpenSSL or osec.net. That's me by the way. That's, they control that version number. But that second piece, that -45, -44, that's us. Okay? That's our version of the software. It means they've made 44 updates, 45 updates, whatever, to the RPM since it started. We always start at one.

Skip Grube:

Tons of credit, by the way, you can't go without listing Red Hat, CentOS Stream, and Fedora as the Rocky Linux upstreams.

Lot of this. And then their upstream would be a Apache I guess at

Scott Shinn:

I don't want to drive everybody nuts with there's yet another. What matters though, at least for the sake of tracking changes to the software itself. Not the builder, not the package, not us, but the actual software is that second field there, where we're saying, that's the second version number so to speak. It's what we call the release field. So the first field is version and the second field is release. And on the security team, somebody says, Hey, I'm running version 2.blah blah of whatever. If you don't mention the release field to us also, or we'll ask you to, so we can help you identify these.

Skip Grube:

We'll see here. Sorry, I'm going to pop this open real quick and show.

How To Read And Understand Errata CVE Change Logs [47:31]

So this is an important concept where you'll notice here, so here is Apache two. I don't know if you guys can read that. I can't really make that bigger, unfortunately. 2.4.37. Like I said this is the file here that contains the Apache source code. It was brought in from, you can go to apache.org, you can download it right now. Same file guaranteed. And you'll notice here though, look at all these .patch files and a lot of them just have CVE numbers listed here. Not all of them, but lots of them. And This is what happens. And we'll just pull up a random one here. I'll show you some, we'll do some code. Yeah. But you'll see this is actually a semi large one. But this little green section here, this is all that was changed, right? They did this and this fixes whatever this CVE was, right? Oh, and this other green section here.

Scott Shinn:

But we don't change the version number, right? Yeah. So the scanning tools can come in two flavors here. The low risk style is what we call a banner grabber. And all it's doing is it's reaching out to Apache or SSH or whatever, and it's grabbing that version field, right? So they see 2.4.13, they don't see 2.14-45. So the tool is limited by what it can gather, right? There are more aggressive tools that actually run the vulnerability. But generally speaking an OpenVAS or Rapid7, what have you, they're just going to do banner grabs because it's not disruptive or it's less disruptive to what you're testing. So it's going to report that this is vulnerable, right? Even if it's fully patched. And if you didn't know about this, you'd be sitting there going, oh geez, I gotta go download the source code from Apache and rebuild it. Well how come the operating system people didn't do that? And the reason is they did well. It's just not as clear.

Skip Grube:

Yeah. It's unfortunate that a lot of scanners are not down with, and to be fair, this is a concept. I've worked with a lot of Windows server admins in the past, and this is a foreign concept in the closed source world that you could, that in order to and a lot of times if you are building or installing something on Windows or even a proprietary piece of software on Linux, you get the new version or else, right? Like you're just going to be vulnerable. There's no in between. But here, because it's open source, we have the option of doing this where we say, okay, if someone fixed this Apache with this patch, and I'm going to give just an example here that we are looking at. So this is the patch that fixes CVE 201910097.

It doesn't really matter, but if you look it up here, it looks like, oh, this was fixed on the jump from the -21 to the -27 here. And that's when this particular vulnerability was fixed with this patch. We're still on the old Apache, but your vulnerability scanner may not notice that. Because it does not track releases. It does not track again what we call backporting. This is Backporting where we just take a little bit of the code that fixes the thing we want to fix and bring it back, backporting to the version we want to run. And that way we can keep using our Apache without crazy changes that might break things.

How To Mitigate Application Failures After Updating Software [50:56]

Scott Shinn:

The other risk is that if I make a change, if I do an update, will it break my application? Right? And PHP is a good example of that, the entire programming language. If you have written an application for say, 5.3 and you've upgraded to 8, there's no guarantee your application will still work. And that's the key.

Skip Grube:

Oh, there's a guarantee it will not work.

Scott Shinn:

There's pretty much not. And there are a lot of different strategies that have been used over time to try to communicate that. Typically when you look at the versioning numbers here, they're trying to tell you something about that. But in practice, you've got thousands of applications and everybody's doing stuff different. And so that's where the operating system comes in, where we're trying to, again, hammer flat that weird stuff.

Skip Grube:

One more place here.

Scott Shinn:

Yeah. And just bring some sanity to the way that we're managing everybody's favorite library here.

Problems With Using The Same Software Version On Different API's [51:46]

Skip Grube:

Yeah. So, yeah. And this is a common, common thing that when people don't understand, like if they say, oh my God, OpenSSL 1.1.1k. Right? Right. Oh, Jesus. Oh gosh. It's so old. And our thing is screaming at us. Our security people are screaming at us to update. And you have to understand that there have been a lot of fixes to OpenSSL here that have been carefully done. I think it wasn't funny, but it was educational. We had someone in the community forums who had the Rocky 8 system here, Rocky 8. You have to understand lots of things in Rocky 8 depending on this OpenSSL. This particular version, this 1.1.1 thing. And he was very determined that he was going to run OpenSSL 3.0, that's in Rocky 9. It's not in Rocky 8, but it is, and that's the version that Rocky 9 is based on. And the problem is OpenSSL 3.0 and OpenSSL 1.1.1 here are very different.

Different APIs. They do not work properly. So if you have a program that's built for 1.0 and you launch it against 3.0, it's not going to be great. you're not going to have a good time. And actually, to his credit, I was surprised. I was impressed. Just a euphemism here. But he did it, he actually took the OpenSSL 3.0 and literally copied it into his Library's folder. And then he said now I can't run anything. Right? And it was like, well, that would be why. And I think it was educational. We had to explain that, hey, I know that your thing is screaming at you, but this is actually cool. This is all right. We have addressed the OpenSSL vulnerabilities with this version. It's okay to run. We just keep updating and keep your system up to date, obviously, because new stuff will come out. But please do not try changing your OpenSSL. It will radically break your system. I promise. It'll be terrible. Like you'll have to reinstall.

How Are Backports Handled When A Security Fix Depends On Other Changes In Previous Versions? [53:56]

Zane Hamilton:

So Jonathan asks a good question here. Skip or Scott. And it says, how are back ports handled when an upstream security fix depends on significant other changes present in previous versions.

Skip Grube:

Very good question. This generally does not happen often, thank God. Because otherwise it would make backporting way more complicated than it is. But decisions have to be made at that point. It depends. And it also depends on the fix upstream with the newer version, like Scott was saying, PHP 5.3 versus PHP 8, right. And maybe there's something that affects both of them, but the way they fix it in PHP 8 completely relies on something you just can't do. Yeah. So you have to make some decisions at that point. And it's like, you look at okay, well are we just going to jump to 8, in PHP that would be a terrible idea. That's not going to happen. But if it's closer though, if they're closer together, depending on the application, it could be feasible, right? So we could say, okay, well I'm going to make up some numbers here, but GNUTLS is on version 2.34 and this would bring us to 2.36. Right? And, it might be okay, we'll see, you know it really depends on, and actually more I think about more stuff, stuff that's linked against GNUTLS, that's probably not going to happen either there.

Scott Shinn:

There are some places that are, it really gets down to where's the risk?

Skip Grube:

Yeah. Right? Yeah. So if it's something like OpenSSL, if it's something like OpenSSL where everything is on the kernel, then that's not going to work because then you have to, if you majorly update this package as our person in the forums found out, shout out to him, you have to recompile everything against the new version. They're all dynamically linked. So, but if it's something like Firefox, absolutely right? Nothing depends on Firefox. It's an end user application. Jump to the version that will get you secure. Right? So, I wish I had a solid answer for that, but the answer is it depends on the action you take and and many times you do, it comes down to, okay, well we gotta develop a fix for the old version.

Scott Shinn:

I will say that if you take the kernel as an example, and nowadays they don't break out every single patch in the kernel anymore, but they used to when I was maintaining, I can tell you back when that was still done, even like say two six kernels, you're only got about 1400 patches. And where I'm going with this, is that there is. Believe me, if there's a way to do it, we will do it. And the kernel is a good example of that. I mean, there's, there's, you're talking about millions of lines of code in those. But if it's like an architectural level thing, I mean the worst case scenario is it doesn't happen often. They'll just say, this package can no longer be maintained and we're going to remove it from the platform.

I can't think of anything recent, at least on the operating system side. Certainly in third party repos, like EPEL things like me and whatnot have just been retired. It has happened. Yep. It has happened. It's not common though. And in this case, at least with Rocky, there's a great big hug of Rocky Fedora and RedHat around this. Right? So, it's not a small team by any stretch. There's a lot of eyeballs on this.

Skip Grube:

Yeah. Sorry, we don't have a lot, and again, I could say a lot of these, they don't have exact answers because security is complicated. And it really depends on the context, right? OpenSSL, not going to work, but something like Firefox, we could bump it.

Scott Shinn:

Or even a kernel. I mean, you can upgrade the kernel. There are kernel projects. And in general, when you're talking about an operating system, we're looking at the whole, like as a whole, right? We look at those interdependencies between something like OpenSSL and GNUTLS or GLibC and how it affects everything else, right? In the environment. So we are just, again, making it stall, we're not doing that. There's a lot more thought that goes into it behind the scenes than that.

Real World Security Concerns With Not Updating Software [58:18]

Skip Grube:

Oh, I have a terrible story about that, so I have, a long time ago, been tasked with upgrading this system that had been implemented. It was a clutch, but it made sense, I'm sure at the time. But this had to be like 10 years ago, and it was a mailman server. Everybody knows Mail. Well, not everyone knows Mailman, but this is mail list server software. And it sends stuff to emails to many people. And part of the project, which again, had been even by the time I was there, had been implemented like years before me, right? Part of the project was building this monitoring system in UI around Mailman to track statistics and some other stuff. It's really not important right now.

The important thing is that I discovered this, the development team's long gone. They had been reassigned or moved on long ago, and it was like, Skip, this is killing us security-wise. We have to check it out. And I looked at it and I've looked at the notes and things that the people that developed it had left. And this young lady, I assume she was young at the time because she made this mistake, was like she said, quote, we always used the latest tarball from mailman.org or from mailman.gnu.org, because we want the latest solid security version of Mailman. And at the time I looked at this, the date stamp on that version of Mailman was, I think six years out of date. And that is because again, if the decision had been made to go with the Mailman package, which was available at the time, at that moment, you would've gotten a slightly older one than what you could get fresh off the presses from gnu.org or wherever the Mailman archives are. However, because the system was enrolled in our IT system, and it was continuously receiving YUM update, RPM update, update, update. And it would've gotten these updates and been fine, and some of the other systems would not have been fine because they did some other things. But in this particular case, it just stuck with me because I was like, it's going to be updated as long as you're willing to download it and manually compile it and install it, but you're, you've been gone six years, right?

Scott Shinn:

So you're eating the risk.

When Is Updating Software Worth The Risk Of API Failure? [1:00:48]

Skip Grube:

So really it's, you know you have to think about those that come after you. You're not going to last forever, wherever you are, right? And so sticking with the packages, it's you're offloading the maintenance of Mailman in this case, or it could be a patch or PHP to experts who get the security updates, who do, you know? So I'd caution anyone, and this is always in my personal IT admin experience, if there is a package available in the repository, especially in enterprise Linux, Rocky Linux and it's available and it can do 95% of what I want to do, I'm going to use that package as opposed to going out and getting it, because I know that, as long as I keep my system up to date, which I have automatic processes in place to do that, it's hands off. I don't have to worry about the security for that package. It's given to me for free. So.

Scott Shinn:

We're taking the risk, right?

Skip Grube:

Yeah. Effectively. Yeah.

How Can You Help Maintain Software Packages? [1:01:54]

Scott Shinn:

Now I'll say this, if you are the person that wants to put in the hours to maintain a package, come join us on the Rocky Lights Project. Absolutely. We absolutely want you, you weirdo. That's how all of us ended up in this space, right?

Skip Grube:

This is our marketing opportunity.

Scott Shinn:

I was like, Hey, I was really just upset that you weren't keeping this thing up to date. And next thing you're running an RPM repo and you're part of an operating system environment, you know? So I would say that, yeah. And we're, believe me, we're all happy to just let you know exactly how we do this stuff. Because most of us obviously are.

Skip Grube:

We're pretty steeped in it, pretty steeped in it at this point, I would say. Yeah. There's no going back now .

Zane Hamilton:

No. Well, we really appreciate it guys. I think we could probably go on for a lot longer. I know we're closed up on time. Scott, Skip, thank you very much for working in the community and doing what you do. I know it can seem like a thankless job, but know that we, thank you very much. It's fantastic. It's great to meet you, Scott. Nice to talk to you. Thank you for coming. We'll certainly have you back. I think there's a lot more to talk about when it comes to security. We didn't even get to the actual tools part of our conversation.

Scott Shinn:

Oh, we have, we have some fun stuff to talk about coming up in Rocky that's pretty big. I'll throw a shout out to CIQ for this, but we are in the process of getting the Rocky Linux cryptographic system certified to the standard called 6140-3, which is a very, very big deal. Certainly, I'd love to talk more about it, but I know I'm running out of time.

Zane Hamilton:

Absolutely. We will certainly have you back soon to discuss that, Scott. So I will let Rose close us out.

Rose Stein:

We're done. That's awesome. Thank you guys. Have a wonderful, wonderful day. And make sure to like to subscribe to share this with your friends. Download it, go back, take all the info that Skip gave you, go to the errata at rockylinux.org and make sure that everything is all good. It's all free. You guys are amazing. Awesome. We'll talk to you later. See you next week.

Zane Hamilton:

Thank you. Thank you very much.