Rocky Linux 9.2 and 8.8 GA Releases!

Join us for an exciting webinar as we unveil the highly anticipated Rocky Linux 8.8 and 9.2 GA releases! Rocky Linux, a robust and community-driven enterprise operating system, continues its mission to provide a free, stable, and secure alternative for organizations and individuals.

Our expert speakers will take you on an immersive journey into the latest features and enhancements introduced in Rocky Linux 8.8 and 9.2 during this webinar.

Webinar Synopsis:

• Rocky Linux 9.2 and 8.8 GA Releases

• Who is Neil Hanlon?

• Who is Skip Grube?

• Does the Rocky Linux Open Source Team Get Paid?

• Why Are There Two New Releases?

• How is Support in Open Source and Support in CIQ Different?

• What Version of Rocky Linux Should You Use?

• What Is The Easiest Way to Get From Rocky 9.1 to 9.2?

• What Is The Statue of Enterprise Linux 9.2 For PPC?

• What is Next for Enterprise Linux for PPC?

• How to Fix Rocky 9 No Sources Error

• What Are The Main Differences Between Rocky 8.8 and 9.2?

• Are There Any Problems Upgrading From 9.1 to 9.2?

• What Does The Software Compatibility Testing Process Look Like?

• Is It Possible To Move From Rocky 8 to 9?

• Rocky 8 To 9 Upgrade With LEAP

• Are There Any New Computer Architectures That Are Coming To Rocky?

• How To Get Involved With Testing

• What Architectures Are Supported In Rocky Today?

• What Are The Best Practices To Enhance Security?

• What Is Configuration Drift?

• What Is Next For The Rocky Linux Team?

Speakers:

• Zane Hamilton, Senior Vice President of Sales, CIQ

  • Linkedin

• Rose Stein, Sales Operations Administrator, CIQ

  • Linkedin

• Neil Hanlon, Infrastructure Lead, CIQ

  • Linkedin

• Skip Grube, Senior Linux Engineer, CIQ

  • Linkedin

---

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Hello, and welcome to another webinar with CIQ. Rose, it's good to see you.

Rose Stein:

You too, Zane. Hello.

Zane Hamilton:

Hope you can hear me okay.

Rose Stein:

Yes, I can hear you. Excellent.

Zane Hamilton:

Excellent. So having some fun technical challenges this morning, afternoon?

Rose Stein:

Story of, I think, all of our lives across the entire planet as of 2023.

Zane Hamilton:

Seems to be a daily upgrading or whatever has to happen for every meeting tool that you open. So it's very true. Thanks to everyone behind the scenes, we appreciate it. We know it was a little stressful, but thank you.

Rocky Linux 9.2 and 8.8 GA Releases

Rose Stein:

We made it. We made it. I mean, it's actually an interesting segue into what we're talking about is a new release of the operating system that we refer to as Rocky Linux.

Zane Hamilton:

Two new releases.

Rose Stein:

Two, two new releases. Two

Zane Hamilton:

New releases.

Rose Stein:

This part, you're going to have to explain why, because it would make sense to me to just have one. But there has to be some really good reason to not only have one, but to have two new releases at the same time.

Zane Hamilton:

Absolutely. So who do we have to bring in today? Neil?

Rose Stein:

The man.

Zane Hamilton:

I see Skip maybe lagging a little bit behind. We'll get him here in a minute. So I think Rose's question is actually a really good question for Neil. It's actually a really good question for Neil to tackle after he introduces himself.

Who is Neil Hanlon?

Neil Hanlon:

Sure. Hey, I'm Neil Hanlon. I'm a Linux Engineer here at CIQ. Also one of the Infrastructure Leads for the Rocky Enterprise Software Foundation. Do we want Skip to introduce himself first?

Zane Hamilton:

Absolutely.

Neil Hanlon:

Tackle the question together.

Zane Hamilton:

Welcome, Skip.

Who is Skip Grube?

Skip Grube:

Oh, I guess so. Can you guys hear me? I had some technical difficulties here. I am Skip Grube. I am on the release engineering team for Rocky Linux, and I am also a Linux Engineer here at CIQ. And I wear a lot of hats, so I'm all over the place.

Does the Rocky Linux Open Source Team Get Paid?

Rose Stein:

Yeah. First of all, I know we'll probably do this again at the end, but I just want to say from us and everybody watching, I'm sure there's a big thank you, right? I mean, this is a lot of work. Like yes, you do have a job that you get paid for, but the Rocky Linux Foundation that you probably do not get paid for.

Skip Grube:

You're correct.

Rose Stein:

So, yes, thank you for that. Good work, keeping that open source, appreciate that.

Zane Hamilton:

And to everybody in that community that puts in a tremendous amount of time after your regular job. We really appreciate it. I know you guys are up and spending a lot of time on a lot of late nights and a lot of calls, so I really appreciate it. Thank you guys.

Neil Hanlon:

Yeah, it absolutely does take a village and I am personally very thankful for all of our community members as well. We have a really great team, great release engineering team, great testing team. Everything really that is supported by the community and by people that do this out of the goodness of their heart and don't get paid for it by their jobs or anything like that. So that's where I am certainly thankful for as well.

Rose Stein:

Yeah.

Zane Hamilton:

Yeah. There's a build team, the testing team, security team, documentation team, it's a lot of different teams.

Neil Hanlon:

It takes a village.

Zane Hamilton:

It does take out a very large village. So, back to your question, Rose.

Why Are There Two New Releases?

Rose Stein:

Yeah. Sorry. I mean there's so many things. I'm like, well what about this? What about that? And they can get involved in the community and go to Mattermost. And I just want to tell you guys like so many amazing things, but yeah, I am curious about the 8.8 and 9.2. Because that's specifically what we're talking about right now. So, Rocky Linux 8.8 has been released, 9.2 has been released. And so tell me why there are two of them.

Zane Hamilton:

And then blow her mind and tell her there's going to be three at the same time at some point in the future.

Skip Grube:

You want this one Neil, or you want me to start? I'm going to dive into it. So two's better than one Rose. Haven't you heard? So this is a common question actually, and I'm glad you asked it. So there are two releases, and like Neil hinted that there's going to be three. And if Rocky Linux had started with the Red Hat CentOS 7 era, there would be three right now even. And that's basically due to the absurdly long release cycles or, I should say, support cycles of this stuff. So Rocky Linux 9 released in 2022, right? Just to give an example. And we will be supporting it with updates and security patches and everything until 2032. I think May, is that right? Neil? May, 2032. Yep. Book the calendar.

But let's talk about 2032, right? By the time 2032 rolls around, we'll be older for sure, but also Rocky Linux 9 is going to be really long in the tooth then, right? And it'll be, as that date approaches, there will be less and less usefulness for the pieces that make it up, right? It will still have Kernel 5.14, which is the current one just supported with patches and so forth. Because the goal is stability, right? And that's great, right? But people are going to not want to, in the year 2032, right? People are not going to want to hop off of Rocky 9 and then hop onto a Rocky 10 that was released in 2032, right? That's 10 years. Like, that's three lifetimes in development in the software world that would be quite lurch, right?

You don't want your 2022 technology and then lurch into 2032 technology. That'd be crazy, right? Who knows what's going to be popular then. So what we do is, and our upstream Red Hat dictates this, we currently have five years, a three year life cycle or a three year new major version cycle. So it's very easy. So, in 2019, Red Hat 8 was introduced. Rocky was founded at the end of 2020. So 2021 we followed, we picked up on RedHat 8 then Red Hat 8, Rocky 8 supported 2019 through 2029. So 2019 Red Hat 8. 2022, Rocky Red Hat 9. 2025, Rocky Red Hat 10. 2028, Rocky Red Hat 11, et cetera. Right? And we introduce a new major version every three years? And so if you do the math, you can see the, there's a graph you can draw of this.

I don't have it handy. But yeah, there will be multiple major versions supported. And if you're brand spanking new and you're building something new, I'd recommend you get the latest one just because you'll have the longest support, right? That means like right now here in 2023 as we speak, Rocky 9 is the latest one supported until 2032. It would be a good idea to use that unless you have a good reason to use 8. Right? 8 is still supported until 2029, right? But personally, if it were me doing an install or a server farm or something, I would pick the latest one, if nothing else than to know that you have more time to move to another version 10 years from now or whatever.

Neil Hanlon:

The other side of that too is the support and life cycle of the underlying hardware. So a great differentiation between 8 and 9 is the fact that for the x86 architecture, the intel architecture they've bumped the micro architecture version. So there's specific optimizations in newer processors and CPUs that you're able to take advantage of by only building your software for those newer processors and just newer hardware. And so we continue to see that, I mean, even the past just 10 years, we've had huge advancements in the speed of what we're doing and how fast were able to do single core processing. And so being able to target specific optimizations in those newer hardwares allows organizations that need that speed to get the latest hardware or get the latest software, get the latest features and roll a little bit faster.

But at the same time, we don't want to just be creating a bunch of e-waste at the end of the day. And so those older servers are not only going to be needed by someone, say an organization upgrades and goes to somewhere else, but there's also organizations that are going to stick on that hardware for the entire life cycle of that hardware cycle, which could be five or 10 plus years, then some could be past the useful age. I mean, I've been at places and I know people that buy parts on eBay for old servers and old equipment. Because they're still running, they're still useful. But, for example, if you have an older server that doesn't have the micro architecture for Rocky 9, you're going to have to run Rocky 8 until you get new hardware basically. Or until an effort goes to rebuild something that targets that, or you have to switch to a different operating system in this specific case. But ultimately having multiple versions is generally a boon for the development and being able to take advantage of the latest features while still having that stable base to build upon.

Zane Hamilton:

Thank you for that Neil.

Skip Grube:

More common situation is you have to use Rocky 8. The database person that you're getting your database from says, well, Rocky 8 or bust, we don't support Rocky 9. What are you going to do?

Zane Hamilton:

And then they run it for a very, very, very long time. We still run across people running some Enterprise Linux 4, which was the end of life in the early 2000's. It's been a long time.

Rose Stein:

I mean that the term end of life has come up a lot in the last couple of years. And I think it probably means a little something different in what you were just talking about. And so I don't know, maybe if I'm going off on a tangent because it'd probably be fun to like to talk a little bit more specifically about what's going on with 8.8 and 9.2.

Skip Grube:

This is relevant though. This is very relevant, this is relevant. Now continue. Yeah.

How is Support in Open Source and Support in CIQ Different?

Rose Stein:

Right. So like this end of life and when you're talking about support and it being supported for 10 years. And you guys also both work for CIQ even though you spend a lot of your off time in the Rocky Linux Enterprise Foundation. It's like, what is the difference between the way in which you view support and the open source and the way that you view support at CIQ? Do you know what I mean? Because when you're talking about something being the end of life and it not being supported anymore, what is that like the word means?

Skip Grube:

I think what you're getting at is how does this tie in with CIQ's long-term support offering, am I heading close to it?

Zane Hamilton:

It's two parts of that Skip. I think it's also at what point in time these communities stop actually doing security patching or patching or releasing new features on a version.

Skip Grube:

In Rocky 8, assuming the world's still here in 2029, we are in it for the long haul. So the Rocky Enterprise Software Foundation will officially, like you can book it and keep checking us, obviously. But in 2026, 2027, 2028, 2029 for Rocky 8, you will see public security package updates for Rocky 8. Right? Until that date. Like we are committed for the whole 10 years. Well, I guess Rocky 8, technically it's like 8 years because we started in 2021. But, regardless.

Zane Hamilton:

On that date it's finished.

Skip Grube:

Yeah. And on that date, as far as the RESF, the public project, is concerned, let's call it May 1st, 2029, Rocky 8 is no more. It had a great run, but there will be no more updates. That's what we mean by end of life. Now, there may be other organizations like CIQ we'll find out at the time that say, hey, we have customers that really like Rocky 8, even though it's 10 years old now and they want to keep rolling with it. And those organizations, just like anything in open source, are free to continue supporting if they would like. But for the public project, no. Done, it's all over.

Neil Hanlon:

That's also true of minor releases for the Rocky Enterprise Software Foundation. We focus on, as CentOS did, the latest version that is being released. And so when 8.8 came out, there are no updates that are provided to 8.7 any longer. That gets moved to our vault, we're actually going to be moving it pulling it off of our public mirrors in the next few days for 8.7 and 9.1. So those will be moved to a vault. There'll be a snapshot of them there as they were. And they won't be modified, they're never to be touched again.

Skip Grube:

Yep. Yep. And for most users, you have to understand this is seamless, right? So if you have a Rocky 8.7 and you just installed it off the internet and you're running along, the next time you do an update, it will automatically update you to 8.8 without any, you don't have to do anything special, just run your normal updates. No big deal. Right?

What Version of Rocky Linux Should You Use?

Rose Stein:

So this is the difference. You're not maintaining the 8.6 anymore. No, you are, you're 8.7 and then 8.8 and then 8.9, and then 8.10. So when you're saying that you are maintaining it, it's not the specific version like you're providing the updates, but it's a different point version.

Skip Grube:

Always the latest version.

Rose Stein:

So you want to stay on 8, you're going to have to update to the other things. And so the difference with the LTS is if you want to stay on 8.6, for example, and you do not want any other updates unless something security wise has to happen.

Skip Grube:

This is excellent. Yes, exactly. So for, I would say the vast majority of users, even corporate users, the latest is okay, because you have to understand, especially in the Linux world, the difference between 8.7 and 8.8. It's significant. Yes. And we're here to talk about that, honestly. But it's not very significant though. It's not like a whole nother thing, right? It's significant in the realm of Enterprise Linux. But in the realm of Enterprise Linux, any change is significant, really. So we're the most conservative Linux out there really. And it's designed to be. And so for the most users, I would say the vast majority of users, this is fine, right? Just do your DNF update.Most people, when they update the 8.8, they don't even realize it, right? It's just a big update, right? You usually have 10 packages a week updated, and now there's like 50, right? And they do it and it's done and now, they're on 8.8, they don't even know, right? But however, there is a section of users, many times larger companies out there that say, well our thing is so sensitive, we really don't want to get off 8.6, or we really don't want to get off 8.4.

Neil Hanlon:

Or our vendor says that we can't because they only support that.

Skip Grube:

Yeah. Or they've been certified.

Neil Hanlon:

Or another common situation is often you'll have closed source binary drivers for some hardware that you're running in enterprises, and not so much anymore, but in the olden days, there used to be really common for binary firmware for NVMe drives and such.

Skip Grube:

Yep. And for those kinds of users, usually large corporations, it depends. But there's a lot of government sector and large corporate use going on there. So if they are just on the public community version of Rocky Linux, they're out of luck, right. Because the public community does not have the resources to continue supporting every single minor point release. It's hard as it is. And that's where companies like CIQ, you mentioned our LTS, steps in where we say, okay, you want to stay on 8.6, great. Contract with us and we will provide you with 8.6 updates so you can stay secure and continue running 8.6 for another year and a half or whatever the term is. And anyone's free to do that. Open source is wonderful like that. There's a lot of innovation in areas like that. You wouldn't think it was that interesting, right? But it really is neat that it gives corporations the freedom if they want to stick around on the older version.

Zane Hamilton:

And what that's really doing is just locking you into security patches or bug fixes in that version. So you're backporting Yeah.

Skip Grube:

Very minor steps.

Zane Hamilton:

Those very specific things.

Skip Grube:

Full disclosure, I am the lead on the CIQ LTS project. So I have a lot of personal experience with this. But yeah, our goal is just to keep it compatible with 8.6, change as little as possible, but still keep the user, the customer in our case, secure.

Zane Hamilton:

Anything you want to add Neil?

Neil Hanlon:

Nope. All right. Skip Got it covered.

What Is The Easiest Way to Get From Rocky 9.1 to 9.2?

Zane Hamilton:

So I think Art had asked the question about what's the easiest way to get from Rocky 9.1 to 9.2? I think Skip, you covered that in your 8.7 to 8.8. So the same thing. DNF update. Thanks for the question. So Jonathan asked a question as well. I think that this is a whole nother topic that we could get into. And if we want to do that now, I'm fine with that too.

Skip Grube:

This is Jonathan asking the question I'm dreading. You want to take this one to start, Neil, and I'll jump in when I need to.

What Is The Statue of Enterprise Linux 9.2 For PPC?

Neil Hanlon:

Yeah. So at a high level, the question is, what's the status of Enterprise Linux 9.2 for PowerPC? We had originally held back the PowerPC artifacts because of a bug that we found. We will be releasing those later this afternoon for a variety of reasons. It is complex and nuanced, and we're going to be having a post come out on the Rocky Linux site about what exactly this is and why it was held back and why it's being released and what we're doing forward. So I'm sure Skip can add some more details there. But at a high level the status is that they're coming out today and they should be available for use, and they're assuming you're using the production revision of the Power 9 firmware or higher, you should not have an issue with them. That is what we are seeing.

Skip Grube:

So I will add on to Neil's comment with the story of this, which is, I'll keep it brief, I promise. Yes, all good. So in release engineering, first of all, if anybody is confused, PowerPC is a processor type architecture, just like you have Intel, it's called x86. You have ARM processors that are found like the Raspberry Pi, some servers, your phone. PowerPC is another one of those. It's different. It's made by IBM, but it's another processor type. And yeah, we, at release engineering were running along very smoothly, very nice, and building our stuff, building our packages for 9.2. And we go to test the PowerPC build, and we produce an ISO. A DVD image of it, and we go to install it in both physical and virtualized environments.

And it crashes at the installer. Like you can't even get past it. Like there's an Anaconda, Anaconda is the installer, error. And we were we scramble for, I think it was a really rough 36 hours or so. And we were just like, oh my God, what did we do wrong? Like, we did something wrong. This is terrible. And then I got an idea. Because I took point on this a little bit at least in the beginning. And I got an idea. I grabbed the Red Hat 9.2 ISO DVD image. And I tried that and it crashed on the installer. And I was like, oh, well this goes deeper than we thought. It's not actually a Rocky Linux problem.

And we actually tried Rocky Linux, Red Hat, Linux, CentOS Stream, they all have the same problem. And it has to do with, I'm not going to go into too much detail, honestly, because more people know more about it than I do. But I noticed that not all PowerPCs are built the same. There are what they call revisions to the processor, right? And I noticed the ones that crashed are the 2.0 revision, which is it's called as it's marked as pre-release by IBM, but the QAMU emulator, which we do a lot of our testing in image building in actually Hat uses this. And so it was crashing on us. We noticed that on some of our IBM boxes, it's called Power 9 P series boxes.

It was actually fine. And those use the 2.2 or 2.3 revision. It's like a newer version of this processor. And so we opened a ticket with our upstream Red Hat. We taught, did a lot of research, and chatted with them a lot about it. They don't see it as a problem. And that's for us, it was definitely an issue because it was a clear regression, right? Like 9.1 worked, 9.2 does not, right? And so yeah, we did a lot of research about this, unfortunately some of our technical build processes for the images and for building out container images and ISOs and so forth rely on using QMU and this 2.0 processor and they're breaking because of it. So yeah, we pretty thoroughly researched the issue. It's a clear regression. It's unfortunate, but the answer is just not to use that old revision of the processor. Don't use QMU either. I mean, I don't know what else.

Neil Hanlon:

That's the other thing that we're, we're working on, I'm working on is talking with QMU developers upstream about the PowerPC the steppings, the revision of the processor here and what we can do to fix QMU so that it supports the proper revisions and, and isn't problematic here. So yeah, that's the other place we're going is addressing the bug in QMU with respect to it, not supporting the production releases of the Power 9 specifications.

Skip Grube:

And you, you have to understand the timeline of this too. So we're planning on releasing Rocky Linux 9.2 on that Monday. This was discovered Saturday, right? So we were just scrambling and it was a long weekend. Yeah. And it's release month, man.

Neil Hanlon:

Think it was Mother's Day?

Skip Grube:

It was Mother's Day actually. Yeah, so it was just, it's one of those things where we definitely had a talk in project leadership and was like, this delays a release, right? Like we are not releasing a Power PC when we know it's not, it doesn't work, right? Like you can put the DVD in your system and it won't install like it crashes. So, we decided it was unique. There's not a ton of PowerPC out there, especially when compared to the Intel in your normal desktop x86 normal servers for that matter. But so we decided everything else was great, it was fine. And we decided to hold back PowerPC while we investigated this, right? So it was a good exercise for us. We've never actually held back a single architecture and then released the rest of them. But there is no reason why people have to wait for their x86, their Intel processors when it's just this one processor that's failing.

What is Next for Enterprise Linux for PPC?

Rose Stein:

So where is it now? Is it still just hanging out trying to figure it out? Neil, did you say you're still in conversation with them?

Neil Hanlon:

Yeah, well, so our bug with our upstream Red Hat has been they've closed it because it's not technically a bug. It works on the production hardware, right? So they have access to IBM hardware that has the newer software.

Skip Grube:

IBM production hardware specifically. There are, yeah. Other vendors out there do not have many. And it's not popular, but we do know that in the wild, there is some hardware with this older 2.0 breaking revision, it's not likely to be experienced by a lot of people because you have to have this, not a lot of people do. You have to be interested in installing Rocky Linux on it, and you'll have to want 9.2. So there's a lot of stepping stones to it, but.

How to Fix Rocky 9 No Sources Error

Zane Hamilton:

Still a good catch. And I think we have another question that came in for you guys from Frank. We can throw that one up maybe. There we go. Telephony user. I have installed Dahdi. It's mandatory for an asterisk to jive with. Why does 9.0 say, I don't have the source. He's been asking the forum, and hasn't seen an answer yet.

Skip Grube:

Interesting. Neil, if you want to say anything I'm not super familiar with installing. I guess it's an add-on to Asterisk or something. It looks like, I'm not sure Dahdi, but if it's saying the sources, like if it needs source RPMs those are available in the vault, as Neil said. You want to throw up a link? Actually, I'll get you a link. We can throw it up on our YouTube chat or wherever we throw up links. Okay.

Zane Hamilton:

Throw it up in the YouTube chat.

Neil Hanlon:

We can take a look for that forum thread as well that you mentioned afterwards and take a look at that after too.

What Are The Main Differences Between Rocky 8.8 and 9.2?

Zane Hamilton:

So I know we've talked a little bit around the different versions that are out, but are there specific things in 8.8 and 9.2? Different features that came out that people should know about?

Neil Hanlon:

Yeah there's a handful. I mean, starting specifically with like Rocky related things or, or things that are specific for us is we've released a couple more variants of our universal base image, which is Mimicking Red Hat's upstream universal base image. We've made some changes to make that more aligned with what Red Hat is providing in their universal based image, as well as providing minimal and init variance of that UBI container. So those can be used for various purposes. The init one's supposed to be used for running multiple things inside a container, which is depending on who you ask a bad thing. But it's often a necessary thing. And then the micro one, I believe is just a much more limited set of features for the UBI that hold on top of.

Rose Stein:

Skip, you weren't on mute, now. You I love. There you go.

Neil Hanlon:

No worries. We have baby troubles. And the other big thing from Rocky or for 8 and 9 would be that we're now publishing our images for Microsoft Azure in their shared galleries, which is a new feature that's just coming out. Should be GA I think they were thinking, or they were telling me last week that it's going to be like mid-June that everyone can access that. But it's very similar to Amazon's community marketplace where you can search for images that are just available from the community that we have published. So Rocky, we publish all of our images both in the commercial marketplace for Amazon, as well as the community side where you can just search for them in AMI and use them that way. So this will be a feature for Microsoft Azure that's very similar to that, where you don't have to go through the marketplace subscription process and also get access to those images as soon as we upload them and make them live instead of waiting for the process to go through the commercial marketplace and step through all of the testing and approvals that are, are required for that, for good reason, of course.

But this will give us, at the RESF, the ability to publish more up to date images that users can consume without ultimately going through that marketplace. So though there are some changes across the board for 9, there's also a change that might be breaking for some people. I worked with the CentOS team for their network function virtualization special interest group to rebuild the open V switch and open virtual network packages and dependencies on top of Red Hat build routes rather than on top of the CentOS Stream 9 build route. We were having an issue with one of the projects I worked with OpenStack Ansible where a change that was made in CentOS Stream was incompatible with building it on top of Red Hat and thus Rocky and Alma and the like. So I worked with that team to rebuild those packages on top of Red Hat directly so that we will not have future incompatibilities with CentOS Stream in the future. So that's possibly a breaking change. It shouldn't really affect anyone as the packages are rebuilt exactly as they were. But yeah, it's definitely something to keep note of above and beyond any of the other changes and things that are happening in 8 and 9.2.

Skip Grube:

Goes to show the breadth and the width of the add-ons that we have. Because there's ones that I'm not even familiar with. I'm like, oh, I didn't know we did that like that. It's really crazy out there. It also goes to show that CentOS stream, while close to Rell and definitely what they call the trendsetter, it's not the same unfortunately.

Neil Hanlon:

In this case. Like it, it's actually a good story, a positive outcome, right? Because it was something that was actually coming in Red Hat 9.2. So it was the change that was going to be coming down in Red Hat 9.2 that came into CentOS Stream. It was detected by myself and some of the other folks in with OpenStack cancelable. But we were able to go and work on it and provide a solution before the release of 9.2 from Red Hat and Rocky.

Skip Grube:

That's great. And working as intended, it sounds like working

Are There Any Problems Upgrading From 9.1 to 9.2?

Neil Hanlon:

As intended. For sure. For sure. One thing to note if you are upgrading from Rocky 9.1 to 9.2 is that there's a smaller regression that we found in the LDM two package, which is a way to manage your file systems or partitions and file systems on your disks. In some environments, both physical and virtual there is a chance that your devices may not be found after upgrading due to a change in how an internal configuration file is read and parsed by the software. So there's bug tracking to fix that. And if you do use LDM, you want to consult the release notes for Rocky 9.2 to make sure that you mitigate that before you reboot after breathing.

Zane Hamilton:

That could be really exciting to have disappeared.

Neil Hanlon:

Yes. And hopefully it's not your desk. 

Skip Grube:

Shout out to our testing team who, like I said, picked up on this almost immediately, I think. Yep. And they test a wide variety of configurations, all kinds of hardware things that I wouldn't even think of. And yeah, we have good documentation about it. Just check out the release notes. There's a way around it. There's ways to check.

What Does The Software Compatibility Testing Process Look Like?

Zane Hamilton:

So what is that process? We keep talking about testing and catching things and it's becoming a really important process. What does that process actually look like? I mean, it sounds like there are a lot of things that get tested. Is that individual people clicking through on specific pieces of infrastructure? Is it automated? Like what does that look like?

Neil Hanlon:

So at a high level, I want to start where we as release engineering for Rocky begin, and that's on the release day or really even before for our building, the look ahead variants know we're constantly building the packages that are coming so that we have them available for when Red Hat releases them on that day. And so we have the betas as well. We try to build those before and ultimately we have them pretty successful. We almost had a 9.2 beta but didn't quite get there. And just went right with our release candidates. But so we start out really on release day when those packages get updated, we see that they're released onto git.centos.org. We see some messages come on the CentOS Fedora message bus, the message queue, and we go and important and build those packages.

We're working to close that gap as well to automate that. But we go and build those packages and we start to ensure that we have everything available for that release. And we've started to use Mattermost playbooks. The RESF uses Mattermost for our communications and a lot of our product management and stuff. And one of the features they have is this thing called playbooks, which allows you to add a bunch of checklists and steps with all different content that can be assigned to people and have due dates and run commands and all sorts of different things. But we've started to use those to organize all the testing and release processes that we have. So we have one for release engineering that we make sure that we get all of the packages built. We go and import the necessary things into our secure boot environments and build those packages and then all the way down until we're building the ISOs and testing them initially just a smoke test against them to ensure that they largely work.

We don't test all of the architectures. We probably would've been spinning our wheels for a bit longer if we had, but once we have the ISOs we kick off the testing team's release. And that's really the main release of Rocky. And that's a, it's like 12 or 11 distinct checklists of testing everything from comparing it directly to REL, checking the modularity. We have Katello Foreman, which is the upstream of the Red Hat satellite. They'll throw the repositories in our staging environment into those, to pull them down, make sure all the content's being sync to be due. So there's integration testing, albeit manual against that. So there is a lot of manual testing that takes place by the users in our testing team.

And there's also a lot of automated testing using openQA, which is a believed open Susa product, if I'm not mistaken. Not product, but software that Fedora also uses to perform quality assurance against their software. So the testing team takes the ISOs and the images or generic cloud images, for example, that we build and will load them into openQA, which will boot them and start running tests on them. It walks through the installer in different manners. It goes through a GUI installer, it does kickstart installs. I believe they have it testing, like installing software, making sure the software works

Skip Grube:

It's very cool because they will take Rocky Linux which we just produced with ISOs and what we call the staging repositories, the pre-release version basically. And they will automatically install it every which way, like 300 times and just to try this, try this, what happens if we did this, what happens if we add this option? Also on all this different hardware that you wouldn't even, I was like, I didn't even know this existed. But yeah, it's really quite impressive actually the level of effort that goes into it.

Is It Possible To Move From Rocky 8 to 9?

Rose Stein:

So you guys, what is like your best suggestion on how to make a smooth transition if you did want to go from an 8 to 9 and so is like a double question, is it a similar thing if they're going to be going from say, CentOS 7 to like a Rocky 8, is that as big of a jump from Rocky 8 to Rocky 9 and like how do you do that as easily as possible?

Skip Grube:

Yeah, so there's a lot of different minds about this. A lot of people will tell you different things. You guys, I think all three of you here know what I'm going to say. So, but I'll say it anyway. And that is taking a single system, right, that you have, let's say CentOS 7 on and going directly to Rocky 8 from your CentOS 7 install is a bad idea. I would not do it on any production system that I cared about, let's put it that way. If you're going to do that, there's a lot of advice around it. The only real way to do that is with what's called the LEAP framework. And the Elevate Project, which is basically an entire set of software designed around doing what you're describing because it's not, you can't just DNF update.

It's not that trivial. There's all kinds of breaking changes between those two. And really, in my mind, the way I see the LEAP framework, it was built out of a desire to do that as smoothly as possible, but it's fraught with peril though. There's all kinds of things that can make it go wrong. If you have a third party repository on your CentOS 7 machine, it's unlikely to work, especially if it replaces some of the packages that are in Stock CentOS 7. If you have CentOS 7 and you installed just the basic Apache and nothing else, and that's all you're running, you'd probably be okay, probably. But any deviation though from just the stock configuration and packages and even then sometimes it's still won't be great.

Rocky 8 To 9 Upgrade With LEAP

Skip Grube:

And the same goes with Rocky 8 to Rocky 9. There's also a LEAP upgrade path from Rocky 8 to Rocky 9. I personally would not recommend it. I can't, it's just, I've seen way too much nonsense happen from trying to do that because you have to remember these releases, let's call it CentOS 7 was released in 2014, right? It's supported through 2024 and it is based on Fedora 19, I think. Is that right, Neil? That sounds right. I said 19. Yeah, yeah. Fedora 19. All right. Wow. And Rocky 8 and REl 8 were released in 2019, which is five years after 2014. Right? It's a long time in the open source world, and it is based on Fedora 28, I believe. And I ask someone, if you have a Fedora 19 system, if you were interested in upgrading to Fedora 28, right?

Any user of Fedora would tell you don't do it, just wipe it and install Fedora 28 and set your stuff up. And that's right. They'd be right. Because you could, you might be able to pull it off after like two nights of work, but that would be insane. Like to try that. And that's what you're doing though when you go from CentOS 7 to Rocky 8, because they're that far apart, right? And ditto for Rocky 8 to Rocky 9 is honestly closer 2022 to 2025, but it's still, what is that six, six or seven Fedora releases between them. So it's dangerous , right? I would thoroughly test and back up stuff before you try it. What I recommend people do and what I love to see is instead of spending all this effort to do all this hackery to get your upgrade done, build out some automated processes where you can just push a button and a new Rocky 9 virtual machine comes up and it's ready to go.

It's got all your stuff installed, it's got all your company's things deployed or whatever, right? And that's configuration management is what I'm talking about. Things like Ansible, Puppet, Chef, SaltStack, there's some other ones out there, but just scripts.

Zane Hamilton:

And those also seem to break from version diversion over time. Whatever your favorite way of doing it is.

Skip Grube:

We've got some servers over here running let's say Rocky 8 and I can take, I can spin up 50 Rocky 9 web servers using my automatic process in like two hours, right? Or less. Right. Do that and then have an automatic process to copy your web data over and turn the 8 ones off and you're upgraded. It's great. Replace them. Right, right.

No, no, you're, I'm, I'm good Neil. I've blabbed enough about pet peeve of mine. I've blabbed enough.

Neil Hanlon:

Yeah, absolutely I agree. There are ways to upgrade, but when you go and abstract the configuration out to something where you can apply that wanted state out to whatever the server is on the other end, you not only gain the ability to upgrade and move that because you can have your configuration be varied based on what version of the softwares you're running. But you also get a lot in terms of your security posture and generally your infrastructure as a whole being more stable and known when you're applying this known state and don't allow for configuration drift. That is one of, if not the most beneficial things to using some sort of configuration management. Even if that is just bash scripts that get applied. Like, I literally wouldn't recommend that, because you can use Ansible and do the same thing and half the time. But if that's what you want to do and you have some bash scripts that could do your configuration and just run them with Ansible and you're on your way there, it's well worth the time to abstract that out versus trying to figure out the nuance of upgrading.

Zane Hamilton:

I feel like Greg is just screaming Pearl in the background. Or Pearl or Pearl!

Skip Grube:

But definitely, not only will you be able to do your upgrade, it's really like a replacement and then you decommission the old one, but you get side benefits too, like the ability to be more secure, to be more consistent in your environment and so forth, right? All this great stuff comes with that. And you get upgrades for free, basically. So.

Zane Hamilton:

It gives you DR Built in too. Yeah.

Skip Grube:

Yes. Actually

Neil Hanlon:

That's the other side too. Recovering from disasters, configuration errors, whatever it is.

Rose Stein:

So just like, to be clear, would there ever be a situation where you did a DNF update and it went from 8 to 9?

Neil Hanlon:

Not unless you misconfigure your system.

Zane Hamilton:

That would take some effort.

Skip Grube:

We have had some people who have attempted it. It's effectively what the LEAP product or the LEAP software I'm talking about does, but it does it very carefully though. There's all kinds of pre-upgrade checks. It runs and it does some stuff because it knows it's going to have problems if you just try that. Right? And no, by default, no. Unless you really know what you're doing and go in and muck with your repo files, that will never happen. You will be 8 on 8 and 9 will be on 9 unless you really change it. So you have to intentionally do that.

Rose Stein:

All right. Thank you

Zane Hamilton:

Neil chuckles, like he's tried this before, just for fun.

Skip Grube:

Nice break.

Neil Hanlon:

Why not? Sweet. Might as well.

Are There Any New Computer Architectures That Are Coming To Rocky?

Zane Hamilton:

We talked a little bit earlier about architecture. So we were talking about PPC. Are there any new architectures that are coming along that we are going to start seeing?

Neil Hanlon:

Anything in general or on Rocky? ?

Zane Hamilton:

On Rocky. For Rocky? No, no. I know you could talk about RISC-V for a while if you wanted to, but are there other new architectures

Skip Grube:

I'm going to jump in on Neil. Sorry. Not officially supported. When Rocky 9 was first started, we set out to support the same architecture that Red Hat Enterprise Linux, which we're a clone of Red Hat Enterprise Linux. So that's Intel, the old x86 that we're used to. That's aarch64, ARM processors, 64 bit PowerPC, which caused all the trouble that we were talking about recently. And then S390X, which is IBM's mainframe series of processors. So you can run Rocky on your mainframe, which I was surprised to learn. Several people do. It's not unheard of. Outside of that, there's a lot of other processors out there. We have what's called the Alternative Architectures Group, which I'm a founding member of, thank you.

How To Get Involved With Testing

And we cover a lot of things. A lot of the group is spent on making Rocky compatible with Raspberry Pi and other ARM-based boards, which is easy because we basically have to get it to boot and then we just use the regular Rocky packages. We already installed them for ARM. There are rumblings and work out there. You have to dig into it a little bit to find it about taking Rocky and porting the entire thing to brand new architectures. RISC-V is one. I'm actually part of that effort. It's touch and go a little bit. It's very difficult. Because there's nothing to go on. There's, there's some fedora packages that we have to bootstrap off of and it's here they are and good luck. Right.

And there's a lot of package issues that are being worked out. Because remember, there are 3000 packages in Rocky Linux 9, and you have to build every single one of them if you want a full stack. So there's a lot of work to do there. Whereas something like Raspberry Pi, we just take advantage of the ARM64 packages that are already built as a matter of course. Right. And then I think ARM32 bit is also being worked on out there. And we've got some rocky ARM32 bits. Again, this is not usable at all right now. It's very you have to really be into it, and also have some hardware that runs it if you want to get involved. But yeah, if you're interested, come ask about it and hang out with the Alternative Architecture Group on Mattermost our chat area. We're always happy to talk.

Zane Hamilton:

I think that's where Dave needs to go. He wants to see Rocky for Cubits.

Skip Grube:

Good luck. What even is that? is that a processor architecture? Quantum. Oh my goodness. The whole Quantum. I'm on Wikipedia now. Go ahead Neil.

Zane Hamilton:

It's funny you get between Dave on Cubits and then Art, wanting to see it for his old x86 stuff.

What Architectures Are Supported In Rocky Today?

It's different places there. So remind me what architectures are supported today? PBC, x86.

Skip Grube:

ARM64 or Arch 64, but it's ARM processors. And S390X and so there's four of them. And in descending order it's probably x86 by a mile, ARM. people use it like Raspberry Pis, ARM servers less popular catching up.

Neil Hanlon:

And it can give me a desktop that I can buy for less than $1,500.

Skip Grube:

Oh yes, please, please. We love it. PowerPC I think, and then probably I would imagine S390X mainframe is bringing up the rear there, but I don't know, actually I don't have good numbers on that, because those last two are pretty, they're pretty specialized. Like you have to be a business that really wants to, if you want to do that. So hardware's out there, but you're going to be paying some money.

Zane Hamilton:

If you really like expensive hardware. Yeah.

Neil Hanlon:

Yeah, I think the Power 10 hardware starts out at like $35,000 for one machine.

Zane Hamilton:

No varied entry, none.

Skip Grube:

Everybody empty your pockets.

Neil Hanlon:

Just rent some time on an L part. It's fine.

What Are The Best Practices To Enhance Security?

Rose Stein:

All right guys, so do you have anything specific from the perspective of the Rocky Linux community to tell people about best practices that are recommended to enhance security of their deployed systems?

Neil Hanlon:

What I think of is just the Ansible hardening role. It's an open source thing to harden your systems. It applies for whatever operating system you're running on, and it generally conforms to your best practices for the security of your system and making changes to that. Above and beyond that you can look into the specific security roles provided by the SCAP security guide for OpenSCAP, which is Security Compliances Code. That's selectable while you're installing a system. So you can apply things like CIS, information security levels one or two, STIG, there's PCIE, HIPAA, all sorts of different compliance profiles for security that may be required and can be applied at installation time or afterwards depending on the profile to generate reports and then apply the configuration medications to make your system in compliance.

Skip Grube:

I will go more general than that. And Rose, you're not the first person and not even the 12th person to ask me that question. And I would say what we talked about earlier, configuration management is better than any patch, than any security framework because it allows you to specify and know for a fact like what your systems are running, right? Like you've set in stone, no matter how many, you have 10 systems, a thousand systems, whatever, and security is a process. It's not a product or a button you push or whatever. So it's a way of thinking and configuration management goes a long way and not swapping in 9 for 8 and doing an update or whatever. Also not recommended.

What Is Configuration Drift?

Rose Stein:

Yeah. Neil, you mentioned a word, I think you called it configuration drift, is that accurate. Yep. So is that what these management tools help with? And what does that mean? Drift? I mean, when I'm thinking of drifting, I'm thinking of just like, I'm going in this direction, but now I'm going over here and it's like, wait, you gotta get back over there.

Skip Grube:

It's pretty much the same thing. I have stories and I know Neil has stories too. We both worked before doing Rocky and even during doing Rocky now we've worked in industry and in IT work. And the story is as old as time. Let's say you're a man, you're a guy and you're managing a hundred Linux servers, right? And you wake up one morning and five of them are out of commission, right? They're just not working. They're broken. And you dig in and you figure out the problem. The problem was somebody went in or somehow the network time thing, right? The network time damian, that controls your PC's clock, right? It talks to the internet and it controls what time it is on your computer. It got mucked up, right? For those five systems, it got changed around and it's not working easily.

Zane Hamilton:

Got patched on something by accident.

Skip Grube:

Yeah. So you go and you're like, what the heck? And you fix it, right? And it caused a problem because some piece of software required the correct time and it completely broke without it. Right? And you look around, you're like, what? And you start looking through logs and then you find out, oh yeah, the developer group from Product A jumped in, modified the TZ stuff because they thought it would fix something and then jumped out and it broke three days later because it took that long to drift the clock or whatever. But if you had a configuration management process that maybe every hour would go through all your systems say, okay, is the time zone data correct? Set correctly? Is your network time set?

Yes. If not, I'll fix it immediately. Right? Then that's what we're talking about. Right? That's an example of configuration drift because somebody went in and changed something, but just on a few servers. Right. Or just on a particular one and it broke something. Because you can imagine if you have a hundred servers that you manage and everybody goes in to everyone once a week, like all of a sudden you're going to have a problem. And you have to go find them and rally them, right? And so yeah, these tools that Neil is talking about, they control all that. They say, okay, we're going to enforce a policy saying this configuration file will look like this. The updates will be applied at the same time and we will run this version and everything will run the same. And that way we don't have drift, we don't wander around trying to fix things. Well said.

Zane Hamilton:

Well said. They really are a problem and have experience in this. By the

Skip Grube:

Way, this, this story is the names have changed, but it's true. Like yes, the names have been changed. Yeah. To protect the guilty really of it.

Zane Hamilton:

You have operation centers that follow the sun. You have multiple people touching multiple things and fixing.

Skip Grube:

Exactly. Yeah.

Zane Hamilton:

It's super fun.

Skip Grube:

And as someone who is responsible for managing several hundred servers Yeah, yeah. You don't want that to happen because you'll go crazy .

Rose Stein:

Hmm. I think we could probably chat forever. Are there any final questions that you have about Zane?

What Is Next For The Rocky Linux Team?

Zane Hamilton:

No, I was just curious, I know you guys have been, you guys, I mean the RESF has been traveling to a lot of different things, part of a lot of different events. What's coming next? Where can we find you next?

Neil Hanlon:

Let's see, what day is it? Okay. We are going to be at Salty Linux Fest the second week of June. That's the 9th through the 11th. That's in Charlotte, North Carolina. And then we're going to be going right after that to the Open Infra Summit, which is in Vancouver, British Columbia. And we'll have a booth there. We'll have a swag at both of these events. So if you're in the area, please come find me and I will give you stickers for free.

Skip Grube:

Neil is a traveling road show. I will be in Charlotte because I live like a two hour drive away. I will not be in British Columbia though. I'm not.

Neil Hanlon:

That's what you think

Skip Grube:

Yeah, that's what my wife thinks too.

Neil Hanlon:

It's going to be a rude thing for you when I put you in a suitcase.

Rose Stein:

So how do they get a hold of you? like what's the best place to get involved with the RESF community?

Neil Hanlon:

Our chat at chat.rocky linux.org. It's also on Matrix and on IRC. We're bridged over to those places. So you can find us there. We also have forums, forums.rockylinux.org. Yeah, if you're interested in joining and collaborating, please come in and take part in the community. Come ask questions and ask how to get involved and we'll help find a place for you.

Zane Hamilton:

That's great.

Rose Stein:

That's perfect.

Zane Hamilton:

We're up on time, Rose. That was very funny.

Rose Stein:

I know. Very funny. There. I know. So now it's time to like, it's time to subscribe. It's time to make sure that you stay in contact with us. Boom, boom, boom. Thank you Skip and Neil and everyone at the RESF and actually anyone who is involved with any open source projects you keep this tech world going and flowing and we appreciate you. So thank you so much. We'll see you at the same time, same place next week.

Zane Hamilton:

Thank you everyone.

Rose Stein:

Bye.