Have you heard Rocky Linux 8.6 has been released?
Webinar Synopsis:
Speakers:
-
Zane Hamilton, Director Sales Engineering at CIQ
-
Gregory Kurtzer, CEO at CIQ
-
Skip Grube, Technical Solutions Architect at CIQ
-
Neil Hanlon, Solutions Architect at CIQ
Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.
Full Webinar Transcript:
Rocky 8.6 [00:00]
Zane Hamilton:
Good morning. Good afternoon. Good evening. Welcome to another CIQ webcast. We appreciate you joining us today and spending the time. Today we're going to be talking about Rocky Linux. We talked about Rocky Linux quite a bit in the past, but something new is coming out and we want to spend time on that. So today we have Skip Grube. Welcome back.
Skip Grube:
Glad to be here.
Zane Hamilton:
Thank you. So while we're waiting for Neil, Skip, you haven't been on before. Why don't you tell us about yourself and kinda what you do here at CIQ and what you do for the Rocky community?
Skip Grube:
Yeah, so I recently joined CIQ but as far as Rocky Linux goes, I kind of fell backwards into it, to be honest with you. I have been there from the start of the project. It was something I was very interested in pursuing and I met up with some of the early contributors and here I am in release engineering. So I'm a package monkey. I go through build, the bug, all kinds of packages from across the distro.
Zane Hamilton:
That's fantastic. Thank you. Neil, better late than never.
Zane Hamilton:
Welcome back, Neil. Introduce yourself again, just for those who haven't seen you in a while.
Neil Hanlon:
My name is Neil Hanlon. I'm a solutions architect here at CIQ and also in my abundant spare time, I run the infrastructure team for the Rocky Linux community.
Zane Hamilton:
Yeah. I love the spare time comments because I know you don't have any spare time. You barely get any sleep. I know Skip's in that boat, too. So one of the things that's come out that I wanted to talk to you guys about today is Rocky 8.6 coming out. Excited to hear what's new in 8.6, and what are we doing?
Neil Hanlon:
Yeah, it's exciting. And it's also, kind of not exciting, which is a theme. It's nice to have new things, but those new things don't change that much. And that's what we like about Enterprise Linux. But there are a handful of new things that have come and changed in 8.6. PHP 8.0 is now available directly from a module, as well as Perl 5.32. Previously, we'd have to go and usually get PHP 8.0 or newer versions from like Remi repo or one of these external third-party repositories. But now PHP 8.0 is right there. Also, a new version of BIND with some additional fixes and changes to make the experience on RHEL at least be able to be matched with the upstream versions of BIND, with new features, especially around DNS over HTTPs and some of those other sorts of features there.
One big change, too, is Ansible. So Ansible has recently changed over to the Ansible core module, or not module necessarily but package naming, so you may experience some upgrade troubles as you go if you already have Ansible installed. So you might need to go and swap that over to Ansible Core. There's a bunch of new modules: there’s a Log4j module that was brand new, as well as just some updates to other modules. It's exciting, and there are a few things to note from there, but in general, it's kind of a solid point release with not too many problems that folks should be comfortable and able to upgrade to.
Zane Hamilton:
And we're talking, simple as a DNF update.
Neil Hanlon:
Yeah, DNF update -y. Maybe that Ansible problem might hit some people where they might not be able to perform an upgrade because of Ansible being installed and the package name changing. But in that case, you could either just do a DNF swap for Ansible to Ansible Core, or you could remove the package, or one of those other options there. But it should be just a DNF upgrade and running on the latest 8.6.
Zane Hamilton:
Very nice. That's more of an Ansible upgrade than it is a Rocky upgrade anyway.
Neil Hanlon:
True, true.
Zane Hamilton:
Thanks, Ansible, for changing everything. Appreciate it.
Neil Hanlon:
Yeah. The only constant is change.
Rocky 9 [04:17]
Zane Hamilton:
Yeah, of course. I'm glad that there's not a lot of change with 8.6. That's great to know. I'm glad that it's boring because that's exciting. But I think one of the other things that I run across a lot, and I'm glad Skip is here to talk about this, is when we start talking about 8.6 and moving forward, a lot of people really soon start asking the question, “What about 9?” So what can you tell us about Rocky 9?
Skip Grube:
We're gonna talk a little bit about 9 here. So first of all, I want to echo what Neil said: everyone is excited about 9, I'm excited about it, but it is also boring. We are in the Enterprise Linux space and we are not out there. We are not Arch Linux. We are not Fedora. We do not move swiftly like those distros do. We don't break things occasionally like those distros do, and that's okay. If you're really out for the latest and greatest, these enterprise distros, you might wanna try something that's a little more of a preview. If you're a business or if you want something stable for 10 years: welcome, nice to have you.
We're boring and that's a good thing. That's what I want to get across. It's going to be great. Now looking forward to 9, one of the behind-the-scenes things that happens is a new build system, which is part of our giant effort towards this new build system called Peridot. I think a lot of people on the stream have probably heard of it, at least in passing. It's going really great for us. There are some growing pains because new things are difficult. And as you can imagine, building a distro is an undertaking, right? There's a lot of stuff to put together and compile.
But it has a lot of new features that we have really wanted and desired. Just some highlights, I'm just gonna give you the real high top-level view of it. Inserting specific versions of dependencies has become very easy for us, which is something that kind of plagued us in the past. a lot of these packages are very fickle in the way that they're built. They're very picky about certain minor versions of dependencies at build time. And Peridot has made that a lot easier for us, which is great. Another feature is, it's going to allow us to integrate SIGs, which we call special interest groups, much closer with the distro than we have previously been able to. We'll be able to build our special interest group packages right there with the main distro, right in conjunction with Rocky 9. It’s very exciting. Also, in the future, we plan on porting the Rocky 8 builds to it as well. And I think those SIGs can also benefit from that switch over.
Perl 5.32 [07:35]
Zane Hamilton:
That's really interesting, Skip. There's one thing I want to ask a question on. First of all, I see Greg joined us. Thank you, Greg. I'm assuming when you heard Perl 5.32, you got excited and decided to jump on.
Gregory Kurtzer:
That was exactly it. I just figured that was Perl, oh yeah, I'm there. I write really good Perl; it's like holding the shift button on my keyboard and dragging my elbow back and forth across the numbers and you can get some really great Perl code that way.
Peridot and Rocky Linux 9 [08:01]
Zane Hamilton:
You're just trying to make it where nobody else can come back behind you. Typical Perl developer. Love it. So Skip, whenever you're talking about the dependencies, it used to be hard and Peridot was gonna fix some of that. Can you dive in a little bit on that? Why was it hard or what made it hard about the dependency that this system's gonna fix a little bit?
Skip Grube:
Let's take Ruby, for example; this is something that happened two days ago. Ruby, we all know and love, right? Just like Perl, everyone loves Ruby, right? Ruby is a package in Red Hat Enterprise Linux 9 and Rocky Linux 9. We go to build it and we get errors. There's test errors. There's automatic tests that run once the package is done compiling and I examine these and I say, “Hey, these look like OpenSSL errors. What is going on?” There's something wrong with the way it's talking to OpenSSL. I go down, after much troubleshooting, much digging through some source, and it looks like the Ruby package was built for CentOS Stream and for Red Hat 9 against OpenSSL 3.00-5.
And that worked because the test passed. But now, there's been an update and OpenSSL 3.00-20 has been released as a patch, as a new package. And when you build Ruby against that OpenSSL, these tests fail. And this comes up a lot. You'd be surprised because every time that an OpenSSL update is released, Red Hat, CentOS, and other distros do not rebuild the entire system. Previously, Ruby worked and it was built and it's fine, but now it fails because another package was updated somewhere, and it’s super frustrating, but this is what we do. This is a large part of what I'm about, by the way. It sounds rough, but we get along.
So we have to either fix the Ruby test, and patch it in some way to not fail or not even run the test, which is not desirable. Or we have to build Ruby at build time with the older OpenSSL version. And previously that was dicey, that was not easy to do. We were effectively using the standard Fedora CentOS build tools, and it worked, but it was hard. And there were some tests that had to be patched out in the past. We wait until we report the issue to Red Hat or to CentOS upstream and hopefully a fix will come in the next version of Ruby and it'll build okay. Until the next time they update OpenSSL, et cetera.
So this is an example where we can say, “Okay, well, we're going to build Ruby, but we're going to inject the older OpenSSL into it.” And it will allow us to match Red Hat Enterprise Linux 9 closer, which is our whole goal, like the whole purpose of the project. This happened when 8 was coming out as well – I’m sorry for everyone who wanted, “We should put this in Rocky 8!” Our charter and our core mission remains: we will rebuild bug for bug compatible, Red Hat Enterprise Linux exactly. We’re religious about it.
Neil Hanlon:
That's a great point. I was going to bring that up as well. It's not just that these things are sometimes more difficult to build in terms of tracking the dependencies and everything else, but there is also something that's important to the project overall. We don’t just ship the version of Ruby that is in RHEL, but we also build it in the same manner. So we're building it with the same dependencies, like that OpenSSL version, even though that OpenSSL version isn't going to end up being distributed in Rocky 9. Previously, we were able to do this with Koji and the Fedora build tools by doing some side tags, importing packages, and some manual work. But the reason for the feature in Peridot is that it eliminates the human process there and makes it easier for us to be able to automate those builds in the future, by just checking what the dependencies are, how they were built, how they need to be built, and then rebuilding them.
8.6 and Ansible [13:01]
Zane Hamilton:
That's great. We have a question. So other than possible changes from Ansible, any known breakages? And this is probably for 8.6. Sorry, we're gonna go back. So Neil, are there any other things that we should pay attention to, or things that we know that get broken?
Neil Hanlon:
There are a few things that are still outstanding with the 8.6 release of RHEL. One of those things is it looks like some possible regression in conntrack or how the kernel is doing connection tracking. So if you're using Rocky Linux or Red Hat as a router, you may experience some troubles there with your traffic being blocked, if it comes in and out of two different interfaces. I've been tracking that one, trying to get some more detail on it. But in terms of breaking changes or problems there, no, there shouldn't be any other than some progressions that are being worked on.
Podman Is a Useful Tool [14:00]
Zane Hamilton:
Very nice. Thank you. Another question: despite Rocky not moving as fast with Podman, one can run the latest tech, as well, in a secure, rootless, sandbox environment. It's more of a statement than a question.
Neil Hanlon:
No, it's absolutely a great point. Podman is a super useful tool. I use it all the time. I'm using it right now to run a bunch of random stuff on my computer and laptop and everywhere else. And I think that's a great statement, as well, because it shows that you can run the latest and greatest stuff on a stable operating system that doesn't change. And you don't have to add to your list of worries about something in the host operating system being critically flawed or broken in some way. You only have to worry about that with the applications running.
Zane Hamilton:
Absolutely.
Skip Grube:
Addendum to that, as well. For the desktop-minded among you, I know we have a lot of users that use Flatpak if they want the latest desktop applications. It's a wonderful tool.
Zane Hamilton:
Absolutely. I think we've talked about that before, Greg, with other tools that we work with as well, and being able to do that with Apptainer, it's pretty cool and interesting stuff as well.
Gregory Kurtzer:
It’s the exact same use case on the HPC side is pretty much what we're talking about now. And Singularity/Apptainer has been a phenomenal tool for that in the HPC space. You can probably imagine HPC centers that have many thousands of nodes in one system are extraordinarily resistant to making any sort of updates or breaking changes to that underlying operating system. Typically, once they integrate a big system, the operating system doesn't move very much for the life of that system, because of all the different kernel modules and support and whatnot that they have to manage. It's just such a fantastic point that containers and having an ability to run these containers on top of this stable operating system is so critical.
One thing I wanted to mention earlier to a point that Skip made is that it is absolutely 100% our charter to be bit for bit, bug for bug level of compatibility with the Enterprise Linux family of distributions, obviously with Red Hat Enterprise Linux setting a lot of that standard. And now that CentOS Stream is kind of the parent of all of us, that actually puts it in a really kind of cool position, and gives us all the ability to put emphasis, code, and support back into that single community that we are all now benefiting from. But I do want to say that there's another piece of this, which is that we have had a lot of people asking for optimizations and enhancements on top of that core base foundation. Now these would always be opt-in enhancements; they would not be default. But you would be able to do things like install new packages or install custom packages based on the special interest groups that we have within Rocky. And we have a couple big dominant special interest groups that are underway right now. We've got SIG Cloud, there's SIG Kernel as well to get a more upstream mainline kernel associated with it and there's others. So if this is something that you're interested in, anybody in the community, please come and take a look and join in our Mattermost and/or IRC, and be part of these special interest groups, because it's a phenomenal way to add value over and above, just that bit for bit and bug for bug level of compatibility. So yeah, join us.
Mainline Kernel in Enterprise Linux [17:31]
Zane Hamilton:
So Greg, you mentioned the mainline kernel version, and I think it's an interesting conversation I've had with several people, and it's exciting to some, but I guess there's other people that wonder why? What benefit do they get out of doing something like that? If you had a mainline kernel in Enterprise Linux, what does that give you?
Gregory Kurtzer:
So there's a number of different aspects of this. So an obvious one is greater hardware support. If anyone's tried running an older Enterprise Linux on a newer laptop you're going to immediately have various pain points that you're going to have to work through depending on the laptop. In most cases, these have already been resolved in the upstream kernels and the one patch into our standard Rocky kernel that I've bugged Neil to see if we can get in there, is for this cool little laptop that I have right here, which is a Dell XPS 15. It works great with Rocky Linux, right out of the box with one exception: the sound driver just hits the tweeters and doesn't hit any of the other speakers. So all the sound sounds totally tinny.
And I just basically took a few lines out of the main line kernel and threw that into a patch. And I was hoping to get that in there, but at the same time, I'm kind of secretly hoping Neil actually says no to me, because to Skip's point, our goal is bit for bit and bug for bug level of compatibility. But this is exactly where a special interest group would be fantastically useful. And that code from the kernel, I didn't write it. I just looked at the current code, found that piece, and I'm like, that's what needs to go in. And luckily it integrated very easily. So that's what I'm running on this, but we actually have someone else on the team who constantly gives me crap about the fact that he's running a newer mainline kernel than me on his same XPS 15.
And he's actually running the mainline kernel and he’s also gotten performance benefits because that mainline kernel was built to support newer processors, so he is actually getting better performance out of that. There are some other aspects as well that people like about it, especially people that are looking towards real-time workloads, maybe some of the different security protocols and security methods that the upstream kernel is doing differently from the Red Hat and Enterprise Linux kernels. So there's a number of reasons here why people may really prefer that upstream kernel. And of course, there's API benefits; there's file system performance benefits. And these are things that I've just heard people bringing up. That's one of the reasons why I think having an updated kernel as an option inside of Rocky is incredibly valuable.
Before I hand over the mic, Greg Kroah-Hartman, who is the maintainer of the stable upstream kernel, is actually helping us with this. So he's actually part of this process and special interest group to ensure that what we're putting out there is going to work. And it puts him directly in the loop so that if anybody finds any bugs or issues, he can resolve those. And so that's a win-win for not only Rocky, but the community in general running Linux.
Zane Hamilton:
Absolutely. That's very exciting.
Neil Hanlon:
I think one of the biggest things that would come to mind for me is like 12 Gen Intel chips or some new CPUs that are coming out. The first support for those things goes into the mainline kernel and will eventually get backported and moved into other places. But as you said, that's a great place for SIGs to be able to work across Enterprise Linux as well, to enable those sorts of things and work to get those features backported into the CentOS Stream kernels and eventually into Rocky.
Zane Hamilton:
So Neil, how long, or how far back are we seeing from Enterprise Linux kernel to mainline? How much of a lag is there? I mean, from what I recall last time I looked, it was like 12, 18 months. Is it kinda better?
Neil Hanlon:
I think that Stream has helped to make that process more visible. And I think that part of it is just that there's some growing pains and stuff that people are learning. And also, there's a history of folks compiling their own kernels and doing things themselves just because, it can take that time and it wasn't necessarily as accessible. In fact, you know, kernels were almost entirely inaccessible to be modified except via a request from Bugzilla with Red Hat. So now I think that we have this contribution opportunity via CentOS Stream, I'm not sure if we've put any patches in to track how long they take to get through. I think that would be something that would be interesting to look into and to evaluate just how it is to contribute to the projects and work on those sorts of things. But as Greg mentioned, you know, the SIGs that we have on Rocky and especially with our new build system that's coming out will really enable us to do some of this iterative work before it gets upstream, and before we go and open those pull requests.
Zane Hamilton:
Very nice.
Gregory Kurtzer:
I'd add something to that as well, which is: one of the goals of Enterprise Linux is to have a completely stable API ABI as the default scenario. So Zane, one way somebody might read your question is, how far behind is it from upstream? Because there are no breaking changes regarding API ABI, I'm actually planning on having a follow up with Skip on how OpenSSL and update an OpenSSL broke a build if it's supposed to be API ABI compatible. And is that a bug that we need to actually be thinking about?
Neil Hanlon:
Yes.
Gregory Kurtzer:
The other aspect of that is once any version of Enterprise Linux is released, think of it as, that is now a fixed set of versions in time. All of the packages, this is not strictly adhered to, but it's pretty close. All of the versions are pretty much stagnant at that point. As I said, there's a few edge cases in which they do update, but generally speaking, all that core is going to remain exactly the same. So the kernel that Enterprise Linux is released with, the major minor patch level version, is not going to change for the entire life of the operating system, which is for 10 years. So you have quite a while there. There are going to be backported fixes, security issues, and whatnot, and the relevance of those fixes is going to change over time.
So for example at some point it will be everything that's broken needs to get fixed. And at some point it's going to be, you know what, we're just doing security updates; we're just doing critical updates. And so that kind of changes over time. But this is why a new release is such a big deal in the Enterprise Linux community. That’s why we’re all super excited about grabbing that, and as soon as it goes beta and then stable, throwing it on everything that we have because we know we have that absolute stability.
Zane Hamilton:
That's very cool.
Skip Grube:
I just want to note that Greg's absolutely right. We will be building kernel 4.18 for Rocky 8 in the year 2029.
Where Is Rocky Built? [25:50]
Zane Hamilton:
So one of the other questions that comes up quite a bit is: where does Rocky actually get built? From a community standpoint, that's a really interesting question.
Skip Grube:
Yeah. So physically, rubber has to hit the road somewhere. We can abstract things away, but we must run instructions, and the primary place is in Amazon Web Services. AWS, perhaps you've heard of it. It’s a large cloud provider. They’re known among some. The Rocky Enterprise Software Foundation enjoys a donation from them of an AWS account in which we conduct a lot of our builds, including a lot of our infrastructure in general, honestly. Things like our Mattermost chat server is located there. The drawback with AWS – and there always has to be a drawback – is that they offer Intel X86 hardware and they also offer what we call AArch64, ARM64 bit hardware. Rocky Linux 9 is interested in building also for PowerPC 64 and for S390X, which is the mainframe System Z processor. We are going to release, for all four of those architectures: Intel, ARM, PowerPC, and System Z. Unfortunately, Amazon doesn't provide mainframes to compile stuff on. So we had to turn to some other sources. IBM actually has lent us out some mainframe time, which we really appreciate. And as far as PowerPC goes, believe it or not, Oregon State University has offered us time on their PowerPC systems to build our packages. So we control the builds from our AWS environment, but the AWS environment reaches out to these other places, IBM, Oregon State. And they perform builds effectively, remotely and collect the artifacts.
Neil Hanlon:
Yeah, I think it's a very interesting change in how things are done, but also not really a change at all. The main difference is that Peridot that we're using to build is completely cloud native and built on top of Kubernetes. So instead of running Rocky 8 right now, we run kojid and it's a similar process that listens for jobs from a queue and it has them scheduled and runs the builds and saves them.
The big difference between the old build system and our new build system that we're using for 9 is that we're able to eliminate a lot of the bottlenecks for IO. Kojid relies on basically everything having access to a single NFS share and writing and reading from that same NFS share and moving it over to a Kubernetes cloud native thing, we're able to use object store to save all of those files and retrieve them in a very highly efficient, highly available manner which makes builds go a lot faster. It doesn't necessarily make builds go faster, but it makes the process of composing after builds go a lot faster. And that's also enabled us to just stick Kubernetes clusters out on these disparate locations and connect them back in through some service mesh to connect everything up and schedule these builds, literally across the entire United States.
Gregory Kurtzer:
It’s also kind of cool because, at least in theory, if we want to build 2,000 packages at once, we could actually submit that into Peridot. Peridot's going to spin up, via Kubernetes, enough pods to literally run all 2,000 builds in parallel. So because it's cloud native, it actually has all of the benefits of a cloud-based infrastructure associated with it. And it's fairly easy to deal with. Clouds offer Kubernetes services already and you can run it on-prem if you wish; it's incredibly flexible. And I'm not sure how much this has been mentioned yet, but the release of Rocky 9 is going to be more than just a release of a bunch of binaries in a repository and some ISOs. We're going to be releasing the entire build environment. So other people can come and replicate our work, which has already been done once now and possibly others. But that's the goal of what we're bringing forth with Rocky, is 100% community-focused. A lot of people have asked us, when is this going to be released?
There was just a discussion on Reddit specifically about people who are really excited about Rocky 9 being released. But we're not in a race to release. It's like racing to the starting line, right? The race starts on release and then it's, “how long can we keep going? How long can we keep it up?” And the way to do that is that giant commitment to the community. And so we are not rushing on the release of 9. And to be blunt, this is an Enterprise operating system. Very few Enterprises are gonna be like, “ah, 9.0 is out. Let's go install that on our entire infrastructure today. Yay.” Most people aren't doing that. They're going to say, well, okay, we have to now start testing; we have to start the validation process and certification processes. And it takes awhile to actually start ramping up a new release.
Now we're not sitting on our hands either; at the same time, there's a lot to get done. We don't anticipate that we're going to be very late, but just keep in mind when we release, we are going to be releasing the entire stack, the entire environment, the entire tool chain, and build environment, everything for somebody to come along and go implement their own Rocky Linux. And that is what's really driving the stability and the commitment to and from the community.
Origin of Rocky 9 [32:32]
Zane Hamilton:
That's great. Thanks, Greg. One of the other questions that I get asked a lot is: where does Rocky 9 come from? Everybody knows it is equivalent to RHEL 9, but up the chain, where does it actually come from? How does it become RHEL and Rocky?
Skip Grube:
This is something that a lot of us people who are in the know about it just assume everybody else knows it, but it needs explaining a little bit. Software comes from different projects, whether it's Firefox, Bash, or the Linux kernel itself; they all have different people working on it. And ultimately, Rocky RHEL and CentOS Stream comes from Fedora. I mentioned earlier, briefly, Fedora is what I would consider a cutting-edge distro. They release a great distro every six months if you're interested in trying it out. Every three years, Red Hat will take the latest version of Fedora and freeze it.
In our case, Red Hat 9, which Rocky 9 is based on, is based on Fedora 34, which is actually just leaving support. I think. Fedora is already on to the next thing; they're very fast. And Red Hat and other community members have taken Fedora 34, worked on it, frozen the major package versions, and they begin patching and developing from there. So, effectively, Fedora 34, they are in a way supporting it for 10 years, and so are we. The chain, upstream to downstream, goes Fedora, CentOS Stream, Red Hat Enterprise Linux, Rocky. And Rocky and Red Hat Enterprise Linux are obviously very closely linked together because our goal, unusual for a distro, is to make a completely 100% compatible distribution. So it's not what everybody likes, but it's worked out well so far, let’s put it that way.
Zane Hamilton:
Absolutely. Is it bad, Greg, that I remember running Fedora in single digit release numbers? I mean, talking about age here, I feel old seeing Fedora 34 up there.
Gregory Kurtzer:
So CHAOS Linux, the predecessor of CentOS, existed before Fedora, and the goal of CHAOS Linux – again, what created CentOS – was to create a community managed, RPM-based distribution of Linux that didn't exist before. There was Red Hat; there was Mandrake, before that, Mandriva, Corel, Yellow Dog. And YUM came out of Yellow Dog, originally. YUM even stands for the Yellowdog Updater, modified. So no community existed, but all of these distributions existed, but they were all commercially controlled. And CHAOS Linux was basically the first RPM-based focused community distribution of Linux.
And then interestingly, Fedora showed up shortly after. Fedora did not start off being an operating system. Fedora was created by a guy named Warren Togami, who worked in Hawaii at the University of Honolulu. Fedora was originally poised to be what EPEL is today. Red Hat hired Warren, brought him in, and pivoted the project to be a community RPM-based distribution of Linux,again, after CHAOS Linux was created, take that for whatever it’s worth. Because of that, there was much less of a need to have CHAOS Linux, and CentOS was taking off so much that we moved a lot of that effort from CHAOS Linux into CentOS. We did a little bit more with Chaos Linux, something called CHAOS NSA node server appliance, and really focused on a lightweight high performance distribution. But most people just wanted that binary compatibility that CentOS was providing. And so we ended up rolling that away and then obviously when Red Hat pivoted the project, it left a gaping hole in the community and that's why Rocky exists now.
Zane Hamilton:
That's awesome. Yeah. I think whenever I hear Mandrake, that’s a funny one, because I remember whenever DSL was new, so having one and a half megabit in your house was pretty awesome. I built a router out of Mandrake. That was how I was sharing it with my roommates at the time. That's how long ago that was.
Gregory Kurtzer:
You didn't have the dual stream ISDN to get you up to 1.5 because each stream of ISDN was 7.68, if I remember correctly,
Zane Hamilton:
I couldn't afford that at the time, Greg. I needed as much bandwidth for the least amount of money that I could possibly pay and dial up wouldn't cut it. That was even before we could afford a wireless router. So it was literally having a physical machine with multiple NICs in it, running Mandrake, and running cables under the carpet, through the apartment.
Gregory Kurtzer:
Wireless didn't exist, and even then it would've been really expensive.
The Rocky Linux 9 Project [38:28]
Zane Hamilton:
Can you post some links to help people who would like to contribute to the RL 9 projects? I would say yes, we absolutely can. Maybe you guys wanna talk a little bit about what that looks like and how you contribute?
Gregory Kurtzer:
So we have Mattermost. Mattermost is where we do a huge amount of the coordination of the project as well as on IRC, and the two are bridged together. So you can come into either one but that's where we do a huge amount of the coordination. People that are really interested in contributing and being part of this, this is a community; we are asking for people to come and join and help. We are always looking to expand not only our teams but, to be quite blunt, the leadership among these teams. And there are a few areas in which we actually actively need help right now. We need more help on the community side for sure. But there's other areas as well. So if people are interested, please jump into Mattermost. You can get there at chat.rockylinux.org, or jump into the IRC. I tend to tell everyone, you go to Mattermost at chat.rockylinux.org, because I spend so much more time there than at an IRC. But join in and jump into the relevant channels and start introducing yourself. We would absolutely love to have more people be associated with everything that we're doing. So, Skip, Neil, anything that you want to add from that perspective?
Skip Grube:
You want to debug packages, come see us in development, please. Have I got work for you!
Neil Hanlon:
That’s pretty much it. The big work we have right now is just some package failures. We're a good percentage of the way there on Rocky 9. Don't wanna give an ETA, but we're looking to have something available to float around internally as a release candidate with our testing teams at Rocky Linux. There is definitely help that can be done there, and Skip and I are more than happy to help give people some introductions to package debugging and building and give you a quick primer on how to RPM.
Gregory Kurtzer:
Probably a little bit of background in some C and some programming languages would help for doing some of that debugging. Even I have jumped into that mix.
Back to Rocky 9 [41:34]
Zane Hamilton:
How long will Rocky 9 be supported with updates and what is the release date? I know we've talked about what gets released with it, but I know that we actually talked about when.
Skip Grube:
Rocky 9, as stated before, Red Hat Enterprise Linux based, bite for bite, package for package, version for version, both of those distros are supported for 10 years. So that would make it 2032.
Skip Grube:
And that's what we do, we start the race. The race in the Enterprise world is a marathon. We don't think in terms of this day or that day; we've got months and years.
Neil Hanlon:
That's another good thing to point out as well; how the updates works. And it works the same way as RHEL, in terms of having a phase one and a phase two for longer term. So it's really just the first five years of the release that you get point releases. And then we will eventually get up to 8.10 and then 9.10 at some point. And those will be the releases that continue for the rest of their meaningful lives. So it's something to understand about the EL phases of support and life, so there are those two phases.
Skip Grube:
Once they reach five years, they get put into what I would call maintenance or “bug fix/security fix-only mode.” If you want an example, Red Hat, RHEL 7, and CentOS 7 are there right now, right? They're in support in 2024, I believe is the last last call for that to start looking at migrating those. I know those people out there, they have to wait until three months before end of life. And they go, “Oh my God, we didn't know about this.” Yes, you did!
Zane Hamilton:
No, Skip. There are people that will wait years, years after.
Skip Grube:
How many RHEL 4 boxes do you have left?
Gregory Kurtzer:
So am I understanding that you guys did not tell the whole community on when 9.0 is being released? What's up with that? I'll tell everybody exactly when it's going to be released.
Neil Hanlon:
Tell us!
Neil Hanlon:
When it's ready.
Gregory Kurtzer:
Before 9.1.
Zane Hamilton:
I hope. It could be an interesting conversation if it wasn't.
Skip Grube:
I personally guarantee before 9.1.
Zane Hamilton:
I don't think we have any more questions and we're kind of getting close to the end of time. I want to give you all a chance for closing thoughts. Let’s start with you, Skip.
Skip Grube:
We're working hard. 9 has three obstacles to it right now. 1) Package builds. I and others are working on that. We're about 90% to 95% complete now. 2) The compose process. This is where we produce your ISOs and your finalized repositories. 3) And the final bit is the testing. Obviously, we're not just going to throw it out there. If anyone’s interested, testing channels is where it’s at, and if you want to develop with us, development channel as well. Those are the hurdles to overcome and once they're there, we will have a distro.
Zane Hamilton:
Very nice. I see someone posted they work for a government entity and that guarantees there's CentOS 6 boxes out there still. I ran across customers within the last two weeks that still have some RHEL 5 boxes out there. So I know that stuff is around.
Gregory Kurtzer:
When I was still at the Department of Energy, we had a case where somebody came and said a piece of hardware that we forgot about that was sitting underneath somebody's desk that's been vacant in an office for many years, is now broken. And we need that system. It's a critical piece of our pipeline. And so the hardware was burnt up. If anyone was there when it died, there was probably smoke coming out of it. You know, that much dust on a motherboard does that.
But we looked at the operating system and it was running Red Hat Linux 8, not Enterprise.
And the source code for the code that was running was not available. So speaking of containers, they not only work to get you newer libraries and binaries and whatnot, it also worked to run, we actually containerized Red Hat Linux 8 and this application, and then we had it running on, at the time I think the current version was, CentOS 7.2, maybe CentOS 6, but it was around that generation. But we were able to do that and bring that system back to life as a container. Granted, don't try to do a PS or a top because it segfaults because the proc filesystem has changed so much in the last 15 years.
Zane Hamilton:
That's like ‘98, ‘99, like pre-2000s type time, wasn't it, Greg?
Gregory Kurtzer:
So if you remember what caused CentOS to exist was when Red Hat Linux was end-of-lifed. And Red Hat Linux went from 7.1, 7.2, 7.3 and then it went to 8 and then there was never an 8.1. Then it went to 9 and then all of a sudden they basically said all that stuff that we just did, let’s just get rid of it. I think they kept one or two of the versions for a little while still alive. But they had this new thing called Red Hat Enterprise Linux, which I believe started with version two. Again, most people shy away from the 1.0s. So I think they just jumped to version 2.0. Cause I don't remember there ever being a Red Hat Enterprise Linux 1.0.
Zane Hamilton:
It actually went to 2.1. I haven't looked at this in a long time, March 23rd, 2002. 20 years ago.
Gregory Kurtzer:
Wow. 20 years ago. So that was what triggered the creation of CentOS. That was in 2003-ish then, if I remember correctly, is when it was first released.
Zane Hamilton:
I think that's the end of time. Neil, Did I give you a chance to close? I'm sorry.
Neil Hanlon:
It's okay. It happens. I don't have anything.
Zane Hamilton:
Well, we really appreciate you guys joining us again today. If you are interested in what we're doing here by all means, go look at the job site.
Zane Hamilton:
Do you want to tell people that we have openings, Greg? I'll let you do that.
Gregory Kurtzer:
We are hiring quite a bit right now. So we are looking for senior engineers, senior developers, C Linux kernel, user space containers, as well as Go, backend developers, full stack developers, and front end developers. If you're interested, reach out to us. You can check out the job board on our website. Honestly, we haven't even been able to keep that up to date. That's how much we're hiring right now. So please reach out to us if you're interested. You can reach out to me personally, or any of these guys on LinkedIn, Mattermost, Slack—wherever you find us—email— we'd love to hear from you. We are looking for great people.
Zane Hamilton:
Excellent. Again, we appreciate you joining us today. We will see you again next week, and if you don't mind, like and subscribe so you can keep up with us and we can keep up with you. Appreciate it. See you later.