
Have you heard Rocky Linux 8.6 has been released?
Webinar Synopsis:
Speakers:
-
Zane Hamilton, Director Sales Engineering at CIQ
-
Gregory Kurtzer, CEO at CIQ
-
Skip Grube, Technical Solutions Architect at CIQ
-
Neil Hanlon, Solutions Architect at CIQ
Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.
Full Webinar Transcript:
Rocky 8.6 [00:00]
Zane Hamilton:
Good morning. Good afternoon. Good evening. Welcome to another CIQ webcast. We appreciate you joining us today and spending the time. Today we're going to be talking about Rocky Linux. We talked about Rocky Linux quite a bit in the past, but something new is coming out and we want to spend time on that. So today we have Skip Grube. Welcome back.
Skip Grube:
Glad to be here.
Zane Hamilton:
Thank you. So while we're waiting for Neil, Skip, you haven't been on before. Why don't you tell us about yourself and kinda what you do here at CIQ and what you do for the Rocky community?
Skip Grube:
Yeah, so I recently joined CIQ but as far as Rocky Linux goes, I kind of fell backwards into it, to be honest with you. I have been there from the start of the project. It was something I was very interested in pursuing and I met up with some of the early contributors and here I am in release engineering. So I'm a package monkey. I go through build, the bug, all kinds of packages from across the distro.
Zane Hamilton:
That's fantastic. Thank you. Neil, better late than never.
Zane Hamilton:
Welcome back, Neil. Introduce yourself again, just for those who haven't seen you in a while.
Neil Hanlon:
My name is Neil Hanlon. I'm a solutions architect here at CIQ and also in my abundant spare time, I run the infrastructure team for the Rocky Linux community.
Zane Hamilton:
Yeah. I love the spare time comments because I know you don't have any spare time. You barely get any sleep. I know Skip's in that boat, too. So one of the things that's come out that I wanted to talk to you guys about today is Rocky 8.6 coming out. Excited to hear what's new in 8.6, and what are we doing?
Neil Hanlon:
Yeah, it's exciting. And it's also, kind of not exciting, which is a theme. It's nice to have new things, but those new things don't change that much. And that's what we like about Enterprise Linux. But there are a handful of new things that have come and changed in 8.6. PHP 8.0 is now available directly from a module, as well as Perl 5.32. Previously, we'd have to go and usually get PHP 8.0 or newer versions from like Remi repo or one of these external third-party repositories. But now PHP 8.0 is right there. Also, a new version of BIND with some additional fixes and changes to make the experience on RHEL at least be able to be matched with the upstream versions of BIND, with new features, especially around DNS over HTTPs and some of those other sorts of features there.
One big change, too, is Ansible. So Ansible has recently changed over to the Ansible core module, or not module necessarily but package naming, so you may experience some upgrade troubles as you go if you already have Ansible installed. So you might need to go and swap that over to Ansible Core. There's a bunch of new modules: there’s a Log4j module that was brand new, as well as just some updates to other modules. It's exciting, and there are a few things to note from there, but in general, it's kind of a solid point release with not too many problems that folks should be comfortable and able to upgrade to.
Zane Hamilton:
And we're talking, simple as a DNF update.
Neil Hanlon:
Yeah, DNF update -y. Maybe that Ansible problem might hit some people where they might not be able to perform an upgrade because of Ansible being installed and the package name changing. But in that case, you could either just do a DNF swap for Ansible to Ansible Core, or you could remove the package, or one of those other options there. But it should be just a DNF upgrade and running on the latest 8.6.
Zane Hamilton:
Very nice. That's more of an Ansible upgrade than it is a Rocky upgrade anyway.
Neil Hanlon:
True, true.
Zane Hamilton:
Thanks, Ansible, for changing everything. Appreciate it.
Neil Hanlon:
Yeah. The only constant is change.
Rocky 9 [04:17]
Zane Hamilton:
Yeah, of course. I'm glad that there's not a lot of change with 8.6. That's great to know. I'm glad that it's boring because that's exciting. But I think one of the other things that I run across a lot, and I'm glad Skip is here to talk about this, is when we start talking about 8.6 and moving forward, a lot of people really soon start asking the question, “What about 9?” So what can you tell us about Rocky 9?
Skip Grube:
We're gonna talk a little bit about 9 here. So first of all, I want to echo what Neil said: everyone is excited about 9, I'm excited about it, but it is also boring. We are in the Enterprise Linux space and we are not out there. We are not Arch Linux. We are not Fedora. We do not move swiftly like those distros do. We don't break things occasionally like those distros do, and that's okay. If you're really out for the latest and greatest, these enterprise distros, you might wanna try something that's a little more of a preview. If you're a business or if you want something stable for 10 years: welcome, nice to have you.
We're boring and that's a good thing. That's what I want to get across. It's going to be great. Now looking forward to 9, one of the behind-the-scenes things that happens is a new build system, which is part of our giant effort towards this new build system called Peridot. I think a lot of people on the stream have probably heard of it, at least in passing. It's going really great for us. There are some growing pains because new things are difficult. And as you can imagine, building a distro is an undertaking, right? There's a lot of stuff to put together and compile.
But it has a lot of new features that we have really wanted and desired. Just some highlights, I'm just gonna give you the real high top-level view of it. Inserting specific versions of dependencies has become very easy for us, which is something that kind of plagued us in the past. a lot of these packages are very fickle in the way that they're built. They're very picky about certain minor versions of dependencies at build time. And Peridot has made that a lot easier for us, which is great. Another feature is, it's going to allow us to integrate SIGs, which we call special interest groups, much closer with the distro than we have previously been able to. We'll be able to build our special interest group packages right there with the main distro, right in conjunction with Rocky 9. It’s very exciting. Also, in the future, we plan on porting the Rocky 8 builds to it as well. And I think those SIGs can also benefit from that switch over.
Perl 5.32 [07:35]
Zane Hamilton:
That's really interesting, Skip. There's one thing I want to ask a question on. First of all, I see Greg joined us. Thank you, Greg. I'm assuming when you heard Perl 5.32, you got excited and decided to jump on.
Gregory Kurtzer:
That was exactly it. I just figured that was Perl, oh yeah, I'm there. I write really good Perl; it's like holding the shift button on my keyboard and dragging my elbow back and forth across the numbers and you can get some really great Perl code that way.
Peridot and Rocky Linux 9 [08:01]
Zane Hamilton:
You're just trying to make it where nobody else can come back behind you. Typical Perl developer. Love it. So Skip, whenever you're talking about the dependencies, it used to be hard and Peridot was gonna fix some of that. Can you dive in a little bit on that? Why was it hard or what made it hard about the dependency that this system's gonna fix a little bit?
Skip Grube:
Let's take Ruby, for example; this is something that happened two days ago. Ruby, we all know and love, right? Just like Perl, everyone loves Ruby, right? Ruby is a package in Red Hat Enterprise Linux 9 and Rocky Linux 9. We go to build it and we get errors. There's test errors. There's automatic tests that run once the package is done compiling and I examine these and I say, “Hey, these look like OpenSSL errors. What is going on?” There's something wrong with the way it's talking to OpenSSL. I go down, after much troubleshooting, much digging through some source, and it looks like the Ruby package was built for CentOS Stream and for Red Hat 9 against OpenSSL 3.00-5.
And that worked because the test passed. But now, there's been an update and OpenSSL 3.00-20 has been released as a patch, as a new package. And when you build Ruby against that OpenSSL, these tests fail. And this comes up a lot. You'd be surprised because every time that an OpenSSL update is released, Red Hat, CentOS, and other distros do not rebuild the entire system. Previously, Ruby worked and it was built and it's fine, but now it fails because another package was updated somewhere, and it’s super frustrating, but this is what we do. This is a large part of what I'm about, by the way. It sounds rough, but we get along.
So we have to either fix the Ruby test, and patch it in some way to not fail or not even run the test, which is not desirable. Or we have to build Ruby at build time with the older OpenSSL version. And previously that was dicey, that was not easy to do. We were effectively using the standard Fedora CentOS build tools, and it worked, but it was hard. And there were some tests that had to be patched out in the past. We wait until we report the issue to Red Hat or to CentOS upstream and hopefully a fix will come in the next version of Ruby and it'll build okay. Until the next time they update OpenSSL, et cetera.
So this is an example where we can say, “Okay, well, we're going to build Ruby, but we're going to inject the older OpenSSL into it.” And it will allow us to match Red Hat Enterprise Linux 9 closer, which is our whole goal, like the whole purpose of the project. This happened when 8 was coming out as well – I’m sorry for everyone who wanted, “We should put this in Rocky 8!” Our charter and our core mission remains: we will rebuild bug for bug compatible, Red Hat Enterprise Linux exactly. We’re religious about it.
Neil Hanlon:
That's a great point. I was going to bring that up as well. It's not just that these things are sometimes more difficult to build in terms of tracking the dependencies and everything else, but there is also something that's important to the project overall. We don’t just ship the version of Ruby that is in RHEL, but we also build it in the same manner. So we're building it with the same dependencies, like that OpenSSL version, even though that OpenSSL version isn't going to end up being distributed in Rocky 9. Previously, we were able to do this with Koji and the Fedora build tools by doing some side tags, importing packages, and some manual work. But the reason for the feature in Peridot is that it eliminates the human process there and makes it easier for us to be able to automate those builds in the future, by just checking what the dependencies are, how they were built, how they need to be built, and then rebuilding them.
8.6 and Ansible [13:01]
Zane Hamilton:
That's great. We have a question. So other than possible changes from Ansible, any known breakages? And this is probably for 8.6. Sorry, we're gonna go back. So Neil, are there any other things that we should pay attention to, or things that we know that get broken?
Neil Hanlon:
There are a few things that are still outstanding with the 8.6 release of RHEL. One of those things is it looks like some possible regression in conntrack or how the kernel is doing connection tracking. So if you're using Rocky Linux or Red Hat as a router, you may experience some troubles there with your traffic being blocked, if it comes in and out of two different interfaces. I've been tracking that one, trying to get some more detail on it. But in terms of breaking changes or problems there, no, there shouldn't be any other than some progressions that are being worked on.
Podman Is a Useful Tool [14:00]
Zane Hamilton:
Very nice. Thank you. Another question: despite Rocky not moving as fast with Podman, one can run the latest tech, as well, in a secure, rootless, sandbox environment. It's more of a statement than a question.
Neil Hanlon:
No, it's absolutely a great point. Podman is a super useful tool. I use it all the time. I'm using it right now to run a bunch of random stuff on my computer and laptop and everywhere else. And I think that's a great statement, as well, because it shows that you can run the latest and greatest stuff on a stable operating system that doesn't change. And you don't have to add to your list of worries about something in the host operating system being critically flawed or broken in some way. You only have to worry about that with the applications running.
Zane Hamilton:
Absolutely.
Skip Grube:
Addendum to that, as well. For the desktop-minded among you, I know we have a lot of users that use Flatpak if they want the latest desktop applications. It's a wonderful tool.
Zane Hamilton:
Absolutely. I think we've talked about that before, Greg, with other tools that we work with as well, and being able to do that with Apptainer, it's pretty cool and interesting stuff as well.
Gregory Kurtzer:
It’s the exact same use case on the HPC side is pretty much what we're talking about now. And Singularity/Apptainer has been a phenomenal tool for that in the HPC space. You can probably imagine HPC centers that have many thousands of nodes in one system are extraordinarily resistant to making any sort of updates or breaking changes to that underlying operating system. Typically, once they integrate a big system, the operating system doesn't move very much for the life of that system, because of all the different kernel modules and support and whatnot that they have to manage. It's just such a fantastic point that containers and having an ability to run these containers on top of this stable operating system is so critical.
One thing I wanted to mention earlier to a point that Skip made is that it is absolutely 100% our charter to be bit for bit, bug for bug level of compatibility with the Enterprise Linux family of distributions, obviously with Red Hat Enterprise Linux setting a lot of that standard. And now that CentOS Stream is kind of the parent of all of us, that actually puts it in a really kind of cool position, and gives us all the ability to put emphasis, code, and support back into that single community that we are all now benefiting from. But I do want to say that there's another piece of this, which is that we have had a lot of people asking for optimizations and enhancements on top of that core base foundation. Now these would always be opt-in enhancements; they would not be default. But you would be able to do things like install new packages or install custom packages based on the special interest groups that we have within Rocky. And we have a couple big dominant special interest groups that are underway right now. We've got SIG Cloud, there's SIG Kernel as well to get a more upstream mainline kernel associated with it and there's others. So if this is something that you're interested in, anybody in the community, please come and take a look and join in our Mattermost and/or IRC, and be part of these special interest groups, because it's a phenomenal way to add value over and above, just that bit for bit and bug for bug level of compatibility. So yeah, join us.
Mainline Kernel in Enterprise Linux [17:31]
Zane Hamilton:
So Greg, you mentioned the mainline kernel version, and I think it's an interesting conversation I've had with several people, and it's exciting to some, but I guess there's other people that wonder why? What benefit do they get out of doing something like that? If you had a mainline kernel in Enterprise Linux, what does that give you?
Gregory Kurtzer:
So there's a number of different aspects of this. So an obvious one is greater hardware support. If anyone's tried running an older Enterprise Linux on a newer laptop you're going to immediately have various pain points that you're going to have to work through depending on the laptop. In most cases, these have already been resolved in the upstream kernels and the one patch into our standard Rocky kernel that I've bugged Neil to see if we can get in there, is for this cool little laptop that I have right here, which is a Dell XPS 15. It works great with Rocky Linux, right out of the box with one exception: the sound driver just hits the tweeters and doesn't hit any of the other speakers. So all the sound sounds totally tinny.
And I just basically took a few lines out of the main line kernel and threw that into a patch. And I was hoping to get that in there, but at the same time, I'm kind of secretly hoping Neil actually says no to me, because to Skip's point, our goal is bit for bit and bug for bug level of compatibility. But this is exactly where a special interest group would be fantastically useful. And that code from the kernel, I didn't write it. I just looked at the current code, found that piece, and I'm like, that's what needs to go in. And luckily it integrated very easily. So that's what I'm running on this, but we actually have someone else on the team who constantly gives me crap about the fact that he's running a newer mainline kernel than me on his same XPS 15.
And he's actually running the mainline kernel and he’s also gotten performance benefits because that mainline kernel was built to support newer processors, so he is actually getting better performance out of that. There are some other aspects as well that people like about it, especially people that are looking towards real-time workloads, maybe some of the different security protocols and security methods that the upstream kernel is doing differently from the Red Hat and Enterprise Linux kernels. So there's a number of reasons here why people may really prefer that upstream kernel. And of course, there's API benefits; there's file system performance benefits. And these are things that I've just heard people bringing up. That's one of the reasons why I think having an updated kernel as an option inside of Rocky is incredibly valuable.
Before I hand over the mic, Greg Kroah-Hartman, who is the maintainer of the stable upstream kernel, is actually helping us with this. So he's actually part of this process and special interest group to ensure that what we're putting out there is going to work. And it puts him directly in the loop so that if anybody finds any bugs or issues, he can resolve those. And so that's a win-win for not only Rocky, but the community in general running Linux.
Zane Hamilton:
Absolutely. That's very exciting.
Neil Hanlon:
I think one of the biggest things that would come to mind for me is like 12 Gen Intel chips or some new CPUs that are coming out. The first support for those things goes into the mainline kernel and will eventually get backported and moved into other places. But as you said, that's a great place for SIGs to be able to work across Enterprise Linux as well, to enable those sorts of things and work to get those features backported into the CentOS Stream kernels and eventually into Rocky.
Zane Hamilton:
So Neil, how long, or how far back are we seeing from Enterprise Linux kernel to mainline? How much of a lag is there? I mean, from what I recall last time I looked, it was like 12, 18 months. Is it kinda better?
Neil Hanlon:
I think that Stream has helped to make that process more visible. And I think that part of it is just that there's some growing pains and stuff that people are learning. And also, there's a history of folks compiling their own kernels and doing things themselves just because, it can take that time and it wasn't necessarily as accessible. In fact, you know, kernels were almost entirely inaccessible to be modified except via a request from Bugzilla with Red Hat. So now I think that we have this contribution opportunity via CentOS Stream, I'm not sure if we've put any patches in to track how long they take to get through. I think that would be something that would be interesting to look into and to evaluate just how it is to contribute to the projects and work on those sorts of things. But as Greg mentioned, you know, the SIGs that we have on Rocky and especially with our new build system that's coming out will really enable us to do some of this iterative work before it gets upstream, and before we go and open those pull requests.
Zane Hamilton:
Very nice.
Gregory Kurtzer:
I'd add something to that as well, which is: one of the goals of Enterprise Linux is to have a completely stable API ABI as the default scenario. So Zane, one way somebody might read your question is, how far behind is it from upstream? Because there are no breaking changes regarding API ABI, I'm actually planning on having a follow up with Skip on how OpenSSL and update an OpenSSL broke a build if it's supposed to be API ABI compatible. And is that a bug that we need to actually be thinking about?
Neil Hanlon:
Yes.
Gregory Kurtzer:
The other aspect of that is once any version of Enterprise Linux is released, think of it as, that is now a fixed set of versions in time. All of the packages, this is not strictly adhered to, but it's pretty close. All of the versions are pretty much stagnant at that point. As I said, there's a few edge cases in which they do update, but generally speaking, all that core is going to remain exactly the same. So the kernel that Enterprise Linux is released with, the major minor patch level version, is not going to change for the entire life of the operating system, which is for 10 years. So you have quite a while there. There are going to be backported fixes, security issues, and whatnot, and the relevance of those fixes is going to change over time.
So for example at some point it will be everything that's broken needs to get fixed. And at some point it's going to be, you know what, we're just doing security updates; we're just doing critical updates. And so that kind of changes over time. But this is why a new release is such a big deal in the Enterprise Linux community. That’s why we’re all super excited about grabbing that, and as soon as it goes beta and then stable, throwing it on everything that we have because we know we have that absolute stability.
Zane Hamilton:
That's very cool.
Skip Grube:
I just want to note that Greg's absolutely right. We will be building kernel 4.18 for Rocky 8 in the year 2029.
Where Is Rocky Built? [25:50]
Zane Hamilton:
So one of the other questions that comes up quite a bit is: where does Rocky actually get built? From a community standpoint, that's a really interesting question.
Skip Grube:
Yeah. So physically, rubber has to hit the road somewhere. We can abstract things away, but we must run instructions, and the primary place is in Amazon Web Services. AWS, perhaps you've heard of it. It’s a large cloud provider. They’re known among some. The Rocky Enterprise Software Foundation enjoys a donation from them of an AWS account in which we conduct a lot of our builds, including a lot of our infrastructure in general, honestly. Things like our Mattermost chat server is located there. The drawback with AWS – and there always has to be a drawback – is that they offer Intel X86 hardware and they also offer what we call aarch64, ARM64 bit hardware. Rocky Linux 9 is interested in building also for PowerPC 64 and for S390X, which is the mainframe System Z processor. We are going to release, for all four of those architectures: Intel, ARM, PowerPC, and System Z. Unfortunately, Amazon doesn't provide mainframes to compile stuff on. So we had to turn to some other sources. IBM actually has lent us out some mainframe time, which we really appreciate. And as far as PowerPC goes, believe it or not, Oregon State University has offered us time on their PowerPC systems to build our packages. So we control the builds from our AWS environment, but the AWS environment reaches out to these other places, IBM, Oregon State. And they perform builds effectively, remotely and collect the artifacts.
Neil Hanlon:
Yeah, I think it's a very interesting change in how things are done, but also not really a change at all. The main difference is that Peridot that we're using to build is completely cloud native and built on top of Kubernetes. So instead of running Rocky 8 right now, we run kojid and it's a similar process that listens for jobs from a queue and it has them scheduled and runs the builds and saves them.
The big difference between the old build system and our new build system that we're using for 9 is that we're able to eliminate a lot of the bottlenecks for IO. Kojid relies on basically everything having access to a single NFS share and writing and reading from that same NFS share and moving it over to a Kubernetes cloud native thing, we're able to use object store to save all of those files and retrieve them in a very highly efficient, highly available manner which makes builds go a lot faster. It doesn't necessarily make builds go faster, but it makes the process of composing after builds go a lot faster. And that's also enabled us to just stick Kubernetes clusters out on these disparate locations and connect them back in through some service mesh to connect everything up and schedule these builds, literally across the entire United States.
Gregory Kurtzer:
It’s also kind of cool because, at least in theory, if we want to build 2,000 packages at once, we could actually submit that into Peridot. Peridot's going to spin up, via Kubernetes, enough pods to literally run all 2,000 builds in parallel. So because it's cloud native, it actually has all of the benefits of a cloud-based infrastructure associated with it. And it's fairly easy to deal with. Clouds offer Kubernetes services already and you can run it on-prem if you wish; it's incredibly flexible. And I'm not sure how much this has been mentioned yet, but the release of Rocky 9 is going to be more than just a release of a bunch of binaries in a repository and some ISOs. We're going to be releasing the entire build environment. So other people can come and replicate our work, which has already been done once now and possibly others. But that's the goal of what we're bringing forth with Rocky, is 100% community-focused. A lot of people have asked us, when is this going to be released?
There was just a discussion on Reddit specifically about people who are really excited about Rocky 9 being released. But we're not in a race to release. It's like racing to the starting line, right? The race starts on release and then it's, “how long can we keep going? How long can we keep it up?” And the way to do that is that giant commitment to the community. And so we are not rushing on the release of 9. And to be blunt, this is an Enterprise operating system. Very few Enterprises are gonna be like, “ah, 9.0 is out. Let's go install that on our entire infrastructure today. Yay.” Most people aren't doing that. They're going to say, well, okay, we have to now start testing; we have to start the validation process and certification processes. And it takes awhile to actually start ramping up a new release.
Now we're not sitting on our hands either; at the same time, there's a lot to get done. We don't anticipate that we're going to be very late, but just keep in mind when we release, we are going to be releasing the entire stack, the entire environment, the entire tool chain, and build environment, everything for somebody to come along and go implement their own Rocky Linux. And that is what's really driving the stability and the commitment to and from the community.
Origin of Rocky 9 [32:32]
Zane Hamilton:
That's great. Thanks, Greg. One of the other questions that I get asked a lot is: where does Rocky 9 come from? Everybody knows it is equivalent to RHEL 9, but up the chain, where does it actually come from? How does it become RHEL and Rocky?
Skip Grube:
This is something that a lot of us people who are in the know about it just assume everybody else knows it, but it needs explaining a little bit. Software comes from different projects, whether it's Firefox, Bash, or the Linux kernel itself; they all have different people working on it. And ultimately, Rocky RHEL and CentOS Stream comes from Fedora. I mentioned earlier, briefly, Fedora is what I would consider a cutting-edge distro. They release a great distro every six months if you're interested in trying it out. Every three years, Red Hat will take the latest version of Fedora and freeze it.
In our case, Red Hat 9, which Rocky 9 is based on, is based on Fedora 34, which is actually just leaving support. I think. Fedora is already on to the next thing; they're very fast. And Red Hat and other community members have taken Fedora 34, worked on it, frozen the major package versions, and they begin patching and developing from there. So, effectively, Fedora 34, they are in a way supporting it for 10 years, and so are we. The chain, upstream to downstream, goes Fedora, CentOS Stream, Red Hat Enterprise Linux, Rocky. And Rocky and Red Hat Enterprise Linux are obviously very closely linked together because our goal, unusual for a distro, is to make a completely 100% compatible distribution. So it's not what everybody likes, but it's worked out well so far, let’s put it that way.
Zane Hamilton:
Absolutely. Is it bad, Greg, that I remember running Fedora in single digit release numbers? I mean, talking about age here, I feel old seeing Fedora 34 up there.
Gregory Kurtzer:
So CHAOS Linux, the predecessor of CentOS, existed before Fedora, and the goal of CHAOS Linux – again, what created CentOS – was to create a community managed, RPM-based distribution of Linux that didn't exist before. There was Red Hat; there was Mandrake, before that, Mandriva, Corel, Yellow Dog. And YUM came out of Yellow Dog, originally. YUM even stands for the Yellowdog Updater, modified. So no community existed, but all of these distributions existed, but they were all commercially controlled. And CHAOS Linux was basically the first RPM-based focused community distribution of Linux.
And then interestingly, Fedora showed up shortly after. Fedora did not start off being an operating system. Fedora was created by a guy named Warren Togami, who worked in Hawaii at the University of Honolulu. Fedora was originally poised to be what EPEL is today. Red Hat hired Warren, brought him in, and pivoted the project to be a community RPM-based distribution of Linux,again, after CHAOS Linux was created, take that for whatever it’s worth. Because of that, there was much less of a need to have CHAOS Linux, and CentOS was taking off so much that we moved a lot of that effort from CHAOS Linux into CentOS. We did a little bit more with Chaos Linux, something called CHAOS NSA node server appliance, and really focused on a lightweight high performance distribution. But most people just wanted that binary compatibility that CentOS was providing. And so we ended up rolling that away and then obviously when Red Hat pivoted the project, it left a gaping hole in the community and that's why Rocky exists now.
Zane Hamilton:
That's awesome. Yeah. I think whenever I hear Mandrake, that’s a funny one, because I remember whenever DSL was new, so having one and a half megabit in your house was pretty awesome. I built a router out of Mandrake. That was how I was sharing it with my roommates at the time. That's how long ago that was.
Gregory Kurtzer:
You didn't have the dual stream ISDN to get you up to 1.5 because each stream of ISDN was 7.68, if I remember correctly,
Zane Hamilton:
I couldn't afford that at the time, Greg. I needed as much bandwidth for the least amount of money that I could possibly pay and dial up wouldn't cut it. That was even before we could afford a wireless router. So it was literally having a physical machine with multiple NICs in it, running Mandrake, and running cables under the carpet, through the apartment.
Gregory Kurtzer:
Wireless didn't exist, and even then it would've been really expensive.
The Rocky Linux 9 Project [38:28]
Zane Hamilton:
Can you post some links to help people who would like to contribute to the RL 9 projects? I would say yes, we absolutely can. Maybe you guys wanna talk a little bit about what that looks like and how you contribute?
Gregory Kurtzer:
So we have Mattermost. Mattermost is where we do a huge amount of the coordination of the project as well as on IRC, and the two are bridged together. So you can come into either one but that's where we do a huge amount of the coordination. People that are really interested in contributing and being part of this, this is a community; we are asking for people to come and join and help. We are always looking to expand not only our teams but, to be quite blunt, the leadership among these teams. And there are a few areas in which we actually actively need help right now. We need more help on the community side for sure. But there's other areas as well. So if people are interested, please jump into Mattermost. You can get there at chat.rockylinux.org, or jump into the IRC. I tend to tell everyone, you go to Mattermost at chat.rockylinux.org, because I spend so much more time there than at an IRC. But join in and jump into the relevant channels and start introducing yourself. We would absolutely love to have more people be associated with everything that we're doing. So, Skip, Neil, anything that you want to add from that perspective?
Skip Grube:
You want to debug packages, come see us in development, please. Have I got work for you!
Neil Hanlon:
That’s pretty much it. The big work we have right now is just some package failures. We're a good percentage of the way there on Rocky 9. Don't wanna give an ETA, but we're looking to have something available to float around internally as a release candidate with our testing teams at Rocky Linux. There is definitely help that can be done there, and Skip and I are more than happy to help give people some introductions to package debugging and building and give you a quick primer on how to RPM.
Gregory Kurtzer:
Probably a little bit of background in some C and some programming languages would help for doing some of that debugging. Even I have jumped into that mix.
Back to Rocky 9 [41:34]
Zane Hamilton:
How long will Rocky 9 be supported with updates and what is the release date? I know we've talked about what gets released with it, but I know that we actually talked about when.
Skip Grube:
Rocky 9, as stated before, Red Hat Enterprise Linux based, bite for bite, package for package, version for version, both of those distros are supported for 10 years. So that would make it 2032.
Skip Grube:
And that's what we do, we start the race. The race in the Enterprise world is a marathon. We don't think in terms of this day or that day; we've got months and years.
Neil Hanlon:
That's another good thing to point out as well; how the updates works. And it works the same way as RHEL, in terms of having a phase one and a phase two for longer term. So it's really just the first five years of the release that you get point releases. And then we will eventually get up to 8.10 and then 9.10 at some point. And those will be the releases that continue for the rest of their meaningful lives. So it's something to understand about the EL phases of support and life, so there are those two phases.
Skip Grube:
Once they reach five years, they get put into what I would call maintenance or “bug fix/security fix-only mode.” If you want an example, Red Hat, RHEL 7, and CentOS 7 are there right now, right? They're in support in 2024, I believe is the last last call for that to start looking at migrating those. I know those people out there, they have to wait until three months before end of life. And they go, “Oh my God, we didn't know about this.” Yes, you did!
Zane Hamilton:
No, Skip. There are people that will wait years, years after.
Skip Grube:
How many RHEL 4 boxes do you have left?
Gregory Kurtzer:
So am I understanding that you guys did not tell the whole community on when 9.0 is being released? What's up with that? I'll tell everybody exactly when it's going to be released.
Neil Hanlon:
Tell us!
Neil Hanlon:
When it's ready.
Gregory Kurtzer:
Before 9.1.
Zane Hamilton:
I hope. It could be an interesting conversation if it wasn't.
Skip Grube:
I personally guarantee before 9.1.
Zane Hamilton:
I don't think we have any more questions and we're kind of getting close to the end of time. I want to give you all a chance for closing thoughts. Let’s start with you, Skip.
Skip Grube:
We're working hard. 9 has three obstacles to it right now. 1) Package builds. I and others are working on that. We're about 90% to 95% complete now. 2) The compose process. This is where we produce your ISOs and your finalized repositories. 3) And the final bit is the testing. Obviously, we're not just going to throw it out there. If anyone’s interested, testing channels is where it’s at, and if you want to develop with us, development channel as well. Those are the hurdles to overcome and once they're there, we will have a distro.
Zane Hamilton:
Very nice. I see someone posted they work for a government entity and that guarantees there's CentOS 6 boxes out there still. I ran across customers within the last two weeks that still have some RHEL 5 boxes out there. So I know that stuff is around.
Gregory Kurtzer:
When I was still at the Department of Energy, we had a case where somebody came and said a piece of hardware that we forgot about that was sitting underneath somebody's desk that's been vacant in an office for many years, is now broken. And we need that system. It's a critical piece of our pipeline. And so the hardware was burnt up. If anyone was there when it died, there was probably smoke coming out of it. You know, that much dust on a motherboard does that.
But we looked at the operating system and it was running Red Hat Linux 8, not Enterprise.
And the source code for the code that was running was not available. So speaking of containers, they not only work to get you newer libraries and binaries and whatnot, it also worked to run, we actually containerized Red Hat Linux 8 and this application, and then we had it running on, at the time I think the current version was, CentOS 7.2, maybe CentOS 6, but it was around that generation. But we were able to do that and bring that system back to life as a container. Granted, don't try to do a PS or a top because it segfaults because the proc filesystem has changed so much in the last 15 years.
Zane Hamilton:
That's like ‘98, ‘99, like pre-2000s type time, wasn't it, Greg?
Gregory Kurtzer:
So if you remember what caused CentOS to exist was when Red Hat Linux was end-of-lifed. And Red Hat Linux went from 7.1, 7.2, 7.3 and then it went to 8 and then there was never an 8.1. Then it went to 9 and then all of a sudden they basically said all that stuff that we just did, let’s just get rid of it. I think they kept one or two of the versions for a little while still alive. But they had this new thing called Red Hat Enterprise Linux, which I believe started with version two. Again, most people shy away from the 1.0s. So I think they just jumped to version 2.0. Cause I don't remember there ever being a Red Hat Enterprise Linux 1.0.
Zane Hamilton:
It actually went to 2.1. I haven't looked at this in a long time, March 23rd, 2002. 20 years ago.
Gregory Kurtzer:
Wow. 20 years ago. So that was what triggered the creation of CentOS. That was in 2003-ish then, if I remember correctly, is when it was first released.
Zane Hamilton:
I think that's the end of time. Neil, Did I give you a chance to close? I'm sorry.
Neil Hanlon:
It's okay. It happens. I don't have anything.
Zane Hamilton:
Well, we really appreciate you guys joining us again today. If you are interested in what we're doing here by all means, go look at the job site.
Zane Hamilton:
Do you want to tell people that we have openings, Greg? I'll let you do that.
Gregory Kurtzer:
We are hiring quite a bit right now. So we are looking for senior engineers, senior developers, C Linux kernel, user space containers, as well as Go, backend developers, full stack developers, and front end developers. If you're interested, reach out to us. You can check out the job board on our website. Honestly, we haven't even been able to keep that up to date. That's how much we're hiring right now. So please reach out to us if you're interested. You can reach out to me personally, or any of these guys on LinkedIn, Mattermost, Slack—wherever you find us—email— we'd love to hear from you. We are looking for great people.
Zane Hamilton:
Excellent. Again, we appreciate you joining us today. We will see you again next week, and if you don't mind, like and subscribe so you can keep up with us and we can keep up with you. Appreciate it. See you later.
Transcript
good morning good afternoon good evening welcome to another ciq webcast we appreciate you joining us today and spending the time today we're going to be talking about rocky linux uh we talked about rocky limits quite a bit in the past but something new is coming out and we want to spend some time on that so today we have skip group open back glad to be here thank you while we're waiting for neil skip you know before why don't you tell us about yourself and kind of what you do here at ciq and what you do for the rocky community yeah so i recently joined ciq
um oh good neil's coming in looks like um i but as far as rocky linux goes i was there kind of fell backwards into it to be honest with you very they're almost from the start of the project um something i was very interested in pursuing and um i kind of met up with some of the early contributors and here i am in release engineering so i'm a package monkey i i go through build the bug uh all kinds of packages for across the distro it's fantastic thank you neil better late than never as usual you know welcome welcome back neil uh you wanna introduce
yourself again just for those who haven't seen in a while yeah for several weeks uh name's neil hanlon i'm a solutions architect here at ciq and i uh also in my my abundant spare time from the infrastructure team for the rocky uh linux community yeah i love the spare time comments because i know you don't have any spare time barely get any sleep i know skips in that boat too so one of the things that's come out and it's exciting that i wanted to talk to you guys about today is rocky 86 coming out so i know that's been something we've talked about for a
while um excited to hear what's new in 86 and kind of what are we doing yeah um you know it's it's exciting and it's it's also you know kind of not exciting which is which is a theme with which is exciting thanks right like we it's nice to have new things but um you know those new things don't change that much and that's what we what we like about enterprise linux but there are a handful of new things that have come um and changed in 8.6 uh php 8 is now available directly from a module as well as pearl 5.32 so you know previously we'd
View full transcriptHide full transcript
have to go and usually get php eight or newer versions from uh like remy repo or one of these external um third-party repositories but uh now php 8 is right there um also a new version of bind with some some additional fixes and changes to make the experience on rel at least be able to be matched um with the upstream versions of bind with new features especially around i believe like dns over https and some of those other sorts of features there um one big change to uh is ansible so ansible has recently changed over to the ansible core module or not module necessarily but
package naming so you may experience some some upgrade troubles as you go if you if you already have ansible installed so you might need to go and swap that over to ansible core um now there's there's a bunch of new modules there's a log4j module that that was um brand new um as well as just some updates to other modules um again you know it's exciting but and there's there's a few things to note um from there but you know in general it's kind of a solid point release with not too many um problems that folks should be comfortable able to upgrade to and we're
talking simple as dnf update yeah yep dnf update dash y maybe that ansible um problem might might hit some people um where they might not be able to perform an upgrade because of ansible being installed and the package name changing but in that case you could either just do a dnf swap for ansible dancing core or you could remove the package or one of those other options there but yeah you know it should be just a dnf upgrade and running on the latest deep.6 very nice that's more of an ansible upgrade than there's a rocky upgrade anyway true true thanks ansible for changing everything appreciate
it yeah only constant is change yeah of course so i'm glad that there's not a lot of change with eight six that's great to know uh i'm glad that it's boring because that's exciting but i think one of the other things that i run across a lot and i'm glad skip you're here to talk about this is when we start talking about 86 and moving forward a lot of people really soon start asking the question what about nine so what can you tell us about about rocky 9 what about nine uh that's classified excuse me okay all right thanks for joining us we're gonna talk
a little yeah everybody say now we're gonna talk a little bit about nine here so first of all i want to echo kind of what neil said and um nine everyone's excited about it i'm excited about it but it is also boring i mean and that's what i mean we are in the enterprise linux space and we are we are not out there we are not arch linux right we are not uh we're not fedora right we do not uh we do not move swiftly like those distress do we don't break things occasionally like those distros do and um and that's okay like if you
you know if you're really out for the latest and greatest um you know the these uh these enterprise distros you know you might want to try something that's a little more of a preview if you're a business or if you want something stable for 10 years that's you know welcome nice to have you um yeah so that's uh um we're boring and that's a good thing that's uh that's what i want to get across um yeah and it's uh um it's gonna be great now looking uh looking forward to nine here one of the kind of behind the scenes things that happens is a new
build system which is part of our giant effort towards this new build system called peridot i think a lot of people on the stream have probably heard of it at least in passing it is uh it's going really great for us there are some growing pains because new things they are they are difficult um and you know as you can imagine building a distro is it's an undertaking right there's a lot of stuff to put together a lot of stuff to compile um but uh it has a lot of new features that we have really wanted and desired um just some highlights i'm just going
to give you the you know the real hot top level view of it inserting specific versions of dependencies has become very easy for us which is something that kind of plagued us in the past these a lot of these packages are very fickle in the way that they're they're built they're very picky about um you know certain versions of you know certain minor versions of dependencies at build time and that's um peridot has made that a lot easier for us which is great and um another feature is it's going to allow us to integrate sigs which we call special interest groups much more closely with the distro than we have previously been able to we'll be able to to build our special interest group packages right there with the main distro right in conjunction with rocky 9.
so very exciting and we also in the future we plan on porting uh the rocky 8 builds to it as well and i think those sigs can also benefit from that that switch over so that's really interesting skip and there's one thing i want to ask a question on first of all i see greg joined us thank you greg i'm assuming when you heard pearl532 you you got excited and decided to jump on that was it that was exactly it i was just like that was good oh yeah i'm there yes i write really good pearl it's like holding the shift button on my keyboard
and dragging my elbow back and forth across the numbers and you can get some really great pearl coats that way yeah you're just trying to make it where nobody else can come back behind you typical pearl developer love it no so skip whenever you're talking about the dependencies and it was it used to be hard and peridot's going to fix some of that can you give me dive in a little bit on that why was it hard or what made it hard about the dependencies this system is going to fix a little bit so for example um you may not know this but build time
this is i can illustrate this with an example here so uh let's take ruby for example this is something that that happened two days ago even um ruby we all know and love right just like pearl everyone loves ruby right ruby is a package in red hat enterprise linux nine and rocky linux nine um we go to build it and we get errors uh there's there's test errors you know there's automatic tests that run once the package is done compiling and i lo i i examine these and i say hey these look like open ssl errors what is going on there's something wrong with you
know the way open way it's talking to open ssl and um i go down uh after much troubleshooting much digging through some source it looks like um the the ruby package will it was built for centos stream and for red hat nine against open ssl 3.00-5 right and that worked because the test passed um but now we're on now there's been an update and openssl 3.0020 has been released right as a as a patch right it's a new package and when you build ruby against that openness of cell these tests fail right and that's and this comes up a lot you'd be surprised because every
time that an open ssl update is released red hat and and centos and other distros do not rebuild the entire system right they you know so previous ruby it worked and it was built and it's fine but now it fails because because another package was updated somewhere and super frustrating but this is what we do you know um this is this is a large part of of what i'm about by the way and it's really sounds rough i know but it's we we get along but um yeah so we have some options here right we can simply you know um we have to either fix
the ruby test you know patch it in some way to to either you know not fail or to or patch it out you know don't even run the test which is not desirable or we have to build ruby at build time with the older open ssl version right and previously that was dicey that was not easy to do um in the uh we're using effectively the standard you know fedora centos build tools and it worked like you could get it done but it it was hard and there were some tests that had to be patched out in the past and you know we wait until
until you know we report the issue to red hat or to centos stream upstream and uh you know hopefully a fix will come and the next version of ruby it'll build okay you know until the until the next time they update open ssl you know etc so um so this is an example where we can say okay well we're going to build ruby but we're going to inject the older open ssl into it and it just we can we can match will allow us to to match red hat enterprise linux 9 closer which is our whole goal like the whole purpose of the project remains
again i'm sorry this happened when eight was coming out as well i'm sorry for everyone who wanted you know like we should put this in in rocky eight we should put this as like now the the our our charter and our core mission remains we will rebuild and you know bug for bug compatible red hat enterprise linux exactly or religious about it you wouldn't wouldn't believe yeah no that's that's a great point i was going to bring that up as well it's not just that these things are sometimes more difficult to build just in terms of tracking down the dependencies and everything else but that's
also something that's important to the project overall is that we we not just ship the version of uh ruby that that is in rel but we also build it in the same manner so we're building it with the same um dependencies like that open the ssl version and even though that open ssl version isn't going to end up being distributed in rocky 9. so previously we were able to do this with koji and the fedora build tools by doing some side tags importing packages and some manual work but the the reason for the feature in peridot is that it it eliminates the human process there
and makes it easier for us to be able to automate those builds in the future by just checking what they've been checking what the dependencies are how they are built how they need to be built and then rebuilding them that's great and i think i think you oh we have a question all right so other than possible changes from ansible uh any known breakages and this is probably for eight six sorry we're gonna go we're gonna go back so neil are there any other things that we should pay attention to or things that we know that they get broken there are a few things um
that are still outstanding with the 8.6 release of rel one of those things is it looks like some possible regression in contract or how the kernel is doing connection tracking so if you're using rocky linux or red hat as a router you may experience some troubles there with your traffic being blocked if it comes out in and out of two different interfaces um so i believe i've been tracking that one trying to get some more detail on it but in terms of breaking changes or problems there no there there shouldn't be any other than some some regressions that are being worked on very nice thank
you another question despite rocky not moving as fast with pod man one can run the latest tech as well in a secure rootless sandbox environment it's more of a statement than a question but no it's it's absolutely a great point podman is um a superhero tool i use it all the time um i'm using it right now to run a bunch of random stuff on my computer and a laptop and everywhere else and i think that's a great statement as well because it shows that you can run the latest and greatest stuff on a stable operating system that doesn't change you don't have to worry
about um you don't have to add to your list of worries something in their host operating system being critically flawed or broken in some way you only have to worry about that with the applications running absolutely addendum to that as well the um for the desktop-minded among you i know we have a lot of users that use flat pack for if want the latest desktop applications it's a wonderful tool absolutely i think we've talked about that before greg with other tools that we we work with as well and being able to do that with apptaner it's pretty pretty cool and interesting stuff as well what's
an exact same use case on the hpc side uh is pretty much what we're what we're talking about now and singularity optaner uh has been a phenomenal tool for that in the hpc space you can probably imagine hpc centers that have many thousands of nodes in one system are extraordinarily resistant to making any sort of updates or breaking changes to that underlying operating system typically once they integrate a big system the operating system doesn't move very much for the life of that system because of all the different kernel modules and support and whatnot that they have to manage and that containers again is is just
that's such a it's just such a fantastic point uh that containers and having an ability to run these containers on top of this stable operating system is so critical uh one thing i wanted to mention earlier to a point that skip made is uh it is absolutely 100 our charter to be bit for bit bug for bug level of compatibility with the enterprise linux family of distributions obviously with red hat enterprise linux kind of uh setting a lot of that standard and now that's now that centos stream is kind of the parent of all of us it actually puts it in a really kind of
cool uh position and gives us all the ability to kind of put put emphasis and code and and support back into that single community that we are all now benefiting from um but i do want to say that there's another piece of this which is we have had a lot of people asking for optimizations and enhancements on top of that core base foundation now these would always be opt-in enhancements they would not be default but you can you would be able to do things like install newer packages or install custom packages based on the special interest groups uh that we have within rocky we have
a couple kind of big dominant uh special interest groups that are underway right now we've got sig cloud there's sig colonel as well to get a more upstream mainline kernel associated with it uh and and there's others so if this is something that you're interested in anybody in the community uh please come and take a look you know join our joining our mattermost and um and or irc and be part of these special interest groups because it's a phenomenal way to add value uh over and above just that bit for bit and bug for bug level of compatibility uh so yeah join us so greg
you mentioned the mainline kernel version and i think it's it's an interesting conversation they've had with several people and it's exciting to some but i guess there's other people that wonder why what benefit do they get out of doing something like that if you had a mainline kernel in enterprise linux what does that give you so there's there's a number of different aspects of this so uh an obvious one is is greater hardware support uh if anyone's tried running uh you know an older enterprise linux on a newer laptop uh you're going to immediately have various pain points that you're going to have to work
through depending on the laptop in many cases i even say in most cases these have already been resolved in the upstream kernels and um the the one patch into our uh our standard rocky kernel that i've bugged neil to see if we can get in there is for this this cool little laptop that i have right here which is a dell xps 15. works great with rocky linux right out of the box with one exception the sound driver just hits the tweeters and doesn't hit any of the other speakers so every all the sound sounds totally tinny and uh so um i just basically took
a patch out of the or took a few lines out of the mainline kernel and threw that into a patch and i was hoping to get that in there but at the same time i'm kind of secretly hoping neil actually says no to me because uh to skip's point our goal is bid for bit and bug for bug level of compatibility but this is exactly where a special interest group would be fantastically useful and that that code from the kernel i didn't write it i just looked at the current code found that piece and i'm like that's what needs to go in and luckily it
integrated very easily but so that's what i'm running on on this but we actually have someone else on the team uh who constantly gives me crap about the fact that he's running a newer kernel than me on his same xps 15 and he's actually running the mainline kernel and he's also gotten performance benefits because that mainline kernel was built for built for to support newer processors and so he's actually getting better performance out of that and there's some other aspects as well that people like about it especially people that are looking towards real-time workloads maybe you know some of the different security protocols and security
methods that the upstream kernel is doing different than the red hat and enterprise linux kernels so there's there's a number of reasons here why people may really prefer that upstream uh that upstream kernel again of course there's api you know benefits there's file system performance benefits and these are things that i've just heard people um bringing up so yeah i don't know if skip and neil have any other comments on that but that's why one of the reasons why i think having an updated kernel as an option inside of rocky is incredibly valuable oh before i hand over the mic and greg crowe hartman uh
who is the uh the maintainer of the stable upstream colonel is actually helping us with this so uh he's actually part of this process part of the special interest group to ensure that what we're putting out there is going to work and it puts him in the loop that if anybody finds any bugs or issues puts him directly in that loop in terms of being able to resolve those and so that's a win-win for for not only rocky but the community in general running linux absolutely that's very exciting i think one of the biggest things that would come to mind for me is like 12th
gen intel chips or some new cpus that are coming out um the first support for those things goes into the mainline kernel and will eventually get back ported and um and moved into other places but as you said that's a that's a great place for sigs to be able to work across enterprise linux as well to um to enable those sorts of things and work to get those features back ported into the centos stream kernels and eventually into rocky of course so you know how long or how far back sorry skipping me to cut you off but how far back are we seeing from enterprise
linux kernel to mainline how how much of a lag is there i mean from what i recall last time i looked it was like 12 to 18 months is it gotten better i think that stream has um helped to make that process more visible and i think that part of it is just that there's some growing pains and stuff that people are learning um and also you know i think there's a history of folks compiling their own kernels and doing them though they're doing things themselves just because of as you said it can't take that time and it wasn't necessarily as accessible in fact you
know colonels were almost entirely unaccessible to be modified except via a request from bugzilla with red hat so you know now i think that we have this um contribution opportunity via sent to us stream um i'm not sure if we've uh you know put any patches in to to track how long they they take to get through um i think that would be something that would be interesting to to look into um and and evaluate just you know how how is it to contribute to the projects and and work on those sorts of things but as greg mentioned you know the the sigs that we
have um on rocky and especially with our new build system that's coming out will really enable us to um do some of this interactive work before it gets upstream before we go and open those pull requests very nice i'd add something to that as well which is one of the goals of enterprise linux is to have a completely stable api abi as the default kind of scenario so zayn one one way somebody might read your question is you know how far behind is it how far behind is it from upstream and because we are there there are no breaking changes regarding api abi um and
i'm actually planning on having a follow up with skip on how openssl um and update an openssl broke a build if it's supposed to be api api compatible and is that a bug that we need to actually be thinking about um but yes so but the aspect of that is or the other aspect of that is once enterprise linux any version is released think of it as that is now a fix fixed um a set of versions in time all of the packages this is not strictly adhered to but it's pretty dang close all of the versions are are pretty much stagnant at that point
as i said there's a few edge cases in which they do update but generally speaking all that core is going to remain exactly the same so the kernel that enterprise linux is released with the the major minor patch level version is not going to change for the entire life of the operating system which is for 10 years so you have quite a while there so they are going to there are going to be backported fixes security issues and whatnot and the the relevance of those fixes is going to change over time so for example uh at some point it'll be every you know everything that's
broken needs to get fixed and at some point it's going to be you know what we're just doing security updates we're just doing critical updates and so that kind of changes over time uh but this is why this is why kind of a new release is such a big deal in the enterprise linux community that's why we're all like just excited super excited about grabbing that and you know as soon as it goes beta and then stable you know throwing it on everything that we have is because we know we have that absolute stability that's very cool skip i thought you you had on me
like you had something to add to that i just want to note that greg is absolutely right we we will be building colonel 4.18 for rocky 8 in 2029 the year 2029 and i will be so old i can't i don't i can't even see you might be you might be almost my age then scary no i think i think it moves faster for me this is what the dream is this is why zoom and and doing you know talks over these sorts of mediums is so much better in real life because i would totally kick skip in the shin for that one exactly my
shins are safe so one of the other questions that comes up quite a bit and i find it interesting too and i know skippy and i were talking about this earlier is where does rocky actually get built i mean from a community standpoint that's a really interesting question yeah so physically rubber has to hit the road somewhere we can abstract things away but uh yeah we must run instructions and the primary place and neil can also speak to this because he knows even more about it than i do but um the primary place is in amazon web services aws perhaps you've heard of it large
cloud provider they're known among some we the rocky enterprise software foundation enjoys buying my books from them gotcha yeah they're just some book seller they might get into computing who knows but um yeah we we enjoy a uh donation from them of uh basically an aws account um in which we uh we conduct a lot of our builds including um a lot of our infrastructure in general honestly um you know things like our our matter most chat server is located there um uh the drawback with aws and there always has to be a drawback is that um they they offer uh intel x86 hardware and
they also offer what we call arch 64 arm 64-bit hardware but rocky linux 9 is interested in building also for powerpc64 and for s390x which is the mainframe system z processor um we're going to release for all five of those or all four of those architectures intel arm powerpc z and uh unfortunately you know in or amazon doesn't provide mainframes to compile stuff on so we had to turn to some other sources um we had we enjoy ibm themselves actually has lent us out uh some mainframe time which we really appreciate and um as far as power pc goes uh believe it or not oregon
state university shout out to them um has has offered us time on their on their power pc systems to to build our packages and so we control the builds from our aws environment but the aws environment reaches out to these other places ibm oregon state and uh yeah they perform builds uh effectively remotely and collect the artifacts yeah i think it's uh it's it's very interesting kind of change in how things are done but also not really a change at all the the main difference really is that uh peridot that we're using to build is completely cloud native and built on top of kubernetes so
instead of running um you know in our for rocky 8 right now we run code gt and it's this similar process that listens for jobs from a queue and it has them scheduled and runs the builds and saves them the the big difference between the old build system and our new build system that we're using for nine is that we're able to eliminate a lot of the bottlenecks for i o that we had koji relies on basically everything having access to a single nfs share and writing and reading from that same nfs share and moving it over to a kubernetes cloud native thing we're able
to use object store to save all of those files and retrieve them in a very highly efficient highly available manner which makes builds go a lot faster it or it doesn't necessarily make builds go faster but it makes the the process of composing after builds um go a lot faster and that's also enabled us to just stick kubernetes clusters out on these disparate locations and connect them back in through some service mesh to to connect everything up in and schedule these builds and you know but look literally across the entire united states so it's also kind of cool too because at least in theory if
we want to build 2000 packages at once i mean we could actually submit that into the into peridot peridot's going to spin up via kubernetes enough pods to literally run all 2000 builds in parallel so because it's cloud native it actually has a com has all of the benefits of a cloud-based infrastructure associated with it and it's fairly easy to deal with uh you know clouds offer kubernetes services already and you can run it on-prem if you wish though it's incredibly flexible uh and and i i'm not sure if this you know how much this has been mentioned yet but the release of rocky 9
is going to be more than just a release of a bunch of binaries in a repository and some isos we're going to be releasing the entire build environment so other people can come and replicate our work which has already been done once now and possibly others but that's that's the goal of what we're bringing forth with with rocky is 100 community focused and a lot of people have asked us like when is this going to get released like you know there was just a whole there's the question and we were we were just you know um there was just a discussion on reddit specifically about
you know if people are really excited about rocky 9 being released uh great timing on that uh sylvie um but it really you know we're we're not in a race to release it's like racing to the starting line right the race starts on release and then it's how long can we keep going how long can we keep it up and the way to do that is that giant commitment to the community and so we are not rushing on the release of of nine and and to be blunt i mean this is an enterprise operating system very few enterprises are going to be like ah 9.0
is out let's go install that on our entire infrastructure today yay right most people aren't doing that they're gonna say well okay we have to now start testing we have to start validation process certification processes and it takes a while to actually start ramping up a new release now we're not sitting on our hands either at the same time we're we're there's a lot to get done we don't anticipate that we're going to be you know very late but just keep in mind when we release it's going to be releasing the entire stack the entire environment the entire tool chain build environment everything for somebody
to come along and go implement their own rocky linux and that is what's really driving this debate stability and the commitment uh to and from the community that's great thanks greg one of the other questions that i get asked a lot i know i think neil you hit on this a little bit earlier skip you probably did too talking about where does rocky 9 come from i mean everybody knows it comes from or equivalent to rel nine but i think kind of that up the chain where does it actually come from like how does it become rel and rocky yeah i could take this one
if you don't mind neil um so this is uh something that um a lot of people we a lot of us people who are kind of in the know about it we just assume everybody else knows it but i it needs explaining a little bit software comes from the different projects right whether it's firefox or bash or the linux kernel itself they all have different people working on it and ultimately rocky rel and stream come from fedora um this uh i mentioned earlier briefly um fedora is a what i would consider a cutting edge distro they release every six months great distro um if you're
interested in trying it out but they uh every so often now it's three years it used to be five years uh red hat will take the latest version of fedora and freeze it um in our case red hat 9 which rocky 9 is based on fedora 34 which is actually just leaving support i think fedora is already on to the next thing they're very fast and um red hat has um and other community members have taken fedora fedora 34 worked on it frozen the package versions the major package versions and they begin patching developing from there and um so effectively fedora 34 they are in
a way supporting it for 10 years right and so are we um and the the chain i guess the the down or the upstream to downstream goes fedora centos stream red hat enterprise linux rocky um and uh rocky and red hat enterprise linux obviously very closely linked together because our goal unusual for a distro but our goal is to make a completely hundred percent compatible uh distribution um so it's it's not what everybody likes but it's also um you know we it's worked out well so far let's put it that way absolutely is it bad greg that i remember running fedora in single digit release numbers i mean i'm talking about 82.
i mean 34 up there so chaos linux the predecessor of centos existed before fedora and the goal of chaos linux again what created centos the goal of this was to create a community managed rpm based distribution of linux that didn't exist before right there was there was red hat there was mandrake there was oh my goodness i'm already holding andre i'm about mandarin now man yeah before mandariva um but there was a bunch of these right cor do you remember carrell um and there you yeah yellow dog of course yeah yeah and yum came out of yellow dog uh originally yum even stands for the
yellow dog updated modified uh so um there was there was all of this existed nothing community existed but all of these distributions existed but they were all community excuse me commercially controlled and chaos linux was basically the first rpm based focus you know community distribution of linux and then interestingly fedora showed up shortly after now fedora just to get onto a little side track fedora did not start off being an operating system fedora was created by a guy named warren tagami who worked at in hawaii at the university of honolulu i believe and he actually created fedora fedora was originally poised to be what apple
is today red hat hired warren brought warren in and pivoted the project to be a community rpm based distribution of linux again after chaos linux was created take that for whatever it's worth i don't know uh if there was anything there no idea but uh because of that there was much less of a need to have chaos linux and centos was taking off so much that we kind of moved a lot of that effort from chaos linux into centos we did a little bit more chaos linux something called the chaos nsa node server appliance and really focused on a lightweight high performance distribution but most
people just wanted that binary compatibility that centos was providing and so uh we ended up kind of rolling that away and uh and then when obviously red hat pivoted the project left a gaping hole in the community and that's why rocky exists now it's awesome yeah i think whenever i hear mandrake that that's a funny one because i remember whenever dsl was new so having one and a half megabit in your house was pretty awesome i built a router out of mandrake that was how i was sharing it with my roommates at the time that's how long ago that was oh you you didn't have
um what was it the dual stream isdn to get you up to 1.5 because each each stream of isdn was 768 if i remember correctly i couldn't afford that at the time greg i i needed as much bandwidth for the least amount of money that could possibly pay and dial up wouldn't cut it so that was even before we could afford a wireless router so it was literally having a physical machine with multiple nicks in it running mandrake and running cables under the carpet through the apartment to the other wireless didn't exist then it would have been really expensive exactly exactly long time ago all
right can you post some kind of links to help people who'd like to contribute to the real life project i would say yes we absolutely can i don't know if you guys want to talk about a little bit what that looks like and how you contribute see that was another that was another shin kick right there zayn zayn rell nine oh my goodness i'm sitting here reading man [Laughter] freudian slip there um so yeah rl not release i just see rl and so we have um uh matter most is where we do a huge amount of coordination of the project um as well as on
irc and the two are bridged together so you can come into either one but that's where we do a huge amount of the coordination so people that are really interested in contributing and being part of this uh this is a community we are asking for people to come and join and help we are always looking to expand not only our teams but to be quite blunt the leadership among these teams and uh there are a few areas in which we actually actively need help right now uh so we need more help on the community side for sure but there's other areas as well so if
people are interested please jump into mattermost you can get there at chat.rockylinux.org or jump into the irc i tend to tell her when to go to mattermost at chat.rockylinux.org because i spend so much more time there than an irc but join in and jump into the relevant channels and start introducing yourself and we would absolutely love to to have more people be associated with everything that we're doing so um skip neil anything that you want to add from that perspective you want to debug packages come see us in development please have i got work for you [Laughter] it's gonna be yeah i think that's that's
that's pretty much it right like um the big work right we have right now is just some package failures where or a good percentage of the way there on rocky 9 don't want to give an eta but um you know we're looking to have something uh available to float around internally as kind of a release candidate with their testing teams at rocky linux um yeah so so there's there's definitely help that can be done there and uh skip and i are more than happy to to help give people some introductions to package debugging and building and uh give you a quick primer on how to
rpm probably a little bit of background in some c and some programming languages would help for doing some of that debugging um but yeah um it's even even even i have jumped into that that mix not so much for the nine but i remember in eight when we were getting our first release out i was i was definitely getting in people's way absolutely getting it people's away oh that's fantastic we're all learning together that's the important thing so how two parts of this one and one of them is yes matt before you say it i know i can read my shirt this time so it's
rocky linux i'm i'm here uh how long will rocky 9 be supported with updates and i don't know that we answered earlier on a release date for nine i know we've talked about what gets released with it but i know we've actually talked about when yeah so rocky nine um as stated before red hat enterprise linux nine uh based fight for bite packet for package version for version both of those distros are supported for 10 years so so that would make it 2032. i'll be so old oh my gosh yeah so that's don't want to hear it scary and that's i see the follow-up question there that when is the end of life and yes 10 years 2022 2032.
we'll see you then [Laughter] wow wow and that's what we do in in the greg mentioned we start the race the race in the enterprise world is a marathon we're we're here for for a long time we don't think in terms of you know this day or that day we've got months and years months you have months another good uh thing to point out as well is um like kind of how the updates and support works and it works the same way as rel in terms of having kind of a phase one and a phase two for uh longer term so it's really just the
first kind of five years of of the release that you get point releases and then we'll eventually get up to 8.10 and then 9.10 at some point and those will be the releases that that continue for the rest of their update their rest of their you know meaningful lives so um it's something to understand about kind of the the el phases of support and life so there there are kind of those two phases once they reach five years they get put into what what i would call maintenance or bug fix bug fix security fix only mode which uh if you want an example red hat
rel7 and centos 7 are there right now right there in support in 2024 i believe is the last uh last call for that and um start looking at migrating those i know those people out there they're not gonna they've gotta wait until three months before end of life and they you know oh my god we didn't know about this yes you did no skip there are people that will wait years years after how many row four boxes you got left so am i understanding that you guys did not tell the whole community on when 9.0 is being released what's up with that i could i'll
tell everybody exactly when it's going to be released tell us when it's ready yeah tell us before 9.1 i hope it could be an interesting conversation if it wasn't personally guaranteed before 9.1 well hold on a second a lot of organizations never use the dot oh exactly let's just skip it i like it no i don't like it never mind no no so i don't think we have any more questions we're kind of getting close to the end of time guys i want to give you all a chance kind of closing thoughts uh start with you skip yeah uh just that we're working hard we've
got um nine has three obstacles to it right now package builds i i and others are working on that we're about i would say 90 to 95 percent complete now very good the compose process this is where we produce your isos and your finalized repositories and the final bit is the testing obviously we're not just going to throw it out there we we and one's interested testing channels where it's at if you want to develop with us development channel as well but those are those are the hurdles to overcome and uh once they're there we will have a distro very nice and i see someone
posted uh they work for government entity and the guarantees there's sent those six boxes out there still i i run across customers within the last week two weeks they still have some uh real five boxes out there so i know i know that stuff's around when i was when i was still at department of energy um we had a case where somebody came and said uh a piece of hardware that we forgot about that was sitting underneath somebody's desk that's been vacant in an office for uh many years has now broken and we need that system it's a critical piece of our pipeline and so
the hardware was was burnt up i mean it was gone i mean probably if anyone was there when it when it died there was probably smoke coming out of it i mean it was pretty bad um you know that much dust on a motherboard you know does that does bad things does bad things especially when the fans stop spinning because of all that crud um but we looked at the operating system it was running you know red hat linux eight is it red hat linux 8.
so yeah uh and the source code for the code uh that that was running was not available so speaking of containers they not only work to get you newer libraries and and binaries and whatnot it also worked to run we actually containerized red hat linux 8 and this application and then we had it running on at the time i think current was like centos 7-2 i think maybe centos 6 something but it was around that generation uh late 706 you know early centos seven but uh we were able to do that and bring that system back to life as a container granted don't try to
do a ps or a top because it's seg faults because proc file system has changed so much in the last 15 years that's eight i think it works that's like 98.99 like pre-2000 type time wasn't it craig um no already close close so if you remember um what what caused centos to exist was when red hat linux was end of life and red hat linux went from 7 1 7 2 7 3 and then it went to 8 and then there was never an 8.1 then it went to 9.
and then all of a sudden they basically said all that stuff that we just did let's just get rid of it and we're gonna actually i think they kept one or two of the versions for a little while uh still alive but they had this new thing called red hat enterprise linux which i believe started with version two again most people shy away from the 1.00 so i think they just jumped to version two because i don't remember there ever being a red hat enterprise linux one it actually went to 2.1 i haven't looked at this in a long time march 23rd 2022.
the first word 2002 was 2.1 20 years ago wow 20 years ago so that was the creation of that was what triggered the creation of sento so uh that was in 2003 if i remember correctly it's when it was first released um i'm not going to tell you how old i was because greg will kick me in the shins again you know i went to a government meeting in the pentagon and and if if he's watching he's going to laugh i'm not going to name him but i went to a government meeting in the pentagon with that with a fairly high-ranking official who who reported
to the secretary of defense and i'm sitting there and we introduced ourselves and i've mentioned kind of my background and and whatnot he goes oh i remember that i was 15 at the time and i i couldn't kick him because he was a high-ranking government official and you know in the defense department i wanted to though wow yeah that's we're getting older not as old as skip is going to be though at the end of uh in in 32.
is that what you said the white the white hairs are coming man it's coming that's what all this is feeling coming it happens and it happens fast all right i think that's the end of time guys neil i don't know did i give you a chance to close i'm sorry i think we just started talking old stories again it's okay it happens i don't have anything okay well we really appreciate you guys joining us again today uh if you are interested in what we're doing here by all means go look at the job site you come on greg you got to close okay greg i
don't have anything man i just didn't like that you you totally went over me you already talked i thought you gave yourself i know i i took neil's spot i know i know okay do you want to tell people that we have openings i'll let you do that oh we're hiring uh we're hiring quite a bit right now so um we are looking for uh senior engineers senior developers c linux kernel user space containers as well as go back end developers full stack developers front end developers if you're interested reach out to us you can check out job board on our on our website honestly
we haven't even been able to keep that up to date that's how much we're hiring right now so um please reach out to us if you're interested um you can reach out to me personally any of these guys um linkedin mattermost slack wherever you find us email uh we'd love to hear from you looking for great people excellent again we appreciate you joining us today we will see you again next week and if you don't mind like and subscribe so you can keep up with us and we keep up with you appreciate it see you later
Built for scale. Chosen by the world’s best.
2.75M+
Rocky Linux instances
Being used world wide
90%
Of fortune 100 companies
Use CIQ supported technologies
250k
Avg. monthly downloads
Rocky Linux
Have questions about your infrastructure?
Talk to a CIQ engineer about Rocky Linux, HPC, and AI infrastructure.