Webinar: Rocky Linux 9: What's New?

Webinar Synopsis:

Speakers:

  • Zane Hamilton, Director of Sales Engineering at CIQ
  • Neil Hanlon, Solutions Architect at CIQ
  • Gregory Kurtzer, Founder of Rocky Linux, Singularity/Apptainer, Warewulf, CentOS, and the CEO of CIQ
  • Skip Grube, Senior Linux Engineer, Rocky Linux

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Good morning. Good afternoon. Good evening. Wherever you are. Thank you for joining us again for another CIQ webinar. This has been an exciting week for Rocky Linux and for us. So, we wanted to share with you guys some of the updates about Rocky Linux 9, the excitement of coming out, and Peridot as well. So today we have Greg, Neil, and Skip. Greg, everybody knows Greg. I'll let him go last.

Neil Hanlon:

Nice transitions. I like it.

Zane Hamilton:

Have you been on with us before?

Neil Hanlon:

Yeah, the 8.6 release a couple months ago.

Zane Hamilton:

Well, go ahead and introduce yourself again, Skip.

Introductions [00:39]

Skip Grube:

Hey, so Skip Grube here. I do a lot of random release and dev work for Rocky Linux; all kinds of things. A lot of package fixes, lots of troubleshooting, lots of helping other people, just all over the place. So I'm on the release engineering team, along with a lot of other talented individuals we have.

Zane Hamilton:

Fantastic. Thanks, Skip. Neil, it's been a while. It's good to see you.

Neil Hanlon:

Yeah. It's been a little bit. We've been heads down with Skip working on Rocky Linux 9 and getting Peridot out the door and a lot of other really fun stuff. So I’m super happy to be here and talk about it. Yeah.

Zane Hamilton:

Great. Thanks. Greg.

Gregory Kurtzer:

Hi, everybody. These guys, I think, are really going to be the star of the show today. My background in it is nowhere near as fun or amazing as these guys and the rest of the Rocky crew. So I'll probably be able to help a little bit with giving context on the story, like how we got here, but these guys are the stars.

Zane Hamilton:

Let's start with that. Let's go ahead and start with the “Why are we here?” story

Rocky Linux Background [01:59]

Zane Hamilton:

Why is Rocky a thing?

Gregory Kurtzer:

Well, Rocky is a thing because our operating system supply chain was kind of disrupted, and due to the project CentOS kind of changing direction a little bit, it left a lot of the community needing a solution that was really kind of back to what CentOS was always designed to be. So, we basically spun up Rocky, I believe within an hour or two of the decision to end-of-life or pivot CentOS, and it really just started off with a comment to their blog. And from that comment, from that blog that I made, and I kind of pointed people to a Slack that I was sitting on, it just blew up. As a matter of fact, I remember watching my Slack messages and getting messages on my phone, and it was just overwhelming, like how many people started joining. Within about a month and a half, we were topping 10,000 people and it just kept continuing on.

So the first task that we had to do was to kind of develop and manage this community of 10,000-ish people. Of course, it didn't start off with 10,000, but it was accelerating quite amazingly. So we broke the Slack into different channels that focused on different areas within the project, and that was a tremendous help, because now, all of a sudden, everybody started kind of congregating towards different areas of where they wanted to help and what their background was. And while I was trying just to manage all of this, I started looking into the channels and it was a perfect meritocracy, like certain people within certain groups just started immediately knowing exactly what to do, what had to happen, and what we had to do. And the next thing I know is, I was planning on rolling up my sleeves and jumping in, but luckily, there are thousands of people, a heck of a lot smarter than me, all jumping in and solving these problems. And that sort of started to become our teams, and then our team leads and our team deputies kind of came out of that. So it took us a little bit to build up the community and set up the starting point of what Rocky is, and we built this infrastructure that would really ensure that we don't have the same problems that CentOS had. 

Now, let me clarify that. CentOS has been a tremendous operating system and value to the community, I mean, for literally like over 15 years at this point, but the last bit and how it kind of transitioned was a little bit adversely affecting many people in the community. So we wanted to make sure that we didn't put Rocky into a situation like that. And that was our goal. So we started off by building an infrastructure, and in this infrastructure, it was really designed so that we can have lots of people from the community take part and be part of this, and that community and that infrastructure was being led by a whole group of people from all over the place. And, the next thing I know, it took a little bit, but within about four months, we had the infrastructure done, threw the operating system at it, built the operating system within about two months, and six months later, we had our first beta, and a month after that, went GA. And that was how Rocky came to be.

Now, we started everything from scratch. We built everything from the ground up specifically for the Rocky community, with, as I said, a lot of different people, completely open development models, and with anybody who wants to be part of this was able to be part of this. And it was just inspiring, and it's an honor just to be part of such an amazing community. I'm gonna fast forward a little bit, and then I'm gonna be quiet. It's hard for me. But I'm gonna fast forward a little bit. When we did version 8, we started off with Koji, which I think these guys will talk a little bit more about here in just a moment, but that's the Fedora build system. That's where we started. From there, we recognized that, "Well, we want to build our operating system a little bit differently, and we want to have the community involved in a little bit of a different way than Fedora and CentOS previously."

So, we identified the need that we really wanted to build a cloud-native build system. And we've been heads down working on that, and Mustafa, who unfortunately can't make it to this session, has done a huge amount of work leading that project and kind of being the visionary behind that project. And what it does is it creates a cloud-native microservice platform that eventually everyone's gonna be able to use Helm charts to install on top of a Kubernetes instance, and that will become an entire build system that you can either replicate everything that we've done and create your own spin, or your own version of Rocky Linux, or a whole new operating system based on Rocky Linux, and/or organizations can now enhance and extend Rocky Linux using this platform. And again, we're going to make this super easy for people to deal with and for organizations to leverage. And it also facilitates and helps the development of Rocky Linux and even more so, and a benefit to the community, it helps to ensure that, again, what happened with CentOS doesn't happen with Rocky, because now we have that capability and that infrastructure that anybody can go and replicate our work.

Zane Hamilton:

That's exciting. It's a great background. Sorry, Neil.

Neil Hanlon:

Not just the replication of our work, either, but the ability for folks to just have a location and ability to see what's going on, how things are being built, and go and look at the details of all of that. And looking into the future, we’re wanting to integrate more software bill of materials features where you can see really exactly where the pieces of your images and your infrastructure are coming from and validate that they're coming from the right place. So the future vision of Peridot and in the build tooling that we're working around for creating EL is not just about Rocky, but about all of enterprise Linux and trying to make it a more secure, better place. And we're excited to be working with partners upstream and whomever else to work on making it a more secure, better place.

Zane Hamilton:

And open.

Neil Hanlon:

And open.

Rocky Linux Release Cadence [08:54]

Gregory Kurtzer:

You know, one of the things that's come up a number of times is the release cadence of Rocky Linux. And I'm going to touch on this just really briefly. It took us a little while to build everything from scratch when we were first coming out with version 8.4, which was our first version of Rocky Linux, and it took a little while to build all of that from scratch, to set up the community, to create our Secure Boot shims and manage all of that infrastructure. With 9, unfortunately, it took a while again, because we were creating Peridot, our new build system, and we wanted to ensure that that build system is ready at the same time as Rocky 9. So when we release Rocky 9, we're not releasing just some binaries and some repository metadata and installers. The goal is to release everything, all at once. And we felt as though we'd be doing a disservice to the community if we released just the binaries that nobody else could replicate, that nobody else can build easily. So we wanted to create all that. So we see it as: it wasn't necessarily a race to the starting line. That wasn't our goal. Our goal is to create a sustainable project that can last and sustain for decades to come and become an extraordinarily, just stable foundation for everybody who wants to use it. And plus, the environment that we've now created will allow us to even faster start managing updates, point releases, and then major .0 releases as well.

Zane Hamilton:

That's great, Greg. Thanks. I think I received, I don't know how many texts and messages yesterday asking about Peridot. So it's a very exciting thing that everybody wants to hear about, and we will get to that. I know we have a couple questions already coming in about Peridot, so let's hold off on those until we get through. Let's talk about Rocky 9. So, tell us about Rocky 9. What are some of the features in it? And then, guys, we will dive into what Peridot is, where it came from, and kind of do that at the end. So, tell us about Rocky 9.

Rocky 9 [11:03]

Skip Grube:

I'll open up, I guess, if you don't mind, Neil. So one of the biggest things that we have improved upon Rocky 9 is our architecture support. So Rocky 8, we're able to do x86 Intel processors, your bread and butter, and AArch 64, runs a lot of phones, your Raspberry Pi, and all kinds of embedded devices. Now we have architecture parity with the upstream, which means PowerPC 64 (ppc) and x390x, which is actually the IBM mainframe Z series processors. So Rocky on mainframe is here, if you want it. It works quite well, actually. We've had some new sponsors, I'm not going to list them all right now, kind of help us out with this bringing our build hardware into the pool, because these things are kind of beasts. Let me tell you: these mainframe and PowerPC build boxes, I can't afford them.

Neil Hanlon:

They build really fast, though. That's for sure.

Skip Grube:

No kidding.

Neil Hanlon:

We were compiling kernels in like 23 minutes or some odd like that. Yeah, it was like the one case where it was beating out building for every other architecture.

Skip Grube:

Opened my eyes a little bit. 

Neil Hanlon:

Also, we have some community members for 9, in particular, because of the 10-year life cycle that we're going to have until 2032 now, they want to work on porting armhfp, which is 32-bit ARM, and possibly want to look into other architectures. There's some work that needs to be done upstream for something like RISC-V, and that kind of support needs to start upstream and come down, and we'll see what we can do there, but it would be interesting to import things back. I know, I got Greg really excited by saying his favorite word.

Skip Grube:

It's on the roadmap. It's just there's a lot of work that has to happen.

Neil Hanlon:

It's on the roadmap, but the road goes like this.

Skip Grube:

Big roadmap.

Updated/New Features [13:21]

Neil Hanlon:

No GPS-enabled on that.

There's some new features in Rocky. I mean, again, like we said, with the 8.6 release, that was a point release. It's both an exciting update and not so exciting. So, there are some changes we have now. I have a modern kernel if we can call it that. It's the 5.14 kernel, so it's still a little bit older and it'll continue to have back ports for 10 years, just like the 4.18 kernel, and Rocky 8 has back ports–tons and tons and tons of back ports–on top of it at this point. But getting to 5.14 is a great advancement for a lot of security changes; you know, the Spectre and minor Spectre vulnerabilities in the hardware, those are kind of baked in now, and there's some improvements there for a lot of newer processor types and improvements in performance on those newer processor types.

Skip Grube:

I'm personally very excited because the 5.14–the 4.18 kernel is going to continue to be backported and supported for seven more years now, I guess through 2029–but 5.14, you'll find, particularly if you use a desktop, will have exponentially more hardware support, particularly for recent hardware. So I like that a lot.

Neil Hanlon:

Yeah. There's an absolute explosion in the past few years of really, really, new and fast and quickly iterating processor types. So there's a lot of kernel development to go along with that.

Skip Grube:

Open end video drivers!

Zane Hamilton:

Hear that one a lot too.

Neil Hanlon:

We also have a lot less modules. So, if you weren't a particular fan of the modules that were introduced in Rocky, or sent to us in RHEL 8,  there are actually no modules right now in 9.1, and I think, Skip, they're supposed to be coming in 9.1?

Skip Grube:

Yeah, so in 9.0, there are no modules, which, as a guy who primarily works in the package world, I'm actually very grateful for. That would've complicated our lives even further. Modules, they're very useful for multiple versions of support, but they are not simple and they're not easy. Some of the tooling around them is a little clunky but, we can see them coming in 9.1 already. They're starting to make their way into the CentOS stream on our upstream providers, and so we're looking forward to different versions of different software coming in as modules, but they'll be introduced gradually instead of the all-at-once shebang that happened with 8, which again, I think it's a good call.

Neil Hanlon:

And hopefully with the default modules–that's a big change as well. Whereas a lot of modules are enabled by default in 8,there’s supposed to be no default modules enabled, and you'll have to pick and choose to enable those, when and where you want them. So that could be a welcome change, but it is something to be aware of as well, going forward, that you won't necessarily automatically get the software that you're looking for without enabling a module.

Skip Grube:

It'll simplify it for end users is my opinion on the matter.

Neil Hanlon:

Agreed.

Skip Grube:

And I like that.

Neil Hanlon:

Some networking changes: 5.14 brings in WireGuard. Red Hat calls it an unsupported tech preview. I haven't gotten a chance to try it out myself, but now if you were familiar with trying to run WireGuard on CentOS 7 or RHEL 8 products, you had to either compile a new kernel or get a new kernel from ELRepo or use some user space drivers instead of the kernel ones that worked as performance. So this could be certainly a welcome change if you're doing site-to-site VPN connections or kind of road warrior things with your people connecting to different servers. And then kind of the biggest change, really, is that the iptables-nft integration point as well as ipset are deprecated now for nftables. So if you haven't yet started to learn about nftables and NFT (not the crypto ones) to manage your firewall rules on your Linux boxes, probably a good time to start looking into NFT or something else to manage those. 

What else do we have for changes? Again, it's not a lot of new stuff. 

Skip Grube:

New programming languages. We got Python 3.9 by default now, which I know a lot of people are excited about, including myself. PHP 8, which–PHP in the Red Hat land has been dragging feet, quite frankly, for a long time. I'm glad to see us kind of catching up in that regard. And for the compile-minded among you, GCC 11.2, nice and shiny as far as Enterprise Linux is concerned, the LLVM, the CLANG, C Lang 13, Rust 1.58 and Go 1.17. So we've got definitely more modern, better, bigger major versions of some of these tools coming out, which again, I know a lot of developers really appreciate.

Neil Hanlon:

For sure, and on the desktop side I had mentioned GNOME 40 and, as well, PipeWire is now replacing PulseAudio for the audio networking into ALSA. So, that's something that you can look forward to. I've been using Fedora here on my desktop for a while and have been greatly enjoying the improved functionality of PipeWire and WirePlumber over PulseAudio, especially around Bluetooth devices. Like, it's very good.

Migrating [19:12]

Zane Hamilton:

That's fantastic. We're starting to get some questions coming in, and this is one that I knew was going to come up eventually, but talk about migrating. So migrating… upgrade path for reprovisioning on new boxes: is that the way to go still? So moving from major version to major version–so 8.4, 8.5, 8.6, moving to 9–is there a direct path, like in-place upgrade, or do we still recommend kind of doing that reprovision?

Skip Grube:

I'll take this one, and Neil can correct me if I say something incorrect.

I'm a little opinionated on this. I'll have to–I'm going to tone it down a bit. There is a project; I think it's called ELevate. It's based on Red Hat's Leapp framework, I believe it's called, that attempts to do major version upgrades from the different Enterprise Linux major versions, 7 to 8. 8 to 9, I don't think is currently–I think it's being worked on? Having said that, I personally don't recommend it, as a working Red Hat-based admin for many years now. It's just not a good idea. Like, unless you have a very, very basic Linux, Apache, MariaDB, PHP stack, even PHP might give you trouble. It's very prone to errors because you have to understand that going from Red Hat 8 or Rocky 8 to Rocky 9, or the other question we could ask about is going from CentOS or Red Hat 7 to Rocky 8, though there's five years between 7 and 8, there's three years between 8 and 9, that's like a lifetime in open source development. There's all kinds of things that have changed and new major versions of different glimpse-dependencies, all kinds of things that get modified. And I think a much better, particularly if you have a lot of servers or machines to migrate, a much better use of your time and resources is coming up with a configuration management where you can stamp out new machines, have them set up the way you want them automatically, very quickly, and have it go even faster than a migration would. And this is just my experience over years, and personal opinions being kind of mangled in there, but check out ELevate if you want to get involved. But again, it's kind of… you have to understand there's a lot of exceptions, particularly if you use third-party repositories, you could run into some trouble, so no matter what you do, back everything up, right? That's regardless, but it's really… it's possible, yes, but it's fraught with peril.

Neil Hanlon:

I think that's a really good way to put it. We have done an install of minimal 8 and swapped the repos over to 9 and done some magic to get it to upgrade. It does appear to work.

Skip Grube:

A minimal application package.

Neil Hanlon:

With minimal, no extra packages, no nothing. If you have, like, a machine and backups and you want to try it, and you're not using it for anything important, then you might be able to get away with it. But from the Rocky Linux community side, that's not something that we support or recommend.

Skip Grube:

Say, as a Deb, I have to put a big giant disclaimer here. So, this is no warranty at all if you try to do this. In our Frankenstein lab, we've been able to take a Rocky 8 machine and turn it into a Rocky 9 machine, given very specific conditions and almost no software installed on it, but...

Neil Hanlon:

It does require erasing some packages...

Skip Grube:

Yeah, there's some surgery involved.

Gregory Kurtzer:

I just wanted to jump in and just echo what these guys are saying. To me, it's the argument of pets versus cattle. And when you look at a single system, that's getting a lot of human attention and whatnot, sure, it's definitely possible. As a matter of fact, I've done it numerous times, and I come originally from the Debian side of the playground and where that was commonplace. Now, the problems with it, though, and even with, when I saw this with Debian, and this is way back in the nineties…

Skip Grube):

Also fraught with peril and Debian.

Gregory Kurtzer:

Yeah. It is configuration files and whatnot. Syntax has changed, parameters in configuration files change, little things like that. It is not necessarily just a seamless transition and it always requires some amount of human attention, unless, as you said, it's just a cookie-cutter kind of temp standard LAMP stack or something along those lines. So, I would typically, and this is just how I think, I would always kind of move towards the cattle model and start doing things like configuration management, automated kickstart provisioning and whatnot, and I actually like to do that anyway, so if I ever have a hardware problem or something, I can always stand up a new system or upgrade a system and/or roll back very, very easily.

Skip Grube:

It will give you more benefits than just upgrading this one time, basically.

Neil Hanlon:

Yep. And you're prepared for not just the upgrade for 9 or rolling upgrade as you roll everything down, but you're prepared for 10 and 11 down the line. It's good hygiene, for sure. Agreed, it very much depends on the situation. There are some cases where it might make sense to do it, and there are some cases where it probably doesn't make sense.

Zane Hamilton:

Tried this on an older version. We're talking like 4 to 5 or 5 to 6, and everything seemed to go fine. And a couple months in, we started installing some new software and started running into some really bizarre issues from weird packages that were left hanging around. So, even if you don't see it day one doesn't mean that it's not going to cause problems later.

Neil Hanlon:

This is true. This is very true. Yeah.

RHEL Packages [25:24]

Skip Grube:

I see a question flash up on the screen here. So, is Rocky 9 currently caught up with the latest RHEL packages? Yes. That's an easy answer.

Neil Hanlon:

I don't think we've published them yet to the mirrors, because we've been trying to give the mirrors time to take the initial sync. We wanted to have those published when we pushed the ISOs, and then the initial compose, but we also wanted to get it out there and get people playing on it while we finished up the internal sides. I guess the one caveat to that would be around the shim package, which is a note for 9 that because we still need to get our shim approved by Microsoft and signed or approved by the shim review committee and then subsequently signed by Microsoft, Rocky is still using an older validated version of the shim package. So that's one difference.

Skip Grube:

I'll elaborate on what Neil says.

Neil Hanlon:

Yeah, please do.

Skip Grube:

Shim is the package that does Secure Boot and Linux. It'll allow your system to boot with a Secure Boot enabled motherboard and Rocky 9 is Secure Boot certified. It will boot on your Secure Boot hardware. There are small little bugs with the current version of Secure Boot we are approved for, basically. And we have currently underway–us and the rest of the world quite frankly–are currently chomping at the bit to get the updated, we call it the shim package, but it's the updated package that works with even more UEFI hardware.

Zane Hamilton:

Thanks, guys. Alan, it's good to see you. Thanks for joining us. Alan's not asking; he's talking about predictable network interfaces and how they're not at all predictable.

Neil Hanlon:

Just real quickly on, the last point about our packages. Skip has a site called Incoming that hosts a list of the packages that are in RHEL, and as they're coming in, it listens to the feed of packages that get published. So you can kind of go there and see a table of what packages are coming and if they're imported into our git, when they're built, and when they were built and packaged for that.

Zane Hamilton:

Is that a link that we can post?

Neil Hanlon:

Yeah, definitely.

Skip Grube:

I'm gonna give it to you right now, actually. Oh, Neil's much faster.

Neil Hanlon:

Yeah, we can, we can post that. Wherever links get posted.

Zane Hamilton:

That'd be great. We will post that below this.

Skip Grube:

Thank you. Because, yeah, it's just something I had noticed that we kind of wanted for our release engineering group where, hey, we want to find out if there's an upstream update. We want to get it as soon as possible. And we want to know about it and find out, almost like an audit trail, right? Like, has it been imported into our source control? Has it been built? Has it been published? Etcetera. It's like Rocky Linux is all open, so why not publish it to everybody? Because you know...

Neil Hanlon:

The question we get pretty frequently too is, "When is X package going to be built?" And now we can point and say, “There, you can go track it yourself. You can see when it will end.” We'll, in the future, integrate that into Peridot, which is probably a good segue to talk about Peridot.

Zane Hamilton:

Absolutely. I was just about to ask. So we're talking about it! So let's go ahead: talk about it. Tell us about Peridot, why it's so exciting.

What is Peridot? [28:51]

Neil Hanlon:

Well, Peridot's pretty exciting because it is a, like, rare earth gemstone that's found in only certain places under high compression, but that's a different type of Peridot. 

Zane Hamilton:

What color is it, though?

Neil Hanlon:

It's green.

Skip Grube:

Green, of course, just like the [points at Rocky Linux logo]. 

Gregory Kurtzer:

I can’t.

Skip Grube:

That's the joke.

Neil Hanlon:

So Peridot is the new Rocky Linux build system. I've taken to calling it our distribution forge because it is the thing that takes Git and forges our distribution. Right now, it is a place that packages go to be built and presented as a repository. And so the process for building and integrating is the same as it was in Koji; In fact, we're using all the same tools under the hood. We're using mock to build packages, which invokes RPM build, and all of these things are being built in a standard way. The only thing that we've really changed is how it gets done and the speed at which we can do builds. And part of the cloud-native build system here is also allowing us to highly parallelize these builds and do them very quickly, which can be extremely helpful when testing and doing other things.

So, as a just-throwing-out-there number, we were able to build over 2,500 packages in parallel for x86 and AArch64, both. So there's probably even a higher limit on that, but we're trying to be good custodians of resources in the cloud. And it was a very interesting place to see that we were building these packages in a time frame that we never would've been able to consider before. In addition to it running in Kubernetes, it means that we can essentially consume Kubernetes on any architecture that we want to as long as we can run Kubernetes on it, and that's how we're building, for example, on x390x and PowerPC. Skip, do you have anything else you want to add about what Peridot is?

Peridot Projects and Structure [31:08]

Skip Grube:

Oh yeah. There's a lot of things. I think one of the great aspects of it is that it has a structure to it that divides things into projects. And so projects can be big, like Rocky Linux 9 is quite a big project–it's got 3000-something packages in it–or they can be small. They can be as little as one package. Right? So, for example, we have a special interest group that builds kernels specifically for the cloud. I think it's called SIG Cloud or something. You guys can correct me if I misspoke. SIG Cloud is in Peridot as a project, right alongside Rocky 9. It uses the same stuff that Rocky 9 uses to build things, and it builds these special kernels for cloud infrastructure. One of the things that we're spinning up, possibly as early as this weekend, is the Raspberry Pi special interest group, which–hey, how you doing? I'm a founding member. I'm really excited to have our own little project in Peridot. That way we can build Raspberry Pi kernels and related packages to make that work right alongside the same infrastructure, the same everything that we use for the rest of our builds. So it's very cool that we can allow different SIGs to have their own space in the official build system or if a company or individual is interested in deploying their own, I think it's pending some more documentation. I think it's very possible and even easy. It's early days yet, but I think it will be quite simple to get your own, as far as building your Kubernetes cluster and such go, but to get your own Peridot running, if you're interested in that kind of thing.

Neil Hanlon:

Yeah. And it runs really well on a desktop or a laptop. I mean, you need a certain amount of RAM, but the resources that you need for it are ultimately just PostgreSQL database and S3 compatible storage–I use MinIO personally, locally, and Peridot. So you need Kubernetes; you can run it on K3s; you can run it in microK8s. There's a whole wide range of options on how you can run this, just to do local builds. And if you're only talking about maybe one or possibly two architectures–I don't really know any companies out there that are running on more than two architectures, especially when the other two are hard to get to–but it should be something that a company can look at and say, "We have these packages. We need to patch them," or, "We want to build our own packages in a way that makes sense." Maybe they're not derivating at all from the upstream or you're not patching but you have a certain set of RPMs that you build for internal use and you want to have a better way to orchestrate the building and integrating those into your CI. Peridot should be something that can be looked to for use in building Enterprise Linux packages and distributions, even outside of making a distribution in and of itself.

Open Patch [34:27]

Skip Grube:

To add on to that, I want to say a couple words about–it's got different names during Rocky Linux's history but I think the official term now is ‘open patch’–so really one of the problems that we wanted to solve, and we've done this in our previous build and import systems to an extent, but it really, really shines for ease of use in Peridot, in my opinion. We have a situation where we have upstream sources and ours are the RHEL sources, but it could be anyone really, it doesn’t really matter where they come from.

Neil Hanlon:

It could be our sources.

Skip Grube:

It could be our sources. And we have a situation where some of those sources, when we import them into our Git, they're required to be patched. We need Rocky-specific patches on a nontrivial number of packages, and doing this manually would be like pulling teeth. I mean, you can do it; absolutely, you can. The advantage comes, and let's say you import these sources and now upstream releases, a new version, right? We have to release an update. And so you have to pull those sources in again, right? Obviously. And then you have to apply your special Rocky patches to them again, and doing this manually over dozens and dozens of packages is not fun. We, quite frankly, we don't really have a lot of manpower to do that and so one of the things we've come up with is kind of almost like a meta patching system where we have a patch and during import, when we import from the upstream RHEL, the patch applies the same way it did for the last version, automatically. So we imported Bash version 4.5, and let's say we needed to patch it–I'm just coming up with a random package name–and then Bash 4.6 gets released and we need to apply the same patch. Instead of doing that manually, we have a list of changes that we want to automatically apply before it actually hits the Rocky Git system. And Peridot allows us to do that. And it's wonderful as far as if you're someone like me, who probably would, if we did it manually, I'd probably end up doing a lot of it. I don't like work, so... 

Neil Hanlon:

Yeah, that's really true. Being able to apply these statements and you can think of changing something in a spec file, maybe you need to add a build requirement for whatever reason, because it doesn't exist in our builder, that's something we've seen before; if you need to apply your own patch on top of the code–so maybe there is a patch that hasn't been backported or hasn't been backported yet and released–that can be taken and applied on top of the existing patch to create a kind of zombie package that has some patches that are pulled from different places, and it automates, not only adding those files into the SRPM, but also adding the commands as required if necessary to add them to prep, to be applied, and dealing with the formatting there.We use it in some cases to just completely replace files at build time. So an example of this is that shim package that we were talking about with Secure Boot. We have to provide our own certificates for that to be built. So, one of the things that srpmproc does is it's able to… srpmproc is a tool or library that works with source RPM processing and open patch. It allows us to say, instead of using this file, replace it with this other file that's in our cache. And so those sorts of patches and flexibility allows us to quickly make the changes we need to without being overly verbose, which is really what we're trying to do. We don't want to spend a whole lot of time going and trying to, like, construct gifs and patches by hand. We just want to write it in a simple language and say, "We want to replace CentOS with Rocky," or whatever else it is.

Similarities to Personal Satellite [38:46]

Zane Hamilton:

So yeah, this is one of the ones that popped up earlier. So the question is, "Is Peridot akin to a personal satellite?"

Skip Grube:

So again, I'm gonna talk about this and Neil's obvious; I'm sure Neil will have something to add here as well. Copr, I pronounce it Copr, but I don't really care. I can see it a little bit. It's not really, I wouldn't call it personal, at least as far as the official Rocky Linux build system goes. And we could put up a link to it as well as we want; it's completely–you can see all, what we're building, what we're doing; you can view Peridot if you want, as an unauthenticated user, and Neil, I don't know if you want to give a link out to that. But I would call it similar in a fashion to Copr because you can build and make available packages. But the similarities kind of end there. I think at least our Rocky Linux instance of it is geared more towards special interest groups, not personal repositories. We could give personal repositories to people. There's no reason why we couldn't, but I think that the idea behind the project is to, and thinking back on it, maybe a special interest group just has really one person behind it. So it would effectively be their personal one, but at least for our official build system, it's got to be related to a group in some way. Right? So for example, the Raspberry Pi special interest group, we have lots of people who I'm sure are very trigger happy to get builds out in Peridot. Copr, as far as I understand, is more of a "Hey, you get an account and you can build and publish whatever packages you want.” Maybe other people will pick them up. Maybe they won't. It’s more of a group thing.

Neil Hanlon:

Yeah. I agree. That's a good description of it. It's less of a self-service method of building packages and more of a central place for our community. Specifically, the Peridot-hosted instance or Rocky Linux-hosted instance of Peridot that we're using to build special interest groups and build the distribution itself –those are for Rocky Linux, but you could definitely see it as kind of a self-hosted Copr. We have discussed in the past maybe even doing something like that, where there is a community build system version where a community member would be able to have their own projects being hosted there without having to go through the uplift of hosting Kubernetes and everything else themselves and getting that running, though we are trying to make it very friendly. Sometimes that's just not necessary. I'd say that if a community member or someone had a project that they wanted to work on, then we'd certainly try to enable that if we could in some way.

Zane Hamilton:

So, we've got a question. "Can I, and how, can I code for or get involved in Rocky and Peridot development?" So how would someone get involved with Rocky and Peridot development?

Getting Involved in Peridot [42:18]

Skip Grube:

Absolutely. Basically we have–I think we covered this last time as well–we have our, I would say, personally, our chat.rockylinux.org is kind of our nexus of communication. And it's bridged with libera.chat, so if anyone is IRC-minded, hop on #Rocky on there and it'll be fine. Hang out with us and our general or development channels and look around the website. You'll see links to where the source is hosted for both Peridot and for other projects that facilitate Rocky development and ask questions. Yeah. Talk to us. We love teaching people. We love chatting it up. We eat, sleep, and breathe this stuff. So, I don't know if you have anything else to add, Neil.

Neil Hanlon:

The best place, so we're doing all the development on GitHub in the Rocky Linux organization. So I think we just posted a link to the Peridot repository; all the code is there.

Skip Grube:

All requests are welcome.

Neil Hanlon:

My favorite way. PR is welcome. Not even in an ironic or malicious way, it's something that we'd love the community to get involved in. The documentation, again in full transparency, is not where it needs to be for a lot of people to get involved, start using it, start breaking it, but hopefully we can get it to a point where, very soon, and that's something we're focusing on now, that it is open source, getting it to a point where the community is able to just come and work on it and fix things and help write the code and help make it better. So lots of work there to be done and we'd love to see you around.

Gregory Kurtzer:

One thing I would add is the amount of communication that occurs on our chat instance, our Mattermost and IRC, is massive. It is massive. We have a fantastic community of people. I believe it's near 7,500 people right now? 7,700 people right now in our Mattermost plus IRC. So there's a lot of people. There's a lot of resources within there, whether if anybody has questions or if they want to be part of development that's going on. Almost 100%–there's a couple small side conversations that occur–but almost everything is in the open, every part of development, every part of how we're managing everything from accounts, GitHub, and whatnot. Of course, credential specific things happen in private and security-related matters, but anybody who wants to come in and literally eavesdrop on exactly what's happening with Rocky, it's a fantastic place and very friendly; a lot of really amazing and great people there.

Skip Grube:

We pick up tips and tricks all the time there too.

Applying Security Patches [45:25]

Zane Hamilton:

It's fantastic. We have one more question from Alan or a question from Alan. So he's asking for clarity on the procedures for applying security patches.

Neil Hanlon:

So I think the best way for you to patch the system and get the security updates would be just # dnf update --security. There's a few other commands that can be done to see what those updates are but the dnf update command with the security flag should go and look at the repository metadata to see which security patches need to be applied on your system, if necessary. If there isn't a guide, we should probably make one on our docs.rockylinux.org site. So I'll look for that and try to get a link over to you, and if we don't have it, then we'll write one. 

CIQ and Rocky Linux Relationship [46:20]

Gregory Kurtzer:

So I just saw a question come into me directly from another Slack channel, just to me personally. So I just wanted to answer it real quick, because it actually is something that comes up quite a bit. The question was along the lines of, "Why is CIQ doing a webinar and intro about Rocky and what is our role with Rocky?" and I wanted to kind of take just a quick moment to kind of explain the relationship between CIQ and Rocky.

CIQ is a principal sponsor and partner of the Rocky Enterprise Software Foundation and Rocky Linux. We are here to help. We contribute many resources as you can see, people as well as PR and other things to the organization, but CIQ has no control, no authority over Rocky Linux.

We don't control it. We don't own it. We don't hold it hostage. The community is free to go in the direction that is best for the community, and CIQ is simply here if anybody needs help. We put resources back into Rocky because we believe in it as an open source project and as a community. Yes, we were there for the founding. We facilitated that foundation, but really we're just a sponsor and a partner, and we provide services and support and other kinds of features, capabilities, enhancements, and whatnot around Rocky Linux. But again, Rocky Linux is an open community and we love it when other organizations come in and also offer support and whatnot. Very recently, there was just an announcement, as an example, of Google partnering with CIQ and supporting Rocky directly for all customers of GCP. There was another press release that actually just went out earlier this week of Google optimizing Rocky Linux for GCP. So, we love having others as well, and other companies being part of this community, and everybody is welcome.

Skip Grube:

I don't have numbers in front of me, but I suspect the biggest contribution would be in the form of engineer time. 

The Four-Eyes Principle [48:38]

Zane Hamilton:

So Martin asked, "Would Rocky 9 further support the implementation of the four-eyes principle for root access directly at the machine without access to external systems for the user?"

Neil Hanlon:

Not sure exactly what's being asked there. I can say at least from our development practices and infrastructure practices, we do provide or require a four-eyes review of everything that's going on. You can see that actually in the Peridot repository, for the review structure there. We require two approvers that are members of the Rocky Linux team. So yeah, please either clarify or maybe follow up with us on chat.rockylinux.org, or send a message to Zane. We'd be happy to get some more information and answer that question for you.

Zane Hamilton:

Thank you for the question. So before we throw up another question, I’d like to ask, “How does making Peridot open source help the community?” That's up to anybody who wants to answer.

Peridot’s Community Aid Through Open Source [50:07]

Skip Grube:

Well, I don't want to say it’s self-evident, but for sure we mentioned PRs just now. If you got a powerup on Peridot and you, or someone you know, is familiar with how to fix problems in code, (it's written in Go by the way), please, please help us out, because we want your fix. We want all your fixes, because it'll make it better. That's just open source in general.

Gregory Kurtzer:

Yeah, and I'd add to that as well. Us being an open source community, we’re really here for the community, including individuals, enterprises, or organizations who need production-capable code, production-ready infrastructure. That's really where we're focused on this, and if we were to build up this project and the binaries for this project in a way that other people don't have the ability to either replicate or is not transparent, then we're not doing the community any favors. As a matter of fact, that's kind of a massive disservice to the community, because it's not truly an open project then. So, from our perspective, it is a requirement that every time that we release an operating system, we're releasing not only all the dependencies, not only everything you need to actually go and replicate and reimplement that, but also extend what it is that we've created and customize it for your specific needs. And again, if all of those builds were happening in a non-open, non-transparent manner, that would make it very difficult and I think it would be doing a disservice to the community. So we always make sure that what we're doing is reproducible before we release.

Neil Hanlon:

To add on to that as well, it's not just people that know code that we want. We want folks that can help us with UI, and UX, testers, documenters, people to come and yell at us and tell us what we need to fix to make it better for their use cases. And a lot of this will evolve naturally as special interest groups are stood up and people start working on it, with it more. There are some other ancillary projects around Peridot that we're going to be starting to work on very soon. Kind of a Rocky package command that's similar to fedpkg or centpkg or the Copper CLI, if you're familiar with any of those or all of those. We're looking to make an interface for everyone to be able to interact with Peridot through our build system, through a very, very methodical way that people are already familiar with, with the improvements and additions for building things for Rocky and in/on our build system. So there's going to be a lot coming. 

Zane Hamilton:

Here's a statement: “Just imported Rocky 9 onto WSL and it works great. Now I can get Snaps and PPAs out of my life forever. Thank you so much, Rocky Linux team.”

Neil Hanlon:

I'm glad I was able to port into WSL. I know we've had a little bit of progress around getting an image into WSL building it from scratch, but we'll have to take a look at that soon, too.

Skip Grube:

Come to our chat. Tell us how you did it.

Neil Hanlon:

Yeah. Come to the chat.

Skip Grube:

Help us out.

Zane Hamilton:

And then back on the security topic, Alan's pointing out that sometimes there will be, before it's in dnf, there will be a security release that Red Hat has documented and given the ability to put in. It used to be available in CentOS, in some cases, as well. Is that something we're working on getting into the communities?

Red Hat Security Releases [54:05]

Skip Grube:

Yeah, okay. So, I think as I read this, what I'm reading is there are things that you can do if there's a security issue in particular before a release package gets fixed; there are instructions for mitigating or working around it. It depends on the details, and it depends on the specific thing you're talking about. But yeah, it’s worth noting that any, almost by definition, unless it involves subscription manager, any documentation relevant to Red Hat 9 or 8 will also be relevant to Rocky 9 and Rocky 8. So, if there's an approved workaround that Red Hat or someone else has come up with, yeah, it should 100% work on Rocky.

Neil Hanlon:

I think the question there is a bit of asking for us to be duplicating the instructions there, and as Skip pointed out, sometimes that's behind a paywall, so that can become tricky legally. In terms of coming up with our own workarounds, obviously that can also be tricky. So, it's something that I don't think we've looked into, but it couldn't hurt, especially as I'm reading Mustafa's comment. He, I guess, joined in the chat because he couldn't make it. But we also just open sourced, along with Peridots and all the same repository, our product Errata portal code and generation for all of the Errata that comes for Rocky Linux. So that could be maybe a future improvement that we add into the Errata service to have some mitigation steps, if we can pull those. I like the idea. It would certainly be helpful to have a centralized place for people to look.

Skip Grube:

I will say also, just in terms of timing that, because it's kind of related to this question, whenever a package drops new in Red Hat Enterprise Linux, at the same moment, almost exactly the same moment, that package is also published on git.centos.org, which is the upstream source repository. If you ask Red Hat for source code, they say, "Go to git.centos.org. That's our official spot for it." You can actually, because of my incoming tool and some of these others, I've noticed that whenever an update happens in RHEL 9, it's like, I believe there's an internal Red Hat system or something that, precisely when it's published in dnf, it's usually within two minutes, it's also published in Git. And then it gets picked up by Rocky, usually less than 12 hours later. So that's our kind of drag time as far as updates and fixes.

Closing Remarks [56:59]

Zane Hamilton:

Thanks for the question, Alan, with the comments. We appreciate it. So we're running up on time here, guys. I really appreciate you guys coming on to talk about Peridot and congratulations on getting Rocky 9 out. Congratulations to the community and thanks for all that you guys do. We really appreciate it, and I will give you guys the last word. I'll start with Skip.

Skip Grube:

Go try Rocky 9. We've got Raspberry Pi, desktop live discs, boot, minimal and full ISOs. You've got everything. So, there's a whole lot of choice and you pick the one that suits you. It's working well.

Neil Hanlon:

Absolutely. There's Azure images on their way. There are AWS AMIs currently under review to be published. So we'll post updates for all of those as they become available. Check out our new minimal Docker container. It's like a hundred megs or so on disk and like 30 megabytes compressed. So, please test that out. It has just the essentials, and if you find any problems, you can ping me on Mattermost and yell at me.

Gregory Kurtzer:

I was going to also say, don't forget the Docker images; we've got cloud images coming as well. We talked a little bit about WSL earlier. If somebody wants to come and give us a hand on those WSL images, we would love the help. And don't forget about the Google cloud images as well. And we're also going to be working with Google to optimize those cloud images for you. So they're going to come with the best support, best driver support for everything on their platform. So hopefully it'll be very efficient. The last point I would just make is come join us. We have a lot of fun in Mattermost and in our chat. So it's chat.rockylinux.org. It's a lot of fun. It's great to get help from other people as well as help other people, and there is a large community there already, and we'd love to have everybody in the community there. So please join.

Zane Hamilton:

That's great, and we thank you for your time this week. We will see you all next week. The links below are the ones that we talked about today. So, go follow those and follow along with Peridot and everything we're doing there. Like and subscribe, and we're looking forward to seeing you. Thanks guys.