What's New: Rocky Linux 8.9 & 9.3

smooth experience for our end users we know that most of them are in the Enterprise space and so they really count on having that stability so we looked early on at how a lot of other projects objects were doing their testing uh I'm not saying any one of them is wrong for doing it but we looked at some of the shell scripts that people wrote and we thought that's too basic we looked at some of the um Kickstarter scripts that were used and thought that doesn't really cover a lot of the use cases of people installing uh graphically for example um we wanted to see

what uh others were doing and we ended up kind of following Fedora lead and op sua lead and we went with product called open QA and this allows us to do a lot more thorough testing uh when we come to a release week um we have a Playbook inside of our mattermost which um we use quite often and that mattermost uh Playbook has a list of tasks that we accomplish and it's hundreds of tasks long of things that we want to check and verify across each one of the architectures that we support and these tasks are open for anybody to help contribute to we have

things as simple as being able to check that make sure that a ISO boots um all the way up to some more difficult things that you need to be kind of involved with with uh the open QA project um but we're always looking for more people to come and help us with the testing and to get involved um we've got a lot of things that are set up for those that are coming in new uh and wanting to learn how to do things um and to be able to contribute in some way and we always recognize those people and we make sure that their names

are on the release uh as having contributed oh wow that's cool I I I wasn't actually aware of that um so you mentioned the mattermost server and Rocky Linux on the mattermost server and and you guys can just kind of like go Google that if you're not familiar with it is that really the best place for people to that are interested in jumping in and want to um be not only part of the community but um you know know more about what's going on so we try to have a lot of presence on different social media and um we have forums for asking questions um so there's a lot of different ways of getting in contact with us but uh chat.

Roy linux.org is by far the best place to get involved with the community um we have IRC bridges for those that prefer IRC for most everything um but especially on release weeks uh all of that happens inside the Playbook which theide of the matter most okay all right thank you I appreciate that that clarification okay so we also have a another amazing Rocky Linux member here Mr skip all right skip why don't you unmute and uh pop in and introduce yourself hey so my intro is not that cool um I am skip gruby uh I've been with the rocky Linux project since really the beginning

um I work on the uh release engineering team um I'm a package monkey if I have to describe my my role that's what I do um trouble shooting packages building packages dissecting them all kinds of stuff um yeah and I uh um like like uh uh Mr stack here I've been a um Linux user for a long long time since the the 90s um and have floated around uh I got involved in Rocky Linux cuz was a big fan of Centos uh rest in peace and um been a long time uh red hat and red hat compatible administrator as as a job and um yeah

just uh um just happy to be here yeah that's awesome well thank you both for being here we want to talk about the most recent release um that Rocky has put out into the world and that is going to be Rocky 8.9 as well as 9.3 so how about you give us a brief overview of how you think that went I think it went well um there's always things that we can take away and uh do better and we've got a uh a list of some of those things that um I have on my personal to-do list for how we can improve the testing process

next time but overall the testing uh went really well um as I said before we've kind of got a very involved process so it sometimes takes longer for us to go through so many different things and so many different checks um but especially when there's uh so many things going on um with you know holidays in the US um superc Computing being a big deal um for a lot in our community um having two releases during this time can often be uh crazy and chaotic um but uh our team handled it really well we um are are volunteers and so uh at least the testing

team is all volunteer-based and um it when we work on it we work on it when we can and so it's uh slowo at times but um it was really good being able to just not worry about it too much because I had such a good group of people that I knew were working on it uh and making progress did you want to add anything to that skip yeah I'll chime in from the build side um the build side went as well as could be expected and it's been getting better um there's I'll be honest the build side has had some rough releases in the

past where there's been a lot of late nights and a lot of uh um you know oh crap we've got to rebuild all 50 of these because we you know screwed up the order or something and um yeah it's uh um and I love the robust testing process thank you stack because it it catches things like that which is great um uh I want to shout out one of the uh um the really cool things that was not my idea and um our uh uh Lewis our head of release engineering is now um we have a a pre-build almost where we track uh CTO stream

um which is you know runs six to 12 months ahead of Rocky Linux and real um and it's fantastic um it really uh effectively most of the things that were released in 8.9 and 9.3 we know we can build them because we already built them right which is fantastic and it helps really smooth the whole process over um yeah that's uh it was um it's always a ride doing these releases there's a lot that goes into them um uh even I always tell people um oh yeah it's just you know the red Rocky for I hope everyone knows by now is a red hat Enterprise

Li Linux clone like we we use the same source code and such and it you'd think it would be easy if and and it's uh it's surprising how difficult it is to build a Linux drro even if you don't have that many code changes right so um yeah very happy with the result so that actually goes into my next question like how much time in between re's release of 89 and 93 let's start with 89 and then Rocky linux's release of 89 like how much lag time was there uh I'll take that one here I can get you a ballp part number let me uh

me pull up my web browser while we're talking I believe it was about two weeks maybe a week and a half between and stack correct me if I'm wrong but that sounds right in my head from the real 9.3 release date to the rocky 9.3 release date and less for 8.9 um I believe sorry that's my dog yes I would say it about a week and a half um we we had opportunities to push it out sooner but uh timing on that would not have been great for a lot of CIS admins um we didn't want to do a late night Friday drop um or

anything crazy like that so we purposely held out uh for that Monday release um and we wanted to make sure that um we had everything right and in order so it gave us a little extra buffer room knowing like okay we've done all the heavy testing we've done all the checks things are looking great these are all would like to haves but are not required for release and we can work and do these um knowing that we're ready for that so you know there's a little extra time on there um just because we were thinking about the CIS admins um but yes I think it

was about a week and a half yeah that's awesome you guys because I mean just like you were saying it's an incredible amount of work so thank you right I mean it's it's not unthanked work because there's lots of people who are thanking you all the time for the work that you were doing but it is you know technically unpaid work so you know thank you again for your your time and patience with that um so that that lag time so it's about a week is that is that like a comfortable Space Week week and a half is kind of what you can imagine looking

into the future that we can expect on each of the releases and that's about every six months there's another release is that that accurate yeah obviously we want I'm I'm gonna step in for a second stack if you don't mind obviously we we want to release as soon as possible but really there I would say um Quality over speed right there there is a high degree of I don't want to call it paranoia but um the the entire Rocky team really really values um um a coherent working product working drro and um we really do not want to you know uh we're I've seen this

team especially testing and release engineering bend over backwards when something is wrong even a little thing that most people wouldn't notice right but there are there are things that you know I um I can see uh other teams certain companies out there saying oh we'll just fix it you know next week and release it now or whatever but it's um part of that is it's nice to have an open source project there's no um you know there's no payroll there's no like you know Revenue issues or anything like that so it's you know we we'll release it when it's ready when it's when it's complete

having said that we want to close that Gap as much as possible do it as as quickly as possible but um yeah we won't we don't put something out that we know has issues because that's um we we found out even the smallest things we have a lot of people that use this stuff and they'll notice like people will notice so yeah and I'll second that I uh was just looking at Wikipedia and I'm just going to trust their numbers and they've got that there was a delay of 13 days for the 9.3 release and that's probably correct um and so that wasn't just you

know a wasted time um there was a lot that was going on during that time and um as skip said we want to be correct in what we release um over just being fast and that doesn't mean that there aren't others out there that could do just as good faster um but we want to be assured that the product that we're releasing is up to our standards which we have set based on community expectation and so in order for us to get there and do a job that we are confident in um that's what time it takes and as we get better at automating things

uh there is more and more that we're able to do faster but there's also complexity that that also means that we're also testing new things that we're adding new features as we bring in new uh hardware and new um supported areas that adds the complex for how much that we're testing and what we can do so that just takes time um as we have more people helping out maybe that will drop a little bit more so it's not just a few people doing it so again another invite to anybody who wants to help out with this we'd love to have you yes we would definitely

love to have you so why don't you tell us a little bit about some of the features some of the new features because I mean that that's kind of the the the point of having different versions that are released it's not just secur updates and fixes and patches and things like that there's new stuff so what's the new stuff in um 89 and 93 I'll start I suppose okay so um really I I wanna i w to take a little issue with your description there um the even the Enterprise Linux world is extraordinarily conservative with their uh new versions new features so we we we're

talking about 9.3 for example like it's this new thing right and it's really it's really not um the uh uh the the whole point the idea is to stay as compatible as po as humanly possible even across you know six months of development um the uh we have some there are some new things um you like I said check the release notes there you know for example node js20 right if if someone's interested in that node js20 has been introduced the reddest database has been updated to version seven I think at 9.3 which is I think it's the latest major Upstream version and we could

go on and on there's there's several of those you know things there all in all there are in 9.3 this just to give you an idea there are about 3,000 packages in Rocky Linux 9 Source packages is approximately 500 of them were updated in 9.3 and only a handful are going to be notable like I just mentioned redus node.js there's a few more um but really um yeah the idea is is incremental and stability is the um yes it was a lot of packages that were updated at once but even those updates were not um the kernel obviously big deal and um you know there's

lots of lots of new stuff that's being backported all the time by Red Hat engineers and cental stream contributors but um yeah the the idea is that it's it hate to say the whole point is that it stays the same though I don't know if that that may not be the answer you like but uh yeah that's a lot of admins that do like it right exactly I mean that that kind of is I guess that that is the point so yeah thank you for that Clarity I appreciate that stack you got you got some thoughts no that was mostly what I was going to

go with was that uh there was mostly just a lot of updated packages it wasn't like there was huge changes to this uh particular round uh at least not that I saw um there were a few uh improvements that um happened in different packages um and one that I understand why they made that change I don't necessarily like it but I understand why they made it I don't know if I would call it an improvement but but it's uh something um but yeah otherwise I think Skip did a great job answering that one awesome okay let's see here um let's just go down so how

does the upgrade process work for users moving from one older version to 89 or 93 and I think probably the answer is going to differ if you're going from a major ver one major version to the next major version that version that's probably different right than just like doing an upgrade careful with your wording here yeah so going from8 or 8.7 or eight. whatever to 8.9 and going from 9.1 .2 to 9.3 um that's easy right dnf update as long as your repositories you know uh are configured the default way or you know if if you have if your company or if your organization or even your home lab has synchronized repositories right to pull that n make sure that 9.3 release is pulled in and run dnf update and it should work right and there's obviously that doesn't work come you know come down to chat.

Roy linux.org bugs. Roy linux.org let us know about that um should go smoothly 100% of the time um now what you're touching on is uh um now if you're going and I guess I should rewind for those that don't know um Rocky Linux just like red hat Enterprise Linux maintains two complete versions simultaneously and in red red Hat's case it's three at the moment there's a red hat seven still out there getting updates um that are you know 10e release Cycles a new one is released every three years um a new major version eight nine will be 10 uh next year sometime I think um

and those are done you know there's just because you're on eight if you're happy with eight stay there there's no you know at the moment there is no compelling reason to bring you to version nine right unless you have a real want or desire for that um version eight of Rocky Linux just like red hat sported through 2029 uh version 9ine will be supported through 2032 if you're thinking of upgrading you've got some time you got some time to think about it it's what six years away now uh so yeah we've got a you're you're good for now um so many people I think make that misconception where it's like oh gosh we've got to upgrade you know 8.

n to n something right and that's not that's not true they're their own thing they each of one is you can think of them as completely separate products they happen to be called Rocky Linux but eight and nine completely different um upgrading from 8 to nine is not recommended quite frankly um it can be depending on your system uh it can go well or it could be a little dangerous um there's really uh there's a lot of differences between those two like I said two completely different products um one based on much newer uh package set than the other underline so um I know there

are some some companies some projects I'm thinking about leap Elevate I know uh CQ has something with their Ascender automation um but those are uh um you can look into those if you're interested in it but again my advice to any admin doing that is to tread lightly be careful um especially if you have thirdparty software at third party repositories installed be very careful um because it yeah you can burn yourself doing that so don't mean to be Negative Nancy but no no no I mean this this is um I mean I'm I'm glad that you're speaking to this this is also CIQ's stance

and it's nice to hear you know you to more coming from a rocky Linux perspective of you know just the clarity on what we're actually working with here because um it is it can be kind of confusing for people for sure with stack was there um something that you wanted to add to what skip was saying about the different versions yeah I would just say that uh there's a lot of home labers like myself who want to use Rocky because it is stable we're not Enterprise we're never going to be Enterprise it's our home lab um and so knowing why you want to upgrade um

from a major version is a um the the first step um for example if you're using something that relies on a very specific version of PHP and the version of PHP and Rocky 8 is too old you might want to be considering mov moving to Rocky 9 uh however there's the aspect where I'm personally at where one of my primary systems that I've had since 2012 uh still runs really well but I can't run Rocky n on it because of the x86 64 V2 uh requirement and so um there's a little bit more consideration that kind of needs to be thought in on this uh

for a lot of Enterprise they're just going to roll with whatever's latest when their application reports it and they're going to have a process for how to deal with a mass update um that's not always true for home labers and small businesses that are just wanting to have something running so Skip's point of jumping from8 to nine needs to have a little bit more thought and understanding for why you want to make that change and will it be a successful change especially if you're having to replace the current running system with a new OS um and you don't have the luxury of just saying well

I'm going to to buy a new system spin it up with a new OS migrate everything and shut down the old one and so um but we're happy to help um come by chat with us say hey here's what I've got um and here's what I'd like to do and the reasons for it and we're happy to have these conversations and Community to guide you on the best way of doing those updates um so whether that's small business or even if you're a corporate and you're not sure how to do that uh we'd love to have these chats to to help you with that process

um all Community Based and focused so being part of that community means you can ask these questions you know that is actually kind of brings me to another question I mean one of the words that you used uh skip I believe and correct me if I Wrong was maintained so there's a 10year life cycle for each major version of Rocky so it's eight you say so I think I don't even remember what words I use but that sounds like me yeah okay well okay so the the life cycle is 10 years though yes you're correct though you're absolutely correct okay so what does that mean

in terms of like are you what does it mean to still maintain it even though you're not working on it right like oh oh well it's I mean working on it yeah okay are are you like in at Rocky linux.org like I go I go get 87 there's things that have that are going to be um worked on in the last six months no no so hang on back up okay thank you back up let's talk about um so major release is eight for example or eight dot you know or nine you just eight or just nine right minor releases we talk about 8.6 8.7

8.8 8.9 recently same 9.1 9.2 9.3 Rocky linux.org um you know our drro only supports the latest minor release so if I am on version nine for example I will only get you know I if I would like updates from Rocky Linux right which I would right um uh 9.3 is the way to go that's that's what I'm U the older versions 9.1 9.2 are vaulted now they're put they are no longer you know they no longer receive updates and the nice thing is that if I'm already on nine which I am right um it's very easy like I said just you know run your

dnf update you'll get 9.3 um to honestly I I noticed this even back in the old Centos days um it looks just like a larger than normal dnf update right if I I I a lot of users I was one of them back in you know Circa 2005 didn't even realize that I had gone from like Centos 4.5 to 4.6 right didn't know anything about it right I just ran update and it and it it worked and so um that's uh honestly I mean obviously if you're you know if you're a corporation and you run thousands of machines and you're into this sort of thing

you'll you'll be familiar with the release cycle but even if you're not it should be very simple right like you won't even again the it's seamless you won't even notice that you're on 9.3 unless you look for it and that's the way we like it that's that's good stuff yeah yeah yeah okay so I I'm seeing what you mean so every six months there are minor version um updates yes that you and you you name them so it's you know now it's 93 and then in six months it will be 94 not very creative it's just numbers but yep same same deal so that is

what is currently being maintained and so when you do your dnf updates if you are on that version it will it will automatically go because it will pull it from the it should be noted again so now we're talking within a release so let's talk about $9 three now right we already have updates from you know there are updates to software uh again where Rel clone so they come from real real source code um our updates to software are released weekly or sometimes even more than weekly in Rocky Linux right they are but within a release though they're tiny they're tiny updates relatively speaking right

they're you know oh op SSL has a little itty bitty bug fix sometimes that's really sometimes tiny updates are very important you know there's a a giant hole in up a cell needs to be patched right but um but yeah these are you know so obviously running dnf update is a good idea often um anyway get the latest security fixes but within a release generally we you know I I say we Red Hat um the philosophy in red hat and CTO stream is not to break things if at all possible Right like to to change as little as possible and still be secure and still

have the proper bugs fixed and so forth and so um um the only way to make you know and again even the big changes I'm using quotations here big changes from say 9.2.3 are not that big in the grand scheme of things right Rocky 9.3 still based on a 5.14 kernel right still it's a newer you know it's got a lot of new backports and such from it but it's you know still based on op SSL 3 do I think it's 0.7 but whatever the uh whatever the current op SSL is um so we're not like you know Earth shat we're not moving mountains with

these minor releases right even then um and the uh um within a minor release again these weekly you know little itty bitty updates that happen they happen sporadically you know couple this week maybe five to 10 the next week whatever um those are even tinier changes right we're trying stability is the name of the game hopefully that makes sense and we we do this for 10 years basically yeah thank you for that um so what are kind of back to this 89 and 93 that we've just gone through what challenges did the development team face while working on these versions um I'll start by saying

that uh I am I personally have not had that many challenges um in at least in my limited work that I do um I know but I've I hear stories and I talk to all the other Rocky Linux developers and yeah collectively we have yeah we have some things um I want to touch on something that's a little it's almost uh like a tangent really which is something for the first time that we came across and we we were forced to think about and it's a good thing because me the project is really picking up steam we're getting lots of people interested um uh Rocky

Linux has special interest groups sigs um that are you know and they they create all kinds of things um I'm involved in what we call the alternative architecture one we do Raspberry Pi images and and images for other little devices that you know red hat for example in a milli years would not consider porting to and we do um there's you desktop sigs that do like live images with you know all kinds of desktop environments and so forth and it's really cool and one of the thing the challenges that came up is um it's not really it's it's a bit of a technical challenge but

really the the key is uh your Sig which is building packages right um when does it cut from 9.2 to 9.3 right that was a it's a it's a question that we haven't had to face yet and it was really I don't want to say it it wasn't neat at the time because it's actually an important question and it was you know we don't want to have people confused but it's something that that you know because there's so many people working on these extras and so forth that we actually have kind of a a schedule to keep now where you know we have to say

Okay well you know the rocky 9.3 release candid is there we got to cut the sigs over or you know perhaps we do it even a little earlier and have them because they want to be able to build their things against 9.3 even as it's in progress right because there's you know there's enough of activity now going on that it's it's actually relevant that that in between time between releases right that you know week and a half or 13 days whatever we talked about um there's kind of a you know our Sig teams we've we've gotten some interesting ideas on solving that in the back

end but um yeah it was it was a challenge but it was kind of cool to it was a good challenge to have because it means we have a lot of neat activity and a lot of people who are concerned about it Zach you want to chime in here no that was uh pretty much kind of where I was first going was we had some issues with uh uh changes in the tool sets for how we build some of our images and um these are aspects that uh aren't necessarily Red Hat um things that they care about these are things that are more in Fedor

land and so by time they made it to us um they work great for Fedora not so much for us we had to make a few changes and um I I'm not sure I not sure that KDE has actually been released yet because we ran into issues with it um so there's there's aspects for some of these additional things that we provide for the community um that kind of caused issues this time around so that brings up a question do you mean that there's extra packages in Rocky Linux than there are in real no the core of Rocky is going to be what matches with

real the sigs these um extras are where there's going to be contributions for the community and so for this example with KDE uh we rely on Apple to build a KDE desktop image that people who want that version can have it without going through a bunch of hopes on their own and so that's an additional that's provided but those that are looking for um just the core Rocky that's going to be the same as Upstream okay my good that's that's and also congratulations you guys for the success of your um Sig groups that's cool all right you guys so as we are coming up to

um the close of this uh segment here again everyone thank you for listening I really want to ask just one last question because it's relevant to you the listener and of course you've got these two amazing men that are on the rocky Linux team so how can users contribute to the development of future Rocky versions so I'll start off by saying that if you go to Rocky linux.org and you drop into the about section and you look at the about Rocky the uh Community Charter um and you really look at what our mission is we are doing this for the community as such we want

to engage the community and we want you to be participating in the community as a whole this isn't something where it's run and dictated by corporations there are other groups that have corporate control we don't you don't get to have a corporation and buy a seat on our project board you don't get to buy votes because you're a corporation who contributes a whole lot of money the way that Rocky runs is that it's Rocky members and how do you become a rocky member we have steps for how you do that and part of that is being a part of the community that doesn't necessarily mean

that you have to be up you know till midnight or 1: am working on things you know during a release week you could just simply lur participate in the the games that are played um actively have conversations with people um let us know what works for you uh whether things are not working you don't have to be considered a hardcore developer or anything like that to participate within the rocky community and then become a member which would give you voting rights to determine the direction of Rocky we want to Foster and encourage that we want people to have that experience of having a community where

we're uplifting not just Rocky but all of Enterprise Linux and contributing back Upstream to projects whether that be something like centas Fedora or even individual projects we do a lot of that kind of work and we want to continue to Foster that community so what I'm saying with that is that anybody can join in and anybody's welcome to participate and we would love to have you when you do you'll find that there are a lot of tasks especially during release week where it is simple for you to come in and we have a whole section dedicated for these are tasks anybody can complete and you

can look at it and say oh I have this Hardware so this random thing that needs to be tested I can do that or oh I have a spare thumb drive I can throw an ISO onto it and verify it boots like there are very simple tasks that anybody can participate with and help out in the testing process similarly in our meetings we do everything out in the uh open um very very very little is still held and isn't out there and that's mostly things like uh the passwords that we use for infrastructure we publish all of our code out there we publish our dots

and our meetings are open so you find that oh hey you know Skip was just mentioning the sigal arch and single board computer I want to be a part of that well we've got some fantastic work going on within the single board computer community in Seal Arch and we're hoping to have some really exciting news very soon about that um so you don't have to just be a part of testing or any one group anything that catches your attention you can just come and just participate see what's going on in the conversations Join one of the meetings listen in um you can say Hey you

know I'm interested because I have this device and I'm curious about uh who else has it and what is being done on it um there's a lot of easy ways to just participate in the community and then when you're ready we would love to have you say you know I want to be a member and I want to contribute more I want to be more involved and from there who knows where you want to go like we're happy to Foster and encourage that any way we can so when you kind of ask about how we can contribute you can start by just being a part

of the community right now uh I don't know when this will air but we're doing the 12 days of rockas which is 12 days leading up to Rocky's birthday there are games being played every day you can simply just participate in those games and still be in a way contributing to the rocky community and we'd love to have you yeah all of those details about how to get involved in the community really beautiful skip what would you like to add to that as as a member of the community as well so thank you I'm G to Echo that um well said stack very well said

and also as a practical matter I think a lot of people out there are not used to contributing to open source projects in this way they're used to a company model where you get assigned work you know you say okay you're gonna you know there's a hierarchy and you're G to go do this because you work for this manager and that's not how this all works at all um I would say um sometimes you have to get creative but there are enormous ways to contribute to the project if you're interested and they range from things like hardcore technical work um including packaging scripting developing that

sort of thing um especially if you have a deep understanding of RPM or you know dnf and uh various programming languages please yeah if you're interested in helping talk to us we're interested in talking to you but you don't need that though in order to contribute you can you can uh stack said it's very easy to get on the testing team just you know look at the tasks they do ask around and you know hey it's it's some of them tasks are quite simple you can try them out yourself um uh there's other things that you can contribute documentation um artwork even and this kind

of thing again don't wait for the work to come to you I want to encourage that right so if you're interested in artwork for Rocky Linux you can't we have like I think there's two people that do Mo most of the graphic design and website work for the rocky Linux project they're very busy they're not going to have time to manage you as well right if you think you want to create if you want to create some rocky Linux artwork do it right who cares do you know and show it off send links to it say hey I created this for Rocky Linux and you

know here's the here's the SVG source to it you can use it if you want here this is great I want to be on the team you know like like let's go out and do things you know we we do it because it's fun we don't do it because we're assigned work by a manager right that's it's not why we're here um and uh even things that people wouldn't think of as helping really are helping like hanging out on the chat channels or forums. Roy linux.org or especially bugs.

Rocky linux.org and if you're interested helping there whenever someone comes with a problem or a bug or an issue you can investigate it on your system right it's uh again there's a lot of people who have a bit of Linux knowledge that can help out with this that are maybe not you know hypers specialists in packaging or you know they're not hardcore coders but they you know somebody says they have a problem with a dis Drive well they can totally they know how to fix that right and that's this is fantastic work this is helping too right so be creative and and go out and

do it right like there's don't wait for things to come to you because that's not how open source works it won't it won't work that way awesome well I'm inspired uh I don't know what help I'm G to be but I I can certainly wave a flag and and say I'm glad to be part of the community and play in the games so thank you guys so much thank you for your time and all the great work that you guys have been doing make sure to go to Rocky linux.org there's an incredible amount of resources Rocky 89 and 93 are out so go play with

them and let us know how it's going thank you so much have a great day byebye