Dave Rush LIVE Q&A With Greg Kurtzer and Skip Grube

confused with another guy in in my class called Kurt and so we were both just Curt so you just totally gave me flashbacks of of the the early 90s late 80s oh my goodness I apologize I've been typing your name and documents all over the place for the last three weeks and it's the the cursor part that comes through though all right glad to bring back a memory and skip welcome skip so the titles that I had for you basically kind of led me to to show coding I know that you're uh you're back all over the place it's administrating Linux systems and you moved

on from there and and became code or extraordinaire and package manager but I don't even know what some of those terms mean so fill us in oh my gosh yeah so I um the way I got into this whole thing we'll we'll tell the story later it's quite a story but uh suffice to say I was uh very very early on hopped on the rocky Linux project got very excited about it and um yeah just started contributing to it more and more and more a lot of late nights a lot of a lot of packaging work troubleshooting work I don't really do much of the

infrastructure I'm kind of out of that game right at the moment anyway but uh yeah there's coding involved there's all kinds of things involved and um yeah eventually uh um just recently actually in the last uh six months Scott picked up by ciq I'm now a ciq employee so I get to do it in part um and get paid for it so it's uh it's very nice very good for me well the contributions that you've been doing with with the massive amount of of project that I've looked up on you uh you pretty much did that on your own time as a voluntary thing yes

yeah I um I mean I've been in Linux been in the open source world for a long time not as long as Greg Greg's definitely got me beat but uh yeah I've been uh it's just because yeah and I'll I'll become old and crotchety eventually yeah for sure but um yeah definitely uh just um um I'm already familiar obviously with the red hat world and you know the RPM world and yeah it's um hopped onto the project and have been I've learned an awful lot and it's part part of why I'm working on it quite frankly is that uh I would say my my day

job just wasn't doing it for me and uh I really um some of the folks we work with like technically wise would blow blow you away like I put them up against anyone in the world quite frankly for for some of this stuff so it's wonderful all right so you've you've opened up the door to the the background that not everybody is familiar with myself included so you said you're familiar with the red hat world and Rocky is certainly tied to red hat and your precursor Centos and Chaos prior to that uh and there are terms like Upstream development and downstream development here we go

give us a primer on what red hat and what upstream and downstream means and and where Rocky fits into all that that's an easy one right you want to take this one Greg or would you like me to hop in you know what uh you keep going I I like okay all right let's see so the story I'm gonna start the story with Rocky first and then we'll talk about upstream and downstream and all that stuff so um a lot of folks still not familiar with this but um I know a lot of I I hope a lot of the audience has heard of Centos

Centos Linux you're familiar with it and um looking at a guy here on our chat feed will Shaw one of the biggest proponents of Cintas you'll ever run across and it's here I know he wants to dig into the world with you absolutely um I I also a big fan of Cintas Linux I love well love it loved it okay I guess if you want to uh you want to call it that so in it all started in December of 2020.

um and there was an announcement and I followed Tech news I was you know just a a member of the general public I would say that's you know when there was this big Tech announcement that came up that said oh yeah by the way Centos 8 which is you know released in 2019-ish and was you know slated to be supported with updates until 2029 right same as red hat because they follow the red hat release right have for decades now and um they said oh this release basically from the the Centos blog says no we're gonna end in 2021 actually like the the end of the year December 31st 2021 after that no more synthos 8 updates um I see Will Will's uh yes synthol7 updated till 2024.

I've got to look at update or upgrading soon but uh yeah which is it's bizarre right like Synthol 7 continues to 2024 Sentosa nope not nothing doing right and so it was like I was like oh that's not great because I love Centos and what you know there's uh and Synthol stream which we'll get into in a second acts as like almost like uh what red hat now comes from but we'll talk about that in a minute that's going to be the only Centos so again um so yeah I looked at this blog thing yeah oh there you okay there you go yep and actually

it's still there and so if you search for gmk on the uh okay there you go yeah so I I was looking at the comments because it's something I do sometimes you know God help me and uh but I saw this I saw this thing it says I am considering creating another rebuild of Rel and may be able to attire some people for the effort and you can come join us in our slack Link in hbcng and I was looking at it and it was like huh huh that might be cool and so I went over to the website and I I know that I

I hopped in probably about six or seven hours after that blog comment was made right and I'd I'd never even joined a slack before I didn't know you know I knew what it was but I hopped into this this slack channel that he had posted there and there were 5 000 people in the slack Channel and they were all talking at once it was it was chaos like like it was just you know everyone was like what are we gonna you know synthos is going what are we gonna do what are we gonna do and there was this it was almost like a crowd stream

of thought it was going a mile a minute down the screen you should have seen it it was first thing we did was uh organize different channels for different things because with one channel it was like you you'd you'd step away for like 30 seconds and you come back and the conversation you were following it was you know because it was yeah okay it's like it's like it's I if you heard Twitch Plays Pokemon it was twitch designs and operating system as well it was nuts and so yeah we um from there it was very organic honestly we organized you know kind of we quickly

settled that you know hey the entire the hive mind I'm going to call it said hey we got to replace Centos because Centos you know would people want this and people rely on it and we like it so let's do it and so and we we divvied ourselves up into different channels people were like well I want to do testing I'll create the website for it you know I'll uh um the reason I ended up where I was actually is because I quickly you know caught on to this idea and was excited about it and I noticed that in the different channels there were actually

there's a lot of people that wanted to do testing a lot of people that wanted to design the logo and to troubleshoot and to but there were not as many people in the develop and package channel right and I was like well I know a little bit about that so I'll join that because you know and there was you know there was a lot of people that wanted to do infrastructure you know like website you know web servers and stuff for it because we have a lot of admins and so forth in the group um but yeah I was I was like man there's not

that many people that actually want to do the engineering work because it's quite frankly it it sounds easy right oh you just take the red hat sources and rebuild them but I'm here to tell you it's uh it's a lot more it's more complicated than it sounds right there's a lot that goes into it and so yeah I said oh that's how I set off on this wild Journey right it was uh um it's pretty crazy would you say package management from what I've come to learn um that sort of exists in in two Realms one is the core packages the the core OS packages

the development chains and things like that and then there's the ancillary packages like uh porting or changing or modifying Apache to so that it works on Rocky or whatever flavor and then it can go in your repositories do I have that roughly right kind of sort of so we have and again I'll I'm gonna have Greg chime in here soon too I can I can tell he has stuff that he wants to talk about as well um really there's uh the core mission of Rocky Linux like Centos before us is to reek Faithfully recreate the binaries for Red Hat Enterprise Linux version for version pound

for pound as closely and most compatible as we can possibly get it right and that way you know the our entire goal is that every piece of software if you compile it for Red Hat Enterprise Linux you know 8.7 or whatever it should also work on Rocky Linux 8.7 without modification no matter what and if something if there's a difference between them we consider that a bug right even if it's like a you know even if the Rocky version is fixed right like if it's somehow better right we don't we don't actually want that we actually want to be a 100 compatible all the way

through but there are um and even that is you know difficult quite frankly because there's a you know more than 3 000 packages and you have to build them all and you have to build them all and you know some of them in very particular ways in a very particular order and so there's there's all kinds of messiness that gets introduced but the um the other side of it is what we call special interest groups these are folks um and I know you are very interested in the Raspberry Pi and I'm sure we'll get to that absolutely these are groups that come together and say

hey we want to enhance this in some way right to you know add extra stuff to it extra functionality like red hat Enterprise Linux you know and core Rocky Linux will not boot on the Raspberry Pi for example right you can't just slap it on an image it won't work right and so we have a group around you know making that happen right and like and it does the the packages and the things that we produce don't get put into the core you know like default Rocky Linux which is you know matches Red Hat exactly but we have other repositories and so forth that we

can add our extras into right there's a whole lot of this stuff there's we have a desktop crew that likes to do um KDE or mate images which are you know different desktop environments and you don't get that if you if you just install Stock Red Hat Enterprise Linux you will not get KDE it does doesn't exist for it and so it's um and they have like they do things like throw you know live live spins right where you can you know we have a KDE live version where you can just grab the iso and boots right to the desktop Red Hat Enterprise Linux does

not have that and so that won't that's not in core Rocky line Linux but it's part of a special interest group sigs we call them I gotta pull back for just a second here and point out uh what guys what I mostly there's mostly guys here but I did see TD Washington here and I'm not sure who all's what we got a lot of Old-Timers in here mostly everybody's in in raptured because we usually have a lot more posts going on they are all just sitting on the edge of their seats but uh I did a presentation one time sometime last year about all the

various desktops and uh tried to put some effort into convincing everybody that it is in fact mate and not make oh thank you very much it is mate I still say mate in my mind a lot but it's it comes from a t in Argentina I think because that's where the the fellows from but yeah it's it's weird yeah all right well let me dive into some of my questions here and we'll hopefully find some stuff that Greg wants to Pile in on this is a actually can I jump in can I jump in real quick so so first thing I gotta ask Dave what

are you drinking that looks awesome sorry this is there's a trademark here I wear a different Hawaiian shirt and I drink the same iced tea every show in the same cup so it's just iced tea with lemon and some kind of keto friendly sweetener but it's like I had some we we've had some uh Rocky parties in Zoom where you know we we encourage people to bring their drinks we don't record the conversations and and whatnot um just in case because people are drinking but yeah we get some pretty cool drinks and and people even doing shot games Rocky after Dark we'll do today uh

is join the mailing list and uh see what level of participation I could do one of the things that I learned about you Greg is with your your long and glorious career of starting all kinds of organizations I go back to chaos uh were you involved in any way with white box so I know if I I remember white box quite a bit um when so we were building chaos Linux we were focused on a new distribution of Linux that was RPM based and we even we even knew right from the beginning we wanted to use a well we architected the operating system a little

bit differently not to go on a tangent but real quick because this is interesting we architected the operating system a little bit differently and what we said was the core of it we split the core which we said had to be a completely stable ABI completely stable base but all of the packages around this core could could uh change versions they can update um and uh and so that was kind of how we looked at it so when we built this core this core had to be completely self-hosting completely reproducible completely rebuildable and uh it didn't have everything needed to boot it was didn't have

a kernel it was the user space so then every other package that we built from there was built inside of this core and every dependency enchant in the chain had to be specific for each package coming into it but that core was based on red hat way back then and this is before Red Hat Linux was end of life so this was this was before Red Hat Enterprise Linux was really has really took off now a little while Red Hat ran Red Hat Linux which was freely available and then read at Enterprise Linux they started for some reason with version two I don't know why

but right afterwards they came out with version three again season one I'll I'll talk about oh beyond that in a moment I want to hear that and um uh so we started we were building chaos Linux which again was kind of based on on Red Hat Linux at the time the first version was Red Hat Linux and um actually I may even be off on that this was a very long time ago this is 22 2002 2003-ish so um uh but that's what we were building on now then Red Hat end of life Red Hat Linux and basically said if you want to continue using

Red Hat you have to go use red hat Enterprise Linux and uh that affected obviously what we were doing and it affected you know a number of people um so we were trying we were still kind of talking about this and trying to figure out what's the best bet for us when all of a sudden there was a slashdot post of um white box Enterprise Linux kind of coming into into the world so the first rebuild of red hat that I know of was in fact white box Enterprise Linux okay and I'm for some reason I'm forgetting the person's name who was running it but

he came out of a he worked at a library a Parish Library and I don't remember exactly where it was now um but oh very cool yeah so he came out of a library and hosted this out of the library using the library's resources and um he it immediately it took off to such magnitude that um you know slash dot hit it and back in the day we used to talk about things getting slash dotted um and white box got slash dotted so badly that nobody could nobody could even see what a project was um because there was so all the servers were down because

they couldn't keep up and so um you know we talked about it and we actually I reached out to the maintainer and I said does it make sense for us to work together you know this is this is an interesting project here's what we're doing and whatnot I didn't get a response from him for for quite some time probably due to just the fact of how much he was dealing with uh and then after a little while he basically said you know I'm I'm doing this for us and for the library I didn't anticipate this being a giant massive thing that uh where I'm maintaining

this for the whole world so if you want to do something you know here you know you got my you got my thumbs up so uh we we decided Well we already have build infrastructure we already have developers and Engineers you know let's go and start building this so the first person who started working on this was a gentleman named Rocky mcdaw and Rocky um he worked uh he was a deep into high performance Computing and high performance uh clusters HPC and uh and he was already doing I I know how close you and Rocky were and there's a lot of press about that and

I am sorry for your loss and for for linux's loss he was a huge fan of Open Source um everybody that knew him knew he just he loved the community and um in person he was a shy guy um but he was brilliant loved open source and um and so he he basically said look I've been doing this already for the community excuse me for my company for the community for some time um and we had a few other people that were also involved in the project and were doing that previously for their companies um and uh and and he basically said so I I

can start this if you want and we can start rebuilding this and we can leverage then those packages for chaos Linux as well those core packages and and uh so we were like yeah that's about that's fantastic didn't even have a name for the project so we just called it chaos El for Enterprise Linux okay and uh he started working on version three and we he was going about it and he kind of you know heads down just chugging away at this he gets the point where he's about 99 done and I I refresh myself on some of these old mailing list posts from the

from the archive just so I can remember what happened in all of this and um and yeah he was about 99 done we announced the name change to Centos right and um and so chaos El became Centos and uh um you know I was leading the project and uh we we continued it up we continued it but at some point we get this weird message in IRC somebody joins IRC didn't recognize the name and um uh she self volunteered self-identified herself as Rocky's girlfriend and to tell us that Rocky is has passed away and um it was it was incredibly as you can probably imagine

awkward and weird um aspect uh so especially considering it was just about done and um so we had we had some other people kind of take over a guy named Lance Davis took over for um version three John new begin was doing version two at the time uh which started after version three and we were able to get everything released and get it out there but Rocky never got to see what was to become of his initial work and where Centos would go and um to a tragedy knowing him that would have been that would have been just amazing for him and he would have

loved that uh so it's funny on the early days of Rocky Linux before we had a name somebody asked me they said what happened with sentos like how did it start and I was telling the story as we were talking and all of a sudden it just hit me it was like that's it that's the name of the project and um and it's it it's such it's meant as a tribute um in honor of what he did but also the fact that he never got to see it and uh that always broke my heart so I want him to uh yeah so I wanted to

dedicate it to him and every single person that heard the story there seconded it and was like yes that's it and then every person since then always said you know can we rename the project oh and we don't like the name because they didn't hear the back story of it they didn't know why were we calling it Rocky and so we had to for the first six months to a year skip how many times did we hear the same thing sometimes it's amazing okay very occasionally been like well I like your Linux distro but the main is really weird you guys ever think about changing

that no that's so awesome uh you guys wrote about that or answered it in an interview and it was addressed right off the bat I would think that that's as common now people wander in they don't you know not everyone's the most informed and that's okay but um you know Skip for a while we even had it on the front page of rockylinux.org it was like the the even the message that I sent was like yeah and that was uh it was funny because I remember this moment and and what I was talking about in the slack channel that we had joined and this was

like a couple days in and there was still chaos because it was you know a million people all talking at once and yeah that was I think we had a whole Channel dedicated to what do we want to name the project or something or maybe it was just general but there were so many suggestions I had a suggestion that I I can only remember what it was but I'm pretty sure it was stupid and and finally Greg's like well you know I had I knew this guy who was original you know basically bootstrapped original Centos and he passed away and everybody was like like it

was it was funny because you get 5 000 voices and it's very rare that you can get them to all agree on so but literally everyone was like that's the name like so it was very it was a moment there and we were just like you know that's going to be the name so not to detract from that but let me go backward for a second uh because I never did read anything about where's Centos got its name oh um so that name was coined by I mentioned Lance Davis he came up with the name and um uh yeah he he proposed it to me

and uh we went back and forth on this for a little bit uh because I said this is funny story I said you know I really like the name Santos I said but I really hate separating the OS from scent so I don't like sent OS okay that's sentos Centos doesn't matter but the reason why this is at this point in my life I was incredibly pedantic and annoying and whatnot and a scent is cheap but it's infinitely more expensive than free which is what it was intended to be right if you separate out the OS and you have now OS and you have sent

and it means something and I said that's not what it is it is community Enterprise operating system and just say the whole thing through and um however it turned out to be I don't know we went through a bunch of logos but the logo that was prominent was sent with OS capital and from that point on name was name was a mixmash of Centos Centos and sent OS and uh for for a little while there uh I had some grumpiness around that but uh and then I got then I got more mature and grew up a little bit well okay so Centos is an upstream

distro Rocky is a downstream can you tell us what that means so um I'll take the first bit and then I'm going to hand it over to skip uh to fix everything I said that's wrong um so Centos started off in the early days um and and this might be important later in our discussion so I'll mention this uh Centos was always a very small team and it was a small team because of the security implications of what it was we were doing and how to manage that um now we call it the supply chain security of each one of those packages but back then

it was just how do we manage the private keys right we can't give everybody to private keys and people were building it literally on their own systems their own workstations their own desktops and whatnot so there was we had to kind of manage the the control structure of security and the way to do that the easiest way of doing that is just to just to keep your your circle of trust very small and so Centos in the early days you know it was a very small number of people and that persisted for a very long period of time and um as I completely went down

that track I forgot your question Dave remind me your question again oh oh yeah where is it where is it okay yeah yeah I'm good um so I I need more coffee um so uh when we first started doing this we were literally logged into and we purchased Red Hat Network subscriptions and we were downloading the software um you know piece by piece from the red hat Network now a little bit later we were able to also download it from an FTP server that we were able to to get access to but initially it came right out of red hat Enterprise Linux like there was

nothing else um at that point Fedora didn't even exist this was just Red Hat Enterprise Linux red hat and engineers and developers created those packages did all the patching everything put it all together and then released the source RPMs either via FTP or the red hat Network so we would log into that and download the these these bits of source code the patches The Source RPMs the specs everything that we needed to go and start the build process to Skip's point it's never as easy as just going you know RPM rebuild on the source RPM if you want if you're going for um output binary

output that is mirroring another binary you have to mirror that entire build infrastructure uh which is is difficult and there's a lot of and Skip will I'm sure talk about this here in just a little bit more uh so Centos was always Downstream it was always kind of red hat Enterprise Linux was in front Centos was below later Fedora came along and Fedora came out highly in front very far in front and then sentos a sued me and then red hat and then Centos now skip correct me if I'm wrong on any of these dates um or approximate dates relative um but it about um

it when Centos 8 was released there was also something released called Cento stream which gave the ability to kind of see into newer versions of packages and so you can install Centos and you can activate stream as a repository now what the announcement was that we just showed the blog post was Centos Linux is going away but Centos stream is continuing and kind of taking over and it's going to be in front of red hat Enterprise Linux or more pedantically between Fedora and Rel but much closer to Rel and the goal of it as it was initially advertised and and somebody's going to comment on

this and I know it because it happens all the time but it was originally kind of message to the world as a rolling release and it's always perpetually updated yes and potentially updated and you can just go grab a snapshot but they are going to do snapshots for installers and whatnot but it is a rolling release I've also heard people at Red Hat call it a beta I've heard people at Red Hat call it um you know uh the where they're going with it kind of like preview release so there's been a lot of names for it and there's been a lot of confusion in

the messaging for it but all in all what it is is it is a community kind of um look I'm going to use this word also Loosely look ahead of what's coming and it gives us the ability now to sink into stream and to see what's coming into red and Enterprise Linux from stream and the same tags and again skip I'm expecting to correct me the same tags uh that Rel uses are represented in the Stream git repos so we can pull from the same exact tags that red hat Enterprise Linux is pulling from and then use that as our mirror source so it went

from behind Rel to now in front of Rel kind of yeah that that's the gist of it really um I think that it's important to distinguish because a lot of people get confused um and there's again the naming uh the naming is weird I have no right to talk about naming though I've named things terrible in the past um Centos Centos stream they are they're similar but they are different things right they are not the same um Synthol stream as Greg said is kind of a red hat Enterprise preview it's uh stuff and sent you know the versions in Cintas stream will be in Rel

in about a year's time maybe you know six to we're talking six to 18 months somewhere in there depending on the time of year and it's um he's quite right when he says that it's it's a rolling release within major versions so it sent off stream is excuse me I'm talking about Cintas stream now where so Cintas stream eight there is no Centos stream 8.1 8.2 8.3 Etc the package is just roll in we you know updates happen you know kind of organically almost but Centos 8 stream does not roll into Centos 9 stream right and that's where the confusion comes in because when people

think rolling distro they think oh it's like Arch Linux right Arch Linux is um you know correct B if I'm wrong but I'm you know 99 sure Arch Linux is a true rolling release distro they have you know there's Arch Linux and you know there's no Arch Linux 10 or Arch Linux 11 or whatever it's just packages and they continuously get updated you know according to the the dev team schedule and um but Centaur stream not like that Centaur stream 8 is rolling release within the bounds of eight and Centos stream 9 is rolling release within the bounds of nine but they do not cross

over each other right like you can't go from you know Synthol streamate does not roll into nine it doesn't work like that so that's where it's kind of you know I would say rolling release within a major version It's kind of half rolling release if you can think about it that way whereas Red Hat Enterprise Linux and Rocky Linux the rebuild of it is uh very segmented we have rocky Linux 9.0 9.1 there will be a 9.2 a 9.3 Etc and um they they still roll right in the in the case of where you if I if I'm sitting at my console and I do

a you know a dnf update which is our update command and I'm on 9.0 our latest right now is 9.1 I will get 9.1 automatically right I'm gonna do a dnf dist intro or a dnf yeah dist it's just a DNS the dnf update or a dnf update there's no yeah there's no special thing and there are updates even within you know 9.1 for example right so 9.1 in um say November is going to look a little different from 9.1 in December although there's not major stuff does not change within a what we call a minor release right nine would be the major release that

dot one is the minor release and so um Red Hat uh who is our our Upstream provider introduces new features and new things in minor releases so 9.2 will have some new stuff that you know may not be quite as compatible with all the stuff in 9.1 but you can still update to it no problem so hopefully that makes sense there's a lot of things yeah I appreciate that guys yeah that's but that's how um so I think the important thing to take away from this and part of the reason for Rocky linux's existence is that lots of people and I think um Red Hat

perhaps underestimated how many people think that Centos is awesome right they love it and when it was announced that oh yeah we're gonna discontinue Centos and just just go with Cintas stream it's very similar for sure but it's not the same and a lot of people I think really desired the same right and they were like well and plus there was the whole thing in the industry yes yes and and there was there was the whole thing where like like well we have you know we kind of cut off eight or nine years of updates there with the discontinuation in 2021 that was a little

odd it was like well maybe you know we started in 2019 maybe we should have even started it if it was you know we're just going to cut it off in 2021 but uh but anyway but yeah basically lots of people came together and was like you know we should we should do something and and we did so with uh Rocky Linux we try to be the the spiritual successor anyway all right so I'm gonna dive right into the the core of what we do here and maybe get it out of the way but so I was casting about one day doing some kind of

research or other and it ran across a blurb that says Rocky available for Raz pi and so all projects get put aside I have a regular job and uh much consternation to my boss as I Disappeared down the well for a couple weeks and didn't do anything really productive for my company but it just had to be played with oh that's fine uh Rocky eight on rasp I do you know regular dnf'd updates on it but I was stunned I was shocked you're not the first major distro to get let's pursue I'm going to use the word ported I'm not a coder or a developer

sadly but we'll call it ported for the pie but you are certainly to my knowledge the first I'm gonna get in trouble here you guys have you're changing my pronunciations of things with Rel I have called it our hell all my life and I'm gonna have a tough time changing that so you are to my knowledge the first art health uh Downstream or Upstream distro that went on the pie and it just gave me a whole new world to play with in large part for Linux plus courses and I'll pick courses that I teach hey you don't have to put this in a VM anymore

you spent your 35 bucks for your res Pi now you can have live so tell us a little bit about that if you could yeah sure I've honestly I think I've heard our hell before but I I don't know that sounds like ominous you know like like oh I gotta go fix our hell again okay I got you okay so I I come from a different tradition I guess but um yeah so and honestly that that post you saw about the Raspberry Pi Port it might have been mine actually um uh this came from a uh um really uh almost a realization and a desire

to you know kind of scratch my own itch quite frankly with the as far as the Raspberry Pi goes I also like the Raspberry Pi a lot um I've got a couple here some well one's downstairs I know and I've um I like I like self-hosting a bit I don't go crazy with it but uh yeah it acts as kind of my Home Server I've got like a media server on it and it holds my files and it's I even run a classic Doom I love you know Doom game server the presentation on how to install and play Doom on R5 so good stuff yeah

I run a zandrono multiplayer Doom server on it and it's um and I was thinking so I had Ubuntu server on this thing before I joined you know the rocky Linux project and it was always my ambition and this is actually part of why I joined quite frankly because it was always my ambition to upgrade and stick Centos on it and be like and part of it is laziness because I'm like oh well Centos 8 if I can upgrade already did that on my Raspberry Pi then I'll be I'll be set for you know 10 years right I don't have to go and like Swap

all my stuff out I'll just dnf update my way into you know and it'll just run itself and um and so yeah when the Centos discontinuation announcement kind of happened I was like oh no my dreams like what happened and um so after getting into Rocky Linux proper because I do a lot of the what we call the core work which is the you know bootstrapping new versions troubleshooting packages that kind of thing definitely not I consider myself a junior member of the dev team quite frankly but we'll get into that later but um yeah I noticed that uh there is a guy and I

wish he's part of the Centos community and I wish he was here but he can't be here right now and his name is Pablo Pablo Greco and he's amazing guy and what he does is um one of the many things he's he is an an arm Enthusiast arm processor Enthusiast and one of the a big shout out to him by the way he's a wonderful Community member and one of the things he does is he has he maintains a kernel port to the Raspberry Pi because for several for several technical reasons you can't take a stock Red Hat Enterprise Linux kernel compile it for the

arm processor and boot the pi with it it doesn't work without major modifications and it has to do with firmware there's there's drivers that are missing with a lot of work you can get it to come up and we're still looking into this but it will be missing like Wi-Fi for example you won't have that or you know the you know Graphics drivers and that kind of thing and so um yeah Pablo uh he really does in my opinion the the heavy lifting the core work because I am I'm no kernel developer right I do packaging work and I'm but I do not consider myself

I compile kernels sure and I fiddled with their settings sometimes but I do not have you know in my opinion nearly the Deep knowledge to do actual kernel development and but Pablo does he's awesome and so um I I asked him a little bit about this and was like oh my gosh you have this kernel that you work that works with you know Red Hat Enterprise Linux and Rocky Linux and it just kind of works with the pie right and we can develop a package for it and if we combine this kernel and the rest of the packages we already compile for arm 64 like out of the box right we already have a rocky Linux you know Arch we call it Arch 64.

it's armed 64 version of everything right we do that by by default and I was like so if we take your kernel and the arm 64 packages and put them together we can make an image and just you know and so really what I've the only contribution that I really make is I I create the images to make them easy for people to use because Pablo is brilliant and it was possible to take his stuff on his uh he has a web folder where he he throws a lot of his work in and it's a gold mine there it's amazing and I love him I

love him for it and um but it's not the you have to figure some things out yourself there's not really a an easy you know I would say um definitely doable for sure but it's for a lot of folks that you know you just want an image file that you can write to your Raspberry Pi at its official more or less and it's ready to go and so I was like well I can and I did this you know almost like a selfish desire to I was like I want you know Rocky Linux 8 on my Raspberry Pi so that I don't have to you

know switch over and like I don't have to upgrade Ubuntu server in like three years or whatever because I don't want to like it so so yeah and that's I got into it and um I got a little further into it than maybe I expected to at first but I was like really what I do is I take the work of others and I make it presentable right which is you know I want to be like I did the all this effort you know it's not really in my opinion it's not that much effort it's more of a last if you've heard of the last

10 problem where all the work's done but to make it you know not perfect but to make it usable for most people there's an extra like um polishing uh going on there so skip that's that's kind of the role of the distribution um I mean all of the sources that we're leveraging I mean the the amount of new code or even edited code that a Linux distribution would Implement um in in terms of the overall code is incredibly small so what we're doing is we're leveraging the entire Community I mean not just like the Linux kernel and you start going through every single package and

all of the people that have been part of each one of those packages both the development contributions bug reports there there is so I mean the the picture is so big the Linux distribution to Skip's point is really we're just there assembling that final mile so it could be consumable easily and leveraged easily and and this is one of the reasons why honestly I truly believe that Linux distribution should always be freely available it is because all of those Upstream sources were released to the community with the intention of them being up freely available and I do I personally do not like um stopping that

or interrupting that from the end users right at that final Last Mile that final stage so to me I do believe that the culmination of all of the work from the open source Community does need to be freely available to everybody because that was the intent that every piece of every package involved in that that distribution that was the intent of every package in there couldn't agree with you more so okay I'm gonna go down two roads here first of all Kudos and compliments to the image and the distro everything that I have ever done with it works perfectly first time out if there's a

package designed to run on an x86 based flavor there may be exceptions I haven't done everything yet uh but I have been running for a year and a half two years um cockpit on there and that that was our Management console of choice when we came to to teach that and I didn't have to do anything I didn't have to do any uh configuration changes to make it work or any kind of recompile it was plug and play and now all we got to do is learn the product and and it's been that way with just with every project that I have ever done I've

read people who I've got a little bug here and I got a little challenge there but it works so awesome thank you guys we stand on the shoulders of giants those giants also stand on the shoulders it's Giants all the way down okay so an Enterprise such as the ones that you guys have been participating with for so long they're not cheap I I'm thinking you know I can't take the money that I made last week and start a new distro and have it turn into Rocky so how do you do that actually you can you can and here's why here's why there was a

number of tenants of what we created Rocky and the rocky enterprise software Foundation to do it was very important to us that what happened uh just to be blunt and I don't mean anything ill out of any of this but what happened with Centos doesn't happen again we wanted to make sure that we're creating a structure an organizational structure that the organization is is protected in as many ways as we possibly can but even more than that we have to create it create the technology such that somebody else can pick up from wherever we are wherever we leave off and go and continue this and

continue running the running with the torch or create another option and so one of the first things that we we identify so let me back up a little bit when we first started this process you know Skip mentioned that we we created a bunch of different channels and he's not joking with regards to how much velocity of communication was happening on these uh just to demonstrate it we were using a free slack tier right so you get 10 000 messages in your free slack tier uh if I didn't respond to people in a couple days messages would poof because we had so many messages coming

in and it wasn't just that they scroll off the screen like I had direct messages to me that I'm like okay I see that there I'm going to get to that one later okay next day goes by and if I went like two or three days I see that there's still a direct message from there and I click on it and it's empty and it says you know go buy slack to see this message uh it became very clear you know and we had very quickly we had over 10 000 people joined it was about a month and a half uh so that's how quick

that the community kind of took off uh from there but it's um yeah it's been it's been just a remarkable experience I I don't know where I was going with that but um well I'll bring you back around then so you mentioned in in our last end of topic uh the rocky enterprise software foundation and I was kind of yeah that would be a big help I don't know I don't know I need more coffee I didn't sleep enough last night yeah yeah keep me keep me on Focus you guys wow I need help uh so one of the things that we identified is the

thing that's most important of what we're creating is not just the operating system it is everything we need to do to build that operating system so we started actually not building operating system so the primary role of what we started doing kind of collectively as a group was to start with the infrastructure now skip immediately identified wait wait wait before we even while you're doing that I'm going to focus on figuring out things like package build orders dependencies hidden dependencies things that are not in the operating system but are dependencies to build and to get binary compatibility uh so we had to go start figuring

out all of that so skip was working on that but a large lions are of the of the organization was working on building that or that that build infrastructure such that we can have members of the community kind of come and be part at least that was the goal now we started off using Koji and Koji is a fantastic build system that Fedora uses and Fedora maintains and it is it is tremendous it's awesome uh but one of the things that we notice is it's built for predominantly physical clusters so somebody who has a physical on-prem uh or or hosted cluster a resource racks of

servers that they can throw builds at and manage those builds in parallel and whatnot it's really good at that but it's not so good at running in the cloud and as we mentioned before or at least in the green room um you know we've we've had so much support from clouds providers that we wanted to make sure we're leveraging these Cloud providers to build the operating system you build it in a secure manner in which other people can kind of come in and join and be part of this big project so we identified right away that we kind of needed to we're going to need

to build a build system and so we did that we've now released it as a matter of fact we didn't even want to release version nine until we were ready to release the build system with it so the moment somebody goes and and downloads any version of Rocky you can actually also go and get all of the infrastructure necessary to go and rebuild Rocky yourself so to your point that you just mentioned it's really important for the longevity of not only the project but of Enterprise Linux itself for for Enterprises I.T organizations and people Engineers system administrators to have so much confidence in what is

enterprise Linux we're doing these air quotes a lot today skip um what is enterprise Linux and uh know that there's always going to be a suitable Enterprise Linux solution that provides that level of compatibility pound for pound as skip said with the standard so resf is a support Foundation that provides resources for all of this development to happen so um I'm going to be very pedantic here because I want to make sure that I get this correct and everybody understands very well so we when when when I posted that message to the blog I was actually talking about ciq so I was thinking ciq is

going to come and go build this and we're going to go you know make this a project with the community we're not when when I saw 10 000 people joining that Community it was clear this is not going to be a ciq effort it can't be a ciq effort this has to be a community effort and we have to do the community in such a way that we're going to protect the project first and foremost and then create a uh a foundation that not only kind of supports the notion and ethics and morals of what it means to be open source but at the same

because there's already foundations that do that but there aren't foundations that really also cater simultaneously to the focus of where we need it the users and in our case the users are predominantly professional professional system administrators I.T organizations large Enterprise organization and those are that is our community and that is where we need to be really thinking about and making sure that yes we're abiding to the morals and standards of of what it means to be open source but we're also providing something that really cross-pollinates the the needs and the necessity of what Enterprise organizations require and that was kind of The Guiding um The

Guiding Light of of what was resf to be and how are we going to get there um so resf is a stands for the rocky Enterprise software Foundation it is a public benefits Corporation and it is separate entity and it runs itself it is managed by the community and it is a standalone entity now ciq just to kind of paint this picture being a for-profit company does not control and does not hold hostage or even own anything about resf aside from the fact that ciq contributes literally now millions of dollars to the resf to help enable this and help enable what we're building and what

we're doing aside from that there's no sort of control structure as a matter of fact we built in control structures to the rsf to ensure that companies including ciq cannot hold the project hostage in a negative way so from a business perspective that means ciq has to add value True Value not by again holding it hostage and saying if you need this if you want this you have to pay us no we're never going to do that we can't do that instead we're here to add that value and it forces an ethical business model as well so that's kind of sorry I don't know what

that sound was that's kind of um uh where we where we came to be with regards to the rocky enterprise software foundation so it is a standalone entity and that's why lots of individuals as well as organizations have supported this and if you look at our sponsors and you look at the companies that are part of what it is that we're doing you're going to see a very impressive lineup of organizations and you've got some resf has some exciting news underway right now yeah oh yes absolutely so we have just gotten um well within the last couple months we've gotten um all of our new

bylaws written our new Charter we've had it validated by you know various open source experts legals um Heather Meeker if anyone knows of Heather Meeker she's now validated it looked over it and and um uh and and contributed to it and again what we've created is this this Foundation excuse me that caters kind of to not only open source but also our user base which is Enterprise and professional I.T organizations and people so that's so all of those bylaws are now done we're right now we're right now in the process of a vote on board members so um right now skip nor I know who's

going to be on the board starting in Jan one but we have voted and we're waiting for all those votes to come in and we are anticipating by mid next week week we will know who's going to be on the board and we're going to be having a press release shortly there afterwards and the board is comprised of individual members of the foundation so contributors like Skip and myself and others as well as several independent at-large directors so these are people that are not associated with the project but do bring value into the project either as users or contributors or or open source people and

and other project leads and so all of that's coming together right now and the one thing I really want to stress as we're talking about board members is even though we have sponsors to the resf we will never sell board seats in my mind this is a Surefire way for organizations to kind of come in and start manipulating the agenda of what an open source Foundation is doing so we never sell board seats uh and that's a very important and critical distinction in our mind yeah we've certainly seen Holy Wars on either side of that coin so kudos to you hey right after you posted

in the chat feed somebody posted a question uh I'm not sure who's gonna best answer this so we'll just throw it out uh tolawit who's a regular here and he is a moderator on The Unofficial Discord channel that kind of supports our efforts here uh he wants to know what does it take to go from let's add this feature to Rocky Linux to making it part of the distro so if somebody's looking at uh the stream release perhaps or you're looking at the most recent binaries that are final with the Rel D I can do that and so there's there's a new feature that somebody

looked at and says we really ought to incorporate this how long from that germ of an idea to delivery and I'm sure it's no answer I'm I I'll take the first step and then skip already talked a little bit about Sig so I want skip to to come in as well um we've had a lot of requests for the Upstream kernel as a alternative kernel to Rocky Linux and and this has been a lot of requests like not a few I mean this is this is a lot of organizations usually it's the larger organizations that are either interested in the security model of the Upstream

kernel or some of the performance benefits of the Upstream kernel and um so Rocky and Rocky stream huh I don't know if I'd call it stream but yeah um but the way that these come to be is through the special interest groups that skip mentioned earlier so if somebody has an idea they have another package they want to include they have another um optimizations or changes um I'll put out a little poke uh if if somebody wanted to have a system five enabled operating system because that one comes up a number of times it makes me nervous to skip but with that said um yeah

I'm still a system five a net person so it's it yeah I still go back to that um and uh if somebody wanted to do that they could they would have to create a special interest group they'd have to design a charter for that special interest group and set up the appropriate leadership structure with that special interest group if you just have a package a single package that you want to bring in and a special interest group does not currently exist that would fit for that package um then we would we would talk about creating a special interest group that would be best for that

package but but generally speaking the the intention here and the build system that we created called Peridot Mustafa if you see him in our matter most which by the way we moved from slack to matter most because of the problem that we mentioned regarding message retention among other things so we moved to matter most if you do see Mustafa and and Skip as well as Neil Lewis and others on the team they're very familiar with the build system so if you do have any questions on that please do feel free to ask uh and that's a little teaser on joining our you know getting everybody

to join our matter most um but with with all that being said the build system is designed to support special interest groups and additional packages so if you come into the matter most you come into our chat or IRC and you want to start maintaining something special some addition to the to the to the operating system we can do that through the special interest groups um uh initiatives that we that we and structure that we already have and that's how we would we would uh we would do that skip is there anything that I wanted that you wanted to add on a technical side or

other yes or just not not so much technical but it's an amazing question we actually get it a lot and I think one of the uh I always say this in our release meetings but um build it and they will come right so I'll give you an exam a good example is how the Raspberry Pi image got started um we had a we had a chat channel in our and by the way hey Greg talking about matter most I would say chat.rockeylinux.org and the bridge we have bridged IRC Channels with it for folks who like IRC is the nerve center of Rocky Linux development and

conversation and everything I just posted the link to uh Rocky website and it's in the tab right on I think not in about but under support I think so and we have you know all kinds of folks there for all kinds of reasons and you know lots of troubleshooting happens there lots of development discussion just everything people are interested in and they're all it's organized into separate channels and so forth um but yeah so we we had a and we and we currently have an ALT Arts Channel alternative architecture right which the Raspberry Pi kind of qualifies because it's not supported by stock Red Hat

Enterprise Linux and so I mentioned before I was doing this work you know but I didn't really have you know I was I was chatting up with you know folks on the chat Channel and asking for ideas advice you know when I'm trying to solve a problem or something but the original you know I I'd recommend the you know doing uh iterative development right so you know try something right and you can you can compile things yourself no problem right do it yourself I'm all about do-it-yourself and so um and you know with with help and advice from other people obviously you want to collaborate

and get something going and you know get have a prototype or something to show off and be like hey look we would love you know here's the RPM that we want to include in a you know a special Sig repository or something right we want to develop it further and then we can you know like the Raspberry Pi image started out on my Home Server right and then you know it was distributed to friends for testing and they were like this is awesome let's let's publish it and then eventually you know we work our way up to the uh um and now it's on the

you know if you go to the downloads page and there's an alternative images section and you can pull it right there and um so it's it's really I I encourage not taking the cart before the horse right don't set up this entire you know giant thing we're you know going to have all this you know stuff going get take it let's go product and then we'll go you know formalize it more right so that's I don't know if that's business speak but yeah it's uh they call that product driven development I don't even know but uh yeah so it looks like that answers another question

that came up but I guess there's a half a question here we can still hit uh Andre posted it two minutes after the hour uh how do you decide what to add and what to skip so when does the nice idea come out where everybody says no no no no no not that so really I mean as here's I'm going to give you some more examples here so first of all I I put a little blurb in there about it but uh so core Rocky Linux you know forgetting the sigs for a moment the special interest groups call Rocky Linux it's actually very easy to

decide what to add and what not to what is Red Hat Enterprise Linux doing that's our whole mission of core Rocky Linux is to match it again pound for pound version for version everything right now and so if someone you know wants to create another you know a newer version of a package right like maybe I think um something that came up the other day actually is someone said hey I don't like the version of USB guard that's included in Rocky Linux and are you know unfortunately our answer to that is well we cannot put the new version of USB guard into Rocky Linux because

our goal is to match Red Hat Enterprise Linux however if you would like to you can totally grab the Upstream Source from Fedora compile it against Rocky Linux 9 and you know if you might there might be some work involved I'm I'm sorry to say like there's there's effort involved in everything right but um yeah and you can get it working and if you get it working for yourself you can share it with people and if people are interested we'll I will host it for you absolutely we'll put it up on our git right and we'll you know and then we you can we'll even

put it you know we'll make it available in a repository space for you it won't be the primary Rocky Linux repository people will have to enable it right they'll have to it's an add-on right but absolutely yeah we'll um you know no problem like it's and there's again there's steps towards doing that and there's a lot of information in the wiki about you know if you want to found a Sig we also tell a lot of folks that if you want a one-off package like that it's Rocky Linux might be appropriate but also there's an there are amazing third-party Repository series for Red Hat Enterprise

Linux and one of the most popular is called Apple I don't know if you've heard of it but it's probably extra packages comments about Apple so surely extra packages for Enterprise Linux and if you're you know a lot of times especially if you just want a slightly newer version of something or if you want this one package and not you know it's not a big thing like the Raspberry Pi supports just a single you know thing then it might be appropriate for you to head over to Fedora and check out apple and see if you can get hosted or you know if you can become

an apple maintainer there highly recommend checking out you know and they're they're not really Upstream from us I suppose but they're they're definitely a great Community there that maintains a lot of stuff and um yeah it's uh so it depends and we have conversations about where stuff fits right it depends on you know what the the there's a million situations and there's a different answer for each situation but uh yeah it's yeah well we've got a really wide variety of audience and most of them aren't here live you know we got a pretty good turnout for our usual but by the end of the week

our our viewership will go up 50 60 times of what the live show is there's a number of them are interested in pursuing development careers though they're not always sure what that means I've got a degree in a podcast or something like that so I can code now what do I do with that so here's one of the things you can do take those skills and find a project that you like find an application that you want to help develop find some flavor of Linux that you like do some bug research maybe do some app development whatever you've got and so here's a wonderful opportunity

and that's I wanted to cover into this I I sidetracked quickly and easily but there was definitely something I wanted to mention when I was asking Greg about uh is some earlier projects particularly chaos I went back and looked at a lot of the old boards and discussions and things like that and one of the things that stood out the most is how friendly everything that you've done is to all levels of contributors from the the most base newbie who only wants to contribute by saying I really like this or I'd like to try this and I'll let you know what I think all the

way to a skip or a Pablo level contributor a rocky level developer so kudos to you on all of those accounts um it's you know the the culture of the open source project is is really important um I've learned a lot and I obviously continue to learn a lot on on how best to maintain that and um and and sometimes you know it's it's easy to get into um arguments and whatnot and and we try not to um we've we've had a couple of reminders of of how important it is to make sure that we're always staying on a positive vibe and a positive note

and um again yeah there there have been mistakes along the way but mostly everyone on the team has learned from every one of those mistakes including myself and uh yeah our goal is to just do better all the time just to do better um you know Skip brought up Apple um if you don't mind I'm gonna share uh I hope I'm not pivoting the conversation but these are our update metrics and it provides a really cool glimpse into what's happening with regards to all of the different Enterprise Linux variants and this is data taken off of out of apple based on what is shared publicly

with the world um so as as as people connect in obviously there's logs they anonymize those logs and then publish a derivative of those logs so people can run metrics like this this is about when the end of life announcement of of Centos hit right at the beginning of this graph you can see it took us a little while to go and build and build Rocky Linux and then to release something took us almost seven months to get that out there uh it was about four months building the infrastructure or predominantly focused on building the infrastructure and then about two months taking the work of

skip and others and then throwing at that infrastructure and then ending up with a with a an operating system in about another month of testing uh and then that's how we come to the seven months and you can see from right there we started increasing and um and you can see kind of where we've been going and and what the trend line looks looks like it's definitely the steepest acceleration out of all of the Enterprise Linux focused operating systems and um and this particular chart is looking at something called um well it's non-effemeral and usually I start with ephemeral but this is the non-ephemeral and

ephemeral is short-lived systems systems that have just been um uh provisioned and running for less than a week and so we end up seeing kind of a lot of CI CD pipelines a lot of microservices devops stuff testing stuff and whatnot ends up ephemeral but more of the persistent systems uh is what this graph is is showing and you can definitely see a really kind of cool uh curve here and you've seen a couple times where you know Red Hat Centos and us have kind of just you know traded back and forth a little bit in terms of leading and it's been a fun race

to watch uh but you can see the incline of this if we look now at the ephemeral systems we see a really different story and it's a really interesting one as well so this is what what you see when you look at the ephemeral systems now it's very it's much more Jagged there's a lot of potential reasons for that we don't have um the specific answers on on who is connecting how are they connecting and whatnot but you can see it's very it is in fact very jaggy but you can see it's almost like we've we've replaced kind of the CI CD usage of of

what red hat was and we're kind of taking over a lot more of that now and you can also see in both of these kind of the effect on Red Hat by end of life incentos in terms of the increase of red hat Enterprise Linux so a lot of people you know ask why didn't why did Red Hat do that of course we don't have the answer we don't really know but this graph tells a very story a very interesting story and I'm sure that red hat um at least I would guess that red hat um was was expecting some of that to occur in

terms of uptake so um they were very successful I would think in terms of that goal if that was in fact their goals um but yeah what I love here is just the amount of uptake that we've had and continue to have certain analysts that we've spoken with uh have shared that they fully expect this to continue and they're very much looking forward to you know what next year and the following year are going to look like with regards to Rocky uptake well all right so I want to work with this chart for one second oh sorry I can go back to it yeah just

for a quick so there was some news that surprised me on December 7th there was a uh a news release out of the Fermi labs and out of uh CERN no you're no doubt familiar oh yeah so for everybody else this lovely blue line down here is Alma Linux they started developing around the same time that Rocky started developing and where do you you fit them are they Downstream are they upstream or did they replace Centos how do we characterize them they are basically solving the same area that we are focused on solving as well um they they came out and announced that they were

going to do something I think a week after we did um I've talked with the founder um one one of the reasons um that he mentioned was he was trying to get a hold of me but honestly it was very difficult at that time to get a hold of me um and because he couldn't get a hold of me he decided to just go ahead and create create their own um and they did so and they announced it way back over here and you can see they were out to Market much sooner it's kind of hard to see when this line starts taking off right

there but it was it was quite a bit sooner than than we were and um and they've done you know a lot of work to to really get that out um soon and they continue to uh to to be you know to give them a huge amount of credit um they're very fast in terms of releasing new versions so Point releases and and major versions We tend to hold back a little bit and and it's it nothing negative to what they're doing or their process but we really want to make sure that we're going through all the appropriate testing and making sure that everything is

completely reproducible before we release um I I would assume that they're doing similar but I don't have transparency into that and I would expect that they're doing great so of course the news which surprised me just because I'm probably a bigger Rocky fan than I am Alma by far and away is that uh both CERN and fermilabs announced that going forward for the most part and not 100 their Linux distro of choice is going to be Alma uh and so I wonder did they come to you and say hey if you give us this much more support or if you do then we will select

you or is that just something that they did on their own do you uh I'm not privy to but I'm not privy to what's why CERN made the made the um but I but I have I do have some um I have gotten some feedback from the Fermi lab side from the CERN side it seems as though you know they made some judgment calls that um you know for whatever reasons you know they they ended up choosing that I I don't have specifics into that but I I have a feeling from based on feedback that I heard um we did not release version eight for

uh PPC 64 for power we did not release that for version eight we wanted to make sure that we are getting the big kind of um mass amount of the community released first which was x86 an arm and so we really focused on x86 and arm and you can see even with that it took us it took us a significant amount of time to get that released we didn't want to delay any further and and really make sure you know try to get PPC out there as well uh so we didn't we didn't do PPC one of the bits of feedback that I've gotten was

CERN does have some PPC use cases and as a result that that influenced their decision moving forward to version 9 we have not only PPC but we've got s390 as well as x86 as you can imagine and arm and we are working with with a few people on risk and so we're hoping to get a risk build done as at some point now it would be a very different type of build than what we're currently doing so it would be an alternative build because that is not supported Upstream so we would be kind of patching and making modifications and kind of forking to do that

and we will of course offer all of our changes and whatnot back Upstream if they're if any of that is interesting uh to to Upstream but for the most part yeah so we're not sure exactly what the cause was um from for CERN to make that decision um but uh Fermi who is basically you know it's it's you know Fermi does a lot of amazing research but they work extremely closely with CERN and they they're they're kind of you know a major point of presence for a lot of the cerm computation and and research that's happening here in the United States and as a result

of that they just followed uh cern's uh surge um decision point on that so that was the feedback that I got back from from Fermi uh but in either case this is one of the benefits of having uh you know even multiple Solutions within the Enterprise Linux uh family of distributions is if anything ever happens to Alma if anything ever happens to any of the distributions not to pick on all of it if anything happens to any of them there is now choice and this is part of why we want to make Rocky reproducible is because we want to make sure that everybody who chooses

uh to go with Enterprise Linux knows that they have the confidence that this is stable not just because one organization is stable but because many organizations are doing the exact same thing and it is a very minor lift going from one to the other um one other point I'll just make on this real quick uh and then I'll stop blabbing uh at least about this um is uh one of the another aspect that Peridot kind of changes is an area of focus that we've had really since day one I spent a lot of my career working for the Department of energy working for U.S federal

I know some many of the aspects of what they need and and what they're looking for couple that with um you know presidential mandate on supply chain security within the federal government and validation of supply chain security uh We've we've spent in our continuing until we release this uh we're putting together a huge amount of transparency so uh if somebody has has a rocky Linux system that they've built right they can go and validate a binary and say that binary came from that RPM and that RPM the checksums all match so I have confidence in that binary because it came from that RPM and that

RPM is signed by the organization who distributed it so I have confidence in that RPM but do you have confidence as we continue walking up the the chain right do you have confidence with the with the organization who built it and how do you know that you have confidence with them this goes right into absolute you know supply chain transparency and security so Peridot is adding additional functionality around not just you know s-bomb um logs and and everything else that you need to get that supply chain security but also cryptographic validation and signing of each one of those artifacts so you can go back and

you can validate um this binary came from this package which came from this organization which came from this build infrastructure which came from this s-bomb and here's the entire build log of that and if somebody really wants they can go back and follow that entire path all the way back up and that's something that is incredibly important to us and then couple in additional security at credit stations that we're doing so as an example of this and I can probably turn this this image off so you can see everybody as an example of that uh I mentioned kind of how ciq is putting capital and

putting resources back into the rocky Community we are funding a fips validation of of Rocky anyone familiar with fips knows that it is not only a very long and drawn out process but it is also a very expensive process so it's approximately two hundred thousand dollars for each cryptographic module that you wish to validate and in the operating system there's five major cryptographic subsystems that need validation so this is a million dollar investment and we are not only doing the fips validation um uh you know for for business reasons but we're actually open sourcing all of that so once it's done we're giving that entire

validation away to the community including those binaries that have been validated to the community for free and anybody can leverage that fips validation so this is a really important step and it's a huge I mean for you know ciq is now you know we're we're just past 70 people so we've grown we're getting we're getting to the point where we're no longer a small company which is which is kind of kind of funny for me because I've been a CEO of a small company for a while now and so we're getting to the point where we're now no longer a small company but it's still

a million dollars is still a huge investment and that's just for version eight for version nine is another million dollar investment and we're already starting to prepare for that now so you can expect so 8.6 is going to have a fips validation on it and 9.2 is going to have a fips validation on it and all of that is going to be freely available again that's amazing uh okay I I'm chewing up a tremendous amount of your time here if you've answered all my questions and then some even the ones I haven't asked you've taken care of I got two quickies just to close out

my list and then anything else you guys want to talk about or if you've had enough of me take off and enjoy the rest of the day um okay you're you are big false proponents as am I um I don't donate to a lot of organizations uh some I've picked up recently and my longest term one is Linux Foundation to my mind the largest Foss supporter organization out there do you work with them you just do you benefit from them where does that fit yeah that's a great question because it's my own personal love yeah so um I can say this and I probably I'll

stop after that okay we work we work we have a very good relationship with the Linux foundation not only the foundation itself but members of the foundation so I can give an example Greg Crow Hartman who is the maintainer of the stable kernel so when I was talking about bringing in the stable kernel support for Rocky as an alternative Colonel we are working with Greg directly on that kernel and Greg has been just fantastic I can also mention that he has been nominated as one of the uh independent at-large board members for the resf so hopefully we'll have some um you know good news with

regards to everybody who's been you know nominated who actually got voted in but he is definitely on the ballot and um and we are very excited and happy to be able to work with Greg um on things like this but again the Linux Foundation has been just fantastic and tremendous not only in in terms of our partnership with them as a as a foundation Foundation partnership but also just in terms of what they've done for the community uh it is it's massive and we're very appreciative to everything that the Linux Foundation has done so I'm glad to hear that you are a donator to to

the LF there's not much money in these Pockets but you got to pick some and that and eff I love and a couple specific applications out there all right last one uh this one is all for Dave um so what I do for a living reportedly is I teach so I'm with an organization where we write books and make videos and uh our primary stand-up customer base are federal law enforcement agencies so we teach them what's a computer what's in it and enough background that they can address technology forensically and so when they're on the stand they've achieved some level of certification typically using our

materials or our stand-up instruction what do you got in the world of Education what does ciq have in the world of an educational program oh my goodness fantastic it's a fantastic question so um I have I and you mentioned LPI earlier um I I have a a weak spot for LPI um I I mentioned I work for the Department of energy before I worked for the Department of energy I work for a little company called Linux care um I don't know if you remember Linux care back from the late late 90s early 2000s uh they were that was one of the most fun companies that

I ever worked with because I got to work with I mean just Legends and and just so many tremendous amazing people in open source people that I mean you you just the moment they start talking everyone shuts up and just listens because of the amount of stuff that's just amazing that they can share um so I'm not going to start name dropping everybody but there's a lot of amazing people that came out of Linux care and um and so I got to work there um for for a little while um and uh it was it was just it was it was fantastic but but the

reason why I bring that up because when I was there uh the Linux care had a um a fantastic role in some of these early kind of Linux education programs and I believe Linux care even even wrote the first check of sponsorship to the LPI so way even way back then so um I you know I was consulted on a little bit you know on some of the early LPI documents um um the same thing on the LSB side they were very you know they were a huge initial supporter of LSB um I helped out a little bit with the um Linux file system hierarchy

standard and um so stuff like that so I have a weak spot for all of that so now to get to your question um which by the by the way I'm gonna name drop actually one or two things um since you remember Linux care um Linux care had three founders do you remember the three founders no ah Dave sifri Arthur tide was the CEO and Dave Leduc was the CMO um Arthur and Dave are currently with us at ciq now oh oh that's wonderful it's it is getting the gang back together so um yeah it's just been ciq has been a lot of fun as

well um but to get to your question um you know the LPI has just done so much uh amazing stuff for the community both in terms of the curriculums that they've created um and the materials that they've created and um and whatnot uh we have had some conversations already with the LPI in terms of helping and sponsoring there and and being part of what the LPI is doing and I'm hoping we've have some really good success there um and uh yeah and we're just going to continue pushing pushing on that and helping everywhere we can within the open source community and LPI is a big

one there outstanding all right so the last completely pointless unrelated question anything at all you gotta love living in the internet age because of what we've just done here you and I ran across each other I ran across you I in some thread on a Reddit subreddit somewhere and I was talking to one person from Alma and said hey you know would you consider talking with us and and that fell apart for very good reasons but you were also in on that thread and oh wait a minute this is a way better thing so okay let's reach out to you with this marvelously semi-anonymous messaging

system that's part of Reddit said hey I got this show would you consider joining absolutely here's the name of of my administrator to handle these things send her an email so I send Jessica an email and you guys all work together I don't know Jessica I don't know where she is I don't know you guys I don't know where you are she doesn't know who I am she doesn't know where I'm at but after a string of emails and a Euro link this is the internet age I still don't know where you are so where are you guys skip you start uh I am uh

we talk about how we're Downstream from Red Hat I'm a little Upstream from Red Hat I am in just north of Raleigh North Carolina so the downtown Raleigh is about a about a 20 25 minute drive from my house yes from your job history Carolina I always go out with your work as a Linux admin for the court system they come and pluck Engineers like apples down here and they they're like they're like oh you can just join our company thank you for graduating and Greg so I spent most of my life in the Bay Area um and uh ciq even started off being a

Silicon Valley kind of startup um even though pretty much everyone in the company is remote uh it was you know I I kind of based it off of that and you know and um I've more recently you know in the middle of covid I moved to Reno so I am now uh yeah so trying to get away we're having some beautiful weather here and I love snow and I love you know actually having Seasons um California is one season which is just always nice um so but I do like all of the different forms of weather and whatnot and it's kind of fun not to

have earthquakes not to have to think because not that far a drive from the Bay Area so you still get to go catch up with the old folks yeah that was one of the reasons where it was like you know we're we're close enough that you know it's just it's what a few hours away and um it's yeah so I do go back to the Bay quite often uh due to meetings and whatnot and it is still it's it's such a common Meetup for um and location for you know so many people of our in our industry so um it makes sense so you know

I do go back and forth at a reasonable amount but yeah I'm I spend most of my time here in Nevada good well is there anything that you guys would like to talk about or promote or anything like that well well Dave you got to tell us where you are now I'm sorry so yeah my my regulars know I'm in a Northern suburb of Houston Texas little tiny Community called Spring uh we went remote to my company uh right around the beginning of the covet lockdowns February of 20.

yeah that sounds about right and while we do have some in-office activities still happen and everybody tries to turn up once every week or two by and large we've become one of those mostly remote companies yeah go in for the occasional meeting and uh our offices I I think today exist for the physical video production that we do we've got production facilities down there and we do uh classes we were doing hiring smes we and I guess we still are uh and that's either bring them in for a week or two and have them shoot or we've done some online over zoom and in similar

sessions and turn those into so we're like so many companies now I've got some very good friends uh that I've developed in the last couple years over at canonical and same thing five thousand ten thousand people and about six people in an office behind a 7-Eleven somewhere foreign is amazing on both of you guys your your current skills your current projects and activities what a pleasure it was having I know I'm throwing you out if you've got more you want to add by all means but you gave us way more time than I had possibly hoped for and I know be careful we'll talk your

ear off if you let it okay great everything he stops when you two guys get together and do something like this it's touching the show and there's no development happening and there's no negotiations it's been quite a journey honestly like I I knew when I when I got into this I was like oh yeah I can I know some uh um you know I've done a little bit of packaging work I know how to I was like nowhere near I had not seen nothing like I think uh um let me tell one one quick Story one quick anecdote with my my previous job is I

think my my boss uh pre and not Greg now but in this this is in you know before before I worked at ciq this was it was shortly after Rocky Linux had got started and we were still bootstrapping the whole thing we had nothing out there um I had mentioned to my my uh job that I was you know working on this in my spare time for you know NDA reasons and that kind of thing they were like yeah sure what Linux geek whatever do what you want but uh they they um my boss was it it should be noted too that like in my

previous job I was I wasn't the only Linux guy but I was as far as infrastructure goes I was pretty much the guy and I know there's admins out there that have have been there as well and um and so but but I I just started working on this Rocky Linux thing for a few months and it was with these amazing people you know it's you know over chat and seeing them you know nights and weekends and we're trying to bootstrap this thing and my boss comes around the corner and he's talking to I think my boss's boss or some someone Superior is like oh

and here's Skip and he's one of our you know he knows everything there is to know about Linux and I was like oh honey oh you have no no I don't like like you you don't know like these people I work with the weekends oh my God did you oh honey your boss I did it in my mind I didn't know honey but in my mind I was like oh baby you have no idea I said something to that effect to him I was like I I think I think his name was Chris and I was like Chris you don't know man you just like

I am nothing compared to this group I work with and they're all just coordinated over the internet and they're all brilliant I'm looking at you you are among the Giants on Whose shoulders more will stand more do stand and and I stand on giant shoulders to boot ditto for Greg what an industry giant a technical giant Visionary is not an unfair term by any stretch so out skip is being really modest um and while we love the modesty um it is yeah it it it you know he's also very accurate and and I want to describe this a little bit um you know we had

we had 10 000 people show up like zero to ten thousand like faster than I've ever seen anywhere in any open source project that has ever existed in my multiple Decades of experience and opens never seen anything like that before I was just lucky to be part of it and the amount of capability on the team uh I mean you can imagine 10 000 people all volunteering to say we can help we split things into channels we had teams kind of like organically start developing it became a a meritocracy across the board like a perfect near perfect meritocracy if not a perfect one in the

sense that I didn't know Skip before I had no idea you know anybody on the team everybody was just yeah everybody was just like you know here we are let's go to work and there was there was almost no way to organize it there was almost no way to manage it manage it I mean how do you manage it if you don't even have a management team right there's no I mean one person me I cannot manage ten thousand people so I was completely overwhelmed and just you know we we had the Good Fortune to you know I don't remember who said this but it

was let's start breaking apart um all of the different areas the functional units of what it is that we're creating and what it is that we're doing and within those those were kind of small enough focused groups that certain people were now able to kind of you know people started working on things and a natural organic leadership started to form and again I didn't know Skip I think Skip your name I think it it was skip in in slack but I don't remember I mean so many people used aliases many people didn't post pictures most people didn't post pictures so or or they used you

know cartoon characters or something else and whatnot he had no idea who anybody was and all of a sudden we now have an organizational structure and we have a leadership structure starting to emerge and we have things getting done and uh now the organization at that point was very manageable so we have 10 000 people show up but we can't have ten thousand people all contributing so that did filter down and um you know the the team that we got is so ridiculously amazing the the amount of intelligence capability um and and just Humanity that is within the rocky group is is so tremendous I

can't even I couldn't have even wished or you know for for a better group um if if I could have hand-picked it was it was it is literally it is fantastic and that is why from a from a ciq perspective I've tried to hire as many as I as many people in the teams that I can because at this point I mean everybody that's associated with Rocky is just so dang um just to put it bluntly and um and just really good people I've seen uh people come in from the community you know into our matter most um or into IRC um everything from trolls

to to newbies to people that have never installed Linux but want to get involved to um even non-technical contributors people that have never never contributed to Linux or Open Source before but they're like hey I do this really good is this of use and it's like yeah it totally is let's figure out how to how you can join and how we can all work together and um you know again it is all 100 Community from the infrastructure going through the operating system through the entire stack it is all community and the community owns it the community controls it and it's just been it's just been

amazing to be part of this and to witness it so I really just wanted to give a big huge kudos to every member of the team we have some of the brightest and smartest technical people um out there in the Linux distribution space and we also have you know many of the best people that I've that I've ever worked with um there so uh between that and and ciq honestly I'm I'm so happy and so excited over everything that's happening it's just fantastic so anyway I'm done raving and whatnot well I know they're looking at you saying you are not allowed to retire for at

least the next 20 years now you've got the energy you've got the vision you've brought them together the confidence uh it's so clear from the timeline do you have four minute and four months of infrastructure development and then three months later you've got a product release that's strong so Kudos yeah it was really I was amazed myself honestly even part of it I was like oh wow we did all that okay yeah and I I was thinking about what Greg just said yeah we were I consider us amazingly lucky but at the same time we had 10 000 shots right we weren't that lucky right

ten thousand tries well for everybody else another fun thing for uh for you guys besides help them out okay contribute if you contribute a couple of books if you can contribute some expertise if you've got some skill Rocky is a great place to use it check out their site rockielinux.org they got a store and I never paid attention to it until today I got Christmas shopping to do before the weekends out there's going to be some rocky stuff there we go there's the cups in the backpacks and the whole ball of wax so yeah hit the store help them out guys and and go Rocky

and we I need Beauty shots of Rocky doing cool things on Raz pies that that that's my shirt and that'll be my hat so Dave I I wanna I wanna just send out a special thank you to you as well so Skip and I um you know I I can tell you know with skip I I know Skip pretty well we we had a really something to speak for him we had a really good time this was actually this was a lot of fun and um you know please if if you ever want us on again just say the word we would love and I'd

love to also bring in some of the other members of the rocky team as well uh maybe at some point we just have a huge party sounds good I won't burden you too much but count on it next year we're going to kill this show in two weeks Drama Dave Rush ask me anything goes away because it existed to serve a niche that doesn't exist anymore and that was people studying for CompTIA certifications during lockdown lockdown is gone we're back to normal procedures but I've got a blessing from the company to start a new uh operation that's going to be pi and Linux Focus it's

not going to be come to your study focused so you guys will fit better into the new module uh whatever we're going to call I think we're going to call it Andromeda we've got drama right now Dave Rush asked me anything so we're gonna surround it and make it Andromeda duh somewhere I don't know something funky like that he's like as you've experience naming things has to be organic and it's got to be a lot of fun but that's what I'm poking at hey guys last thing real quick yeah everybody watching the show um please you know we have we have a lot of fun

hanging out in matter most and being part of the community uh if you if you if you're on IRC just just head over to the rocky Linux channels um uh if you're if you're not an IRC or you want somebody's a little bit more slack-like we use matter most if you just go to um chat.rockeylinux.org chat.rockeylinux.org you're going to find our chat system over there it's running matter most there's local clients there's phone clients and there's a web web client as well so we'd love to have you join uh and and just be part of what it is that we're doing and um yeah and

help other people there be part of the community we we would love that so please join please be part of everything that that we're working on and don't leave now because what I have uh I completely ignore the chat I get enraptured by what you were saying and so lots of good stuff here will Shaw's uh sold on rock is his next distro I've been pushing him on that for a long time xerxist is here I need another machine to run it uh what else I saw a couple of good ones Roll by I can't imagine building an OS though getting overwhelmed boy you're exactly

right Xerxes I see this on the Rock on the Rocky on the Raspberry Pi forums all the time uh I'm a beginning coder I bought a Raspberry Pi I want to write an operating system for it nobody writes an operating system anymore you do what you guys do you take pieces and improve them and fix them and support them and do whatever but they're always gung-ho in the beginning until they realize what's really involved um by the way Dave while you're looking that up you mentioned um our merchandise uh store um we we're gonna send you I'm gonna have Jessica reach out we're gonna send

you some shirts some hats and and whatnot um so please please wear it on the show and if you want give some away we'll give you a bunch absolutely do that I I always think about putting the bite on a guest and saying hey you know we got a giveaway contest here what do you got but I just didn't want to do that with you guys it was a time issue and lots of others oh Jason you're you're two hours from Skip's house you guys need to get together Jason Helms is one of our strong regulars and one of the mods of uh The Unofficial

total seminars Discord Channel there's a whole bunch of the mods here there's Asheville down the road I didn't realize that betland that you were from there Andre is from the Netherlands though he thinks for some reason that he's from Portugal but uh he's not that close okay so okay skip is responding in here some of the air you went to high school in Nashville very cool and Will likes to okay will is a a Delta Airlines Captain so he gets everywhere and visits everybody I'm gonna visit with him in two weeks when he gets into Houston next time hellowitt is in fact on a small

island in the Pacific he's in Hawaii Paradise I don't know how we go dragging down these things it's there's always something that somebody can contribute to a little piece of conversation that gets everybody's triggers tripped what do you got here uh Skip's talking to Linux Fest okay oh yes so hey uh while I'm looking in here where's the next conference or two that we're gonna see you guys at oh that's a great question um I wish I I wish I can even just answer it I I don't know oh no it's even coming up next we have this huge list of all of the conferences

that are going to be um that we're planning on going on in 23 and um yeah I'm I met a loss website that's something that I ran across here recently lots of people uh want to know where my boss is day to day he's a big personality in the industry and Mike Myers writes all of the top books in in our Niche and people call it where are you going to be next weekend where are you going to be and he's the same thing I don't know my Administrative Assistant knows that they tell me the day they hand me my plane tickets uh and so

I've moved forward within our organization to get a tab on the websites that says well here's where you'll Mike and maybe the same thing for the nice Folks at Rocky here's where we're going to be we should watch might be a good idea yeah I'm with you on that skip I'm a contributor yes Alex would have loved this show she's our photo section I'm up in uh what's the Milan she's up in Milan in a big Linux head and a ninja Dave Leduc mentioned that we're going to be at scale X scale 20x okay scale 20x and join our holy war pineapple on pizza or

not absolutely not easy easy question no okay I I'm good with it and I've actually uh sometimes I'm actually in the mood for it because it's just domination lighter it's a lighter pizza now granted I'm not like committed to it like I can do any kind of pizza um but yeah yeah so um but I'm but you know here here's another one that I usually ask is it tells me a lot about people um Pirates are ninjas Pirates Pirates I'm too out of shape to be a ninja all right we got we got the we got the pirate crew here okay and it's you know

Pirates had their had their Heyday their movie Hay Day and their and their pirate characteristics were the same in the beginning to the end you don't want to see I want to see how people vote Pirates or ninjas yeah yeah get it out guys Pirates are ninjas we should have asked that at the beginning the the hanging thing to the hanging thing and I'm like Pirates are for me we got bigger swords anyways oh my gosh yeah oh I I get seasick too I wouldn't actually want to be one but I but I okay one of our presenters and uh swag for all and I

was thinking about that and yes I am actually in a position to give away uh some rocky Linux so you get a rocky Linux and you get a rocky Linux and you get a it's Foss guys everybody should be playing with it in a VM or on a res pi it's a gimme Dave we'll send you some swag as well so feel free to um send that out to whoever you like and and enjoy we got more ninjas than Pirates I'm sorry to say hey guys thank you I am honored that you were able to spend time with us at all and so much thank

you thank you to Jessica best to Rocky I'm gonna be a big cheerleader for you uh and to some level I will find a way to contribute to it's been just one I had every bit as much fun as you guys did you're lucky you're fun to learn from if I were leaving the total seminar I have to hang out that was a lot of fun Dave thank you and everyone who's who's watching and in the chat that was a lot of fun I really appreciate how um interactive it was and and whatnot so thank you all right merry Christmas happy holidays to you guys

thank you skip thank you Greg bye all right bye-bye now let's watch them be awkward for a minute while they figure out how to quit this is always the time in which you like yes this is always the time where it's like yeah do do we leave or do we just mute and wait and then go back into Green Room I'm just gonna do uh what have I got about oh I got two minutes of the end of the show I got nothing to do but say goodbye to everybody so I'm gonna do that uh guys next week that's the way I'm gonna do this

I'm just gonna do our usual shutdown stuff and talk about what we're doing next week and the following and do the the grand departure here so upcoming episodes next week I have two developers coming in from mix Mi triple X the uh boss DJ software runs on Linux boxes runs on windows boxes we did a an episode on how to install and use mix so now we're going to talk to a couple of the developers the uh the lead developers not available Mr MIT super brilliant also has so many polls on his uh time strings that uh he sent me a couple of his top

developers so we're going to talk to them uh then we got Christmas and I don't know Mike is going to do a regular show this Monday and on Wednesday then we've got uh Steve coming back to Wednesday after that on the 28th and then we're going to close down drama on the 30th but again we got a replacement coming up so as ever and always first of all my my incredible gratitude to Greg kurtzer and Skip group from Rocky Linux they were wonderful we will have them on again next year on Andromeda or whatever the new name is going to be my thanks of course

to Mike Myers my former boss and now co-worker and everybody at total and National cyber group who so generously provide us with time and resources for us to get together here every week my most heartfelt Thanks goes to you and everyone who comes here to participate I am Dave Rush the senior instructor I told seminars and Resident Pi Specialists wishing you a great weekend take care of each other take Serious steps to stay healthy and if at all possible call or visit your parents and never forget technology is great but the greatest resource we have are you and I and with that good night I

will stop in on the Discord channel in a little while and I'll see you on the amas next week until then I guys stay online uh when I hit the button here here I am out of here later guys okay