One Year After EOL - The State of CentOS

So when we talk about lineage of CentOS and kind of this company and kind of where we're talking about I don't know Greg it kind of starts and ends with Greg Curtzer a little bit right so um Greg just can you give me just a short you know like this whole chaos Linux is CentOS Linux I didn't know about that right and so like can you just kind of walk through the the just the early history of of CentOS and how it all got started yeah absolutely so um as you mentioned uh Chaos Linux Chaos Linux was founded originally to be uh kind of a community uh based RPM distribution of Linux.

So at the time uh this just didn't exist. It was it was um you know I came out of the world of Debian and which was 100% community-based and communitybacked. Um and so when I when I moved to uh Department of Energy and they standardized already on uh Red Hat or more specifically an RPM based distribution of Linux, one of the things that became immediately obvious to me was I I missed the community. I wanted that community behind uh behind behind an RPM version of Linux. And so um so Chaos Linux was born. And it started off uh if anyone has been to Berkeley um you'll know that there is a uh a little pizza bar place called Jupiter on Shadock Avenue and uh we used to meet there uh once or twice a week.

we used to meet there from uh with including people from LBL as well as UC Berkeley and we we talked about this idea of you know let's create this community distribution and then at some point the word kind of got out and I don't remember how it got out but somebody said Greg you should join IRC and so on to Free Node which wasn't Free Node at the time but onto Freode and uh created some channels the next thing we know is we started getting some more people interested in this and the kind of got out and and more and more people started joining the IRC community and uh and it kind of grew from there and then and the the idea of Chaos Linux was not to create just a whole new operating system but because we understood compatibility was important.

So we we leveraged a lot of the base that was very similar to Red Hat. Uh and we did this because we really wanted to ensure that that base, you know, had that level of application compatibility, but also system administration y um alignment. So people could just immediately install Chaos Linux and just know all the tools, know where all the files are, know how things work, and everything would just just be exactly as you'd expect. Meaning it wouldn't be like going from from Red Hat to Debbian, which are for the most part, it's fairly different versions of versions of of of Linux. So that was kind of how this whole thing started.

Yeah. Uh we we leveraged a lot of a lot of the initial kind of core RPMs out of Red Hat Linux, not Enterprise Linux, but Red Hat Linux before enterprise Linux existed. And then when Red Hat Enterprise Linux came out with version two first, there was never a version one. Uh we basically said, well, let's leverage let's leverage that. And so we started building some of the core operating system from that and then extended uh out from there. We had a number of u of people and and parts of the community that were involved at the time. Jeff Johnson who's one of the main RPM uh founders um and uh Seth Vidal who ended up uh after this writing Yum.

Many people are familiar with Yum. And um uh Warren Tagami who was the creator of Fedora before Red Hat hired him and brought Fedora the Fedora project into Red Hat. Uh now the Fedora project I should mention was not the Fedora project as we know it today and love. It was basically it was Apple uh back then but then Fedora ended up turning it into an operating system and thus Fedora core came out and those of you who may remember uh Fedora core being the original operating system and they dropped the core later. So that's how all of that kind of came to be. Uh and so this community was developing.

we were all collaborating very nicely and then Red Hat u made a decision and um and I do not judge them at all for this especially being a leader of a company uh you have to make the decisions in many cases of what's right for the company and for whatever reasons they made the decisions that uh their freely available distribution Red Hat Linux should no longer uh be freely available and they wanted to really focus on Red Hat Enterprise Linux decisions that make perfect sense so Red Hat Linux kind of went away That's right. Enterprise Linux took over. Now, what this meant is well, we now need uh we still need to have have the our our core our build infrastructure and our our core operating system.

But many people in the world were were massively adversely affected by this because they were using Red Hat Linux for free. And so um we started thinking about well we could throw all of our you know all the source RPMs out of Red Hat Enterprise Linux at our build server fix everything just you know and we we could end up with with an entire build of of this. Uh we honestly at the beginning we didn't think anybody would even want that. Yeah. And then there was this thing that that that hit Slash Dot like like a ton of bricks and all of a sudden Whitebox Enterprise Linux was released to the world and white.

Wow. Flashot and Whitebox Enterprise Linux was the first Weeble. I'm sorry. I love Weeble. I don't know. Weevil's wobble. Sorry. Sorry. and and that was released now at the and and right at about the same time and I may be getting some of my dates wrong but there was the the RH rebuild um community that was also starting to come forth and uh so all these things came up.

I reached out to the maintainer or the the the person who released WhiteBox and I basically asked um I believe his name uh John Morris if I remember correctly and I asked him and I said you know is this something you want to do moving forward and do you need any help you know do you want to collaborate on this because we've got a lot of build infrastructure we got a lot of contributors and collaborators and whatnot he basically said this is bigger than I anticipated I don't I I can't do this myself uh I need help if you guys just want to run with this go for I like I work at a library, dude.

This is not what I what I had in mind. I released this not knowing the whole world was going to jump on this. So, wow. Um so, we basically said, "Okay." And um uh one of the early developers um in contributors into Chaos Linux, his name was Lance Davis. Uh he coined the name CentOS. He asked me, you know, what do you think of this name for this project? And I said, I think the name is cool. Um, uh, Rocky McGaw worked for a tip at the time and Rocky McGaw said, "We I'm already rebuilding this for a TA. We have a TA Linux." He goes, "I can help with this.

I can even, you know, lead this to some extent." And so, he kind of took over a lot of um the initial development and at this time Re was coming out at about the same time. And so he led the development of Rail 3 and um uh that was he was almost done with it. The last email on the list that I recall um said that he was about 99% done with CentOS 3 and uh I announced the project on the mailing list.

he announced the name change and uh me mentioned he's 99% done and he went off the radar uh for a little bit um due to personal issues and personal matters he was going through and then uh later passed away um and he never got to see where CentOS would end up and thus the Rocky is a hat tip to to Rocky Rocky Greg I is there music Oh my god I I think my music just turned on. I There's ghosts in the machine. Um, and I'm not playing the police, by the way. Greg, is it good music? Like, what is it? Yeah. Yes, of course. Uh, Greg, um, I I, you know, I love talking to you sometimes because you hear the names of the people that are involved.

And to me, like those, you know, it's like the the unsung like superheroes of this world that we live in right now. honestly for everybody like on this convers like literally these names of these people because I think it's just really important um and I think that's why I really love being in tech I think because I you know that so I you know I really appreciate the names Greg and it's you know and it's the people that ultimately I think communities are people right Chris and we've talked about this before and like and I think that community in particular was loved for a lot of different reasons you know it's funny Greg you were talking about like sitting in a pizza store on Shadic yeah that's how because you know what the guy next to you probably was listening to you, you know what I mean?

Like, but it's community and it's people, you know what I mean? And so, like, I always thought of CentOS as kind of one of these things that was just loved because I think a lot of like it was respected. Uh, there were great people involved in the community. I don't know what what what was what why do you think it like was just so loved, Greg? That's my thoughts. So, before I mention that, I also want to name drop a couple others who were were super important. So Russ Herald, um, as I'm thinking back, he was the person who really did a great job helping to build that community initially.

Um, huge number of contracts, uh, contacts, excuse me, in the community. Uh, huge wealth of experience, and he just he was tremendously helpful. Um, and then, uh, John Nubigan, uh, built CentOS 2, uh, way back in the day. So, uh, and CentOS 3 was actually done before CentOS 2, not because Rocky was faster or anything of the like, just because we started with Rocky 3, excuse me, CentOS 3 first and and then started with and then John started with uh CentOS 2. Um, so to answer your question, why was Sentos so loved? Um it's because it played a a it it filled a critical gaping hole that was created when rock when when Red Hat Linux not enterprise but Red Hat Linux was end of life.

How many people in the world that were running that uh that were negatively affected was massive. I remember being at a supercomputing conference I think it was in I think it was SC 04. It may have been even No, it must have been 04 because 03 this was all still just kind of in development. So it must have been 04. Um I remember it was in Phoenix. So if anyone wants to check on me I think it I'm 99% certain it was in Phoenix. Uh we I I remember just walking the floor and hearing people starting to talk about Centos. And that was the first time I really started hearing people talking about Centos.

Now, Rocky worked for AIPA and then worked to another company called Team HPC uh when he was going through some of this personal turmoil. And um both AIPA as well as team HPC are HPC companies. Obviously, Team HPC is they're HBC companies as well as several others that were in the early chaos and CentOS community. So we used SC kind of as an initial kind of um like meeting place and and we we got the opportunity to meet everybody face to face and talk with people. But that was the first place where I really started seeing it gain traction and I think it gained traction because of a couple things.

Uh one being it's freely available. uh and I think this is a really important one because uh there are thousands of separate projects inside of any one any Linux distribution there are thousands and in each one of those separate projects you can have thousands of developers. Now, of course, the bigger ones may have thousands. Some of them may have many thousands and many of the smaller ones will have, you know, maybe a dozen, two dozen, you know, much smaller number. But the amount of people that are involved in creating a Linux distribution is ridiculously it's a lot. And and all of these people working together yet independently.

And then what I mean by that is all these projects are me are managed separately. And the Linux distribution's job is to take all these separate projects and pull them together and make them work together in a way that you can have a usable operating system. And in some cases, you have to make changes to those and you upstream those changes. But in other cases, it's really just about building everything in a way that every all the dependencies are managed, all of your build cycles are managed, um uh and and everything is just going to come together and work properly together. So, uh, the idea of putting any of that, meaning what the Linux distribution is doing behind a payw wall for either the binaries or the source code, right, rubs people the wrong way.

I should say the binaries, but especially the source code really rubs people the wrong way because you've got thousands upon thousands upon thousands of developers worldwide contributing to to the greater good for free to give all of these things away and for somebody a company to come along and say, you know what, we're going to take all that and put a payw wall on that doesn't work for people very well. So, yeah. No, the area that um that CentOS really excelled was it was freely available. Yep. But it was also it came forth as a standard, not the standard, but a standard. And the standard that it really solidified was the standard of enterprise Linux.

I'm going to put air quotes around that. And enterprise Linux is a um is an idea and kind of a a I I want to say it's a specification, but it's not a specification. It's an implementation of of how enterprise systems uh and operating systems should be run. Now, it is my personal belief that that should be a standard. It should be a specification and says here's what an enterprise system enterprise Linux system looks like. Here's what it means to be enterprise Linux. And it shouldn't be driven by a single company. This should be a collaborative community experience. And uh but CentOS barring that, Centos served as the freely available standard across the world and thus it was free.

It was a standard and everybody was relying on it. Yeah. Thus everybody loved it and and I think I I it sets like the like the core tenants of any successful open source project comes down to those like those key things, right, Greg? But there's also kind of like, you know, this this this network effect. I there's like two times in my life where I, you know, I walked around a show, Greg, and I felt it, you know, like it was the first time I saw Hadoop in uh I don't know what year that was, but I was like Hadoop World. Hadoop World. It was nuts.

Like people were in the lobby until like 10:30 in the like in the night talking. Like it was awesome. The second time I saw it was the very first CubeCon. And I I will I'm I'm sorry. I No, this is before CNCF. Yes. This is Joseph Jax at the the Intercontinental Hotel in San Francisco. Like a hundred people. Oh my god, it was amazing. It was like the the the vibe, you know what I mean? It's like Chris, you've been involved in these sort of things, too. Like how does that because there's there's like this weird initial like love that just happens, you know, and like what what's the magic, man?

There's a few things that like make an open-source project successful. Obviously, adoption is one of those. Free is sometimes a good one. you know, open source being, you know, freely available is a good thing in my opinion. So yeah, but like you need a couple other things like good solid technical architectures, good sustainability and community health metrics and making sure that there's good docs and even like transparent governance is something that you have to consider when you know putting something out to the world that is used or developed by more than 50 people. Let's just say that. Um, and like the number of projects that actually get to 50 people is something like almost a percent.

One super rare, dude. Very rare. Super super rare. Um, so those things all have to come together in the right formula to make something successful. But I think for me it was, you know, my first DRO was Red Hat Linux before enterprise was ever even thought of. in the Linux space, right? Like I remember downloading and then at some point I was burning CDs over Red Hat Linux to for friends. Of course, that was when like CDRs were new, which is crazy to think about now and how they've completely cycled out of the distribution stream for the most part. So, you know, helping people discover Linux was one thing, but actually making it available in a consistent and meaningful way for businesses to adopt Linux really made it take off, I think, right?

Like it mattered that other businesses were using this. And yeah, I remember having to isolate some rail three boxes at, you know, one of my jobs in the 201s because they were still needed, but nobody could maintain them in terms of the code that was running on them. But yeah, we had to isolate them in our environment because they were so past end of life that uh yeah, they were getting we were having to basically restore them from backup every morning because they'd get hacked during the day. It was pretty wild and this was like some obscure, you know, two server installation.

So yeah, it's well there's you know there's the the individual impact of what happens on EOL which kind of I think you were just kind of like referring to that Chris but there's like you know you build these communities guys and like you've both been involved in these things where it is it's more you went to like 50 people it's that's extrapolated to like thousands and thousands of people who use the thing like it's like there's some weird direct like chart I'm sure I could build of like five contributors to like 8,000 or you know 50 contributors. It gets to it. These things get massive. You know what I mean?

And it's one of two ways people come. Oh, and there's an impact to the people who use the thing is kind of what you were talking about. There's an impact to the community as well. And so EOL is not a it's not a fun thing to go through, I guess, but it happens, right? It's necessary, though. You know, you shouldn't have Why is it necessary, Chris? You know what I mean? Like what is it, right? like at some point in time the systems you're building for or the software you're packaging for update, right? Yeah. So, you know, if OpenSSL was using, you know, hardware encryption back in the day, now it can just use the standard CPU.

That was an update. That was a version update. And those libraries and everything else got updated along with that. But when you think of OpenSSL before Heartble Bleed, it was like one or two people working on it and it was used everywhere. Um, it's that Nebraska problem from the XKCD comic. The when you end of life something, having an actual product or software life cycle matters. When you end something that means there's a new something taking its place that's updated architecture, updated software versions, updated all the things. But having that end of life process is much needed because otherwise, you know, we'd all be running Red Hat Linux still.

No, I mean because we would have set up everything the way we wanted and there's no updates. Chris, I'm still running Postgress. Like talk about well sustainability and like like wow you know like sustainability is a big part of open source it really is right like and in Postgress's case like MySQL came of age everybody was using it and now Postgress you know you wind back the clock to like 2010 its usage has exponentially grown and it's old by you know software standards so yes sustainability is a big thing sometimes a small group of people building something awesome will make an impact. It's just you don't realize when.

Yes, Greg, you you bring up a really interesting point that I've actually never considered, which is end of life's are the um are are actually a very common occurrence. Yes. Open source every version you have at some point you're going to stop maintaining that version. I never, you know, it's funny. I never considered it at the version level before, but you're exactly right. Yeah.

uh at some point you're going to end a life of a particular version uh and say we're no longer supporting that version anymore and you have to do that because otherwise you don't have forward progress right yes and and sometimes it gets to a point Greg it's okay if it didn't have any forward progress like I you guys I have friends who are still using from like you know like from 15 20 years ago you know what I mean that thing is amazing it still is there's the one version of Photoshop you learn and then it you you use that version forever. It seems like it's still very happens, right?

Yeah. But it's also not mission critical software. I mean, an operating system has a fundamentally different kind of position. Uh and it's like a strategic importance in the stack, right? So an OS is different and you know I'm sure the the who knows what the conversations were uh for the people who actually ended up I don't know maybe Greg you kind of had a little bit insight but like there was a community conversation around the EOL at the time as well of CentOS and that's what how many years ago now Greg that's December 2020. Yeah. Yeah.

So, so I I I wanted to clarify something because um it so so what there there was an announcement in December of 2020 from Red Hat that Centos is changing direction if I remember correctly the the original post it's changing direction um Centos went from being the Santos that we all knew loved and completely understood to more of a I'm going to use some words and I know that uh the messaging on on this has been very hard to follow because originally it came out as it's one thing then it's something else then it's then it's something different um in good faith you know I'm

trying to recapitulate this but it's it's it's testing ground for for for enterprise Linux it is it is enterprise Linux before it is released so people can have an ability to kind of test and see what it's going be like and and have the ability to provide feedback etc. Um and so CentOS went from being kind of behind Red Hat Enterprise Linux to now being in front of it, right? And the how that was recognized by the community was was not good. I I'll leave it as that. There was a blog um that was released as I mentioned. Um, I got wind of the blog about an hour or two after it was after it was posted.

Um, by the time I got there, there was already hundreds of comments that were very, I'll just say there were very nasty to to Red Hat and to CentOS and the decision makers. Uh, I came in and I basically just said, "Hi, everybody. Um, uh, one of the founders of Centos. Uh, I'm thinking of recreating it now. Uh, if anybody wants to join me, there's this HPC Slack that I'm hanging out in. It doesn't have much usage. Anybody wants to join, I'm hanging out over there. Um, within about a month and a half, we had over 10,000 people join. This was clear that this was a big deal.

And oh yeah, whether this was um you know whether whether these these these people that joined were were developers, contributors or just very interested in the in the future of what this is going. I don't know. But what I can share with you is about a month or two before I was talking with one of my uh older colleagues. um older in the sense that I've known him for a long time, not I was about to say so um one of my colleagues and there that I've known for a very long time. There you go.

And um uh and we were talking about the newer generation and it's kind of a shame that the newer generation isn't going to see and we didn't believe at least that we were going to see any more kind of underground open-source movements come into action where all of a sudden individuals in all different corners of the earth could now change the world. And when I was when I was growing up in open source, almost every project was like that. Like you join you're joining projects because you see the potential of where that project can come where where it can go. And um and you know that you just a few people sitting in a basement or working on the couch could in fact change the world.

And it it it kind of made me sad because like I don't think this is ever going to happen again. like open source now is predominantly driven by giant corporate, you know, organizations that have billions of dollars at stake, you know. Um, and so it it was it was it was ironic that within just a few months of saying that Rocky Linux boom came into existence and tens of thousands of people joined and ma trying to manage 10,000 people when you have like almost no management layer in place is a trick. Um, so the first thing first thing and and this actually goes directly into even how Rocky Linux runs today.

First thing that I did is I created a whole bunch of channels in Slack. A whole bunch and from um engineering development to infrastructure to design to website to community to collaboration. And I just sent out a a a general message to everybody, a channel message that basically just said I created all these different channels. Everybody go to what interests you and talk in there because there was too much talking to. So, Greg, I'm I'm sorry, Greg. I'm just laughing like that is so you, dude. I love it. And that's your MO. And people started searching the channels and started finding all these different rooms that they can go into.

And uh what happened there is is is beyond me. Like I couldn't have done this. All of these different channels started started collaborating. And like here's an example. And actually someone someone a while ago was making fun of us a little bit about it because we had shirts and merchandise available before having any bits available. We had shirts, hats, all this stuff because we had a separate team that was focused on design. They weren't developers. They weren't infrastructure experts. They were designers. They knew how to deal with merchandise. So they're off here in their corner doing their thing. We've got the web team over here doing their thing.

infrastructure starting to plan this out and work with sponsors and collaborators, but like everybody kind of got finished and and got to their milestones at different rate. So merchandise came out I think shortly after the website, which is hilarious. And so one of the merchandise shirts that we had was it said Rocky Linux and underneath in bracketed tests it said early supporter. That's because that that swag was available even before the bits were available. It was all just hype at that point. Um but I mean like all of these different groups started coming forth and within each one of these groups there was a uh a a complete meritocracybased hierarchy that naturally formed.

Some people immediately took leadership kind of positions where they were helping to direct what was happening and helping to take you know jump on things and helping to make sure things were getting done. We have other people that were um IC's and like okay what do you need me to do? I'm I'm ready to go. And then we've got and and everything in between. And so in each one of these different groups, we now had teams forming. And the entire management structure and organizational structure of Rocky Linux to this day, it revolves around this concept of team leads and and teams. So that's how we ended up kind of where we are today with with with Rocky.

Well, it's really interesting, Greg and Chris, you've been involved in this with like from the like being in a company working within open source projects as well. And I always find like I think one of the most difficult challenges like look at everything you just talked about Greg takes funding. It it t-shirts and stickers cost something infrastructure build systems like those things cost like it's this is not like some and either it's a foundation or a company. I don't know, Chris, you kind of see it from that side of point, you know, that that point of view as well. And like, and we we we we just talked a little bit about how like there's kind of like you just said it yourself, Chris.

There's like sometimes there's an organization that's funding the thing somewhere, right? Like how how does that balance work, dude? Like so that's that's a tough one, but normally the balance is like Greg described, there's different teams working on different things and how things get funded often gets figured out within those teams. So I imagine someone paid for the t-shirt order knowing full good and well like that's just money I'm spending right now, right? And when they got the t-shirts, I'm sure they were very enthusiastic. 100% sure they were probably all over the internet trying to uh you know either ship or share or however those were distributed, right?

But normally like these like a t-shirt is kind of easy to make, right? So normally this enthusiasm turns into like I it's part of my identity very quickly and you know I've I've seen it twice in my life now with like Linux and then Kubernetes. So funny, Greg, you mentioned the blog post from CentOS and how there were a lot of comments. I was actually at Red Hat when the change happened and I was clicking go live and a very, you know, OBS at the time. Uh, so this is December 2020 and I was on an open shift live stream I was hosting and this is all about Kubernetes and containers and all these other things.

But because it was a Red Hat live stream and marketed as such and it just happened to fall, you know, within hours of the announcement, get out of here, Chris. All those people from the comments that were already so toxic an hour or two, you know, when Greg saw them after you walked into a storm, walked into an absolute nightfight with nothing, right? Like knock down, drag out. And I went back and I listened to that stream again. And what you don't see or hear in the stream is my boss on my phone saying, "Hey, that was a little harsh. Take it down a notch.

I I know they're blaming you for something that didn't, you know, is none of your fault and you didn't even know about until this morning, but you know, maybe the word castigating is not something you should use on a live stream, you know, because it was just it was nonstop because new people would join and be like, you know, whatever, whatever, you know, Red Hat, come on. You're blowing up my entire infrastructure stack. And that was the thing that hurt me the most because I've worked in companies that were 100% CentOS based because they had the skills inhouse to manage and support their own Linux distributions.

Right. So, hey Chris, going from that, you know, sure. I saw that I I went and so as as we were going through this, I went and I re reviewed that video. You did a phenomenal job, dude. Like cuz that is that is a tough thing to deal with because I think often in sometimes in certain communities there's certain people that are in communities that start from a bad place and like you you weren't the person who made the decision. No, you just work at the place like Yeah. You know, at the time I think there was like 10,000 other people that worked at Red Hat and now something you know twice that maybe.

But yeah, it was very much a uh an event that I had nothing to do with, no control over. I just happened to work at the company there, right? You know, and yeah, that was that's probably a live stream I'll never forget because of when it happened hours after the CentOS announcement. Yeah. So, yeah, it was great. I it's I it's just I it's kind of and that blog post was written by Rich Bowen who I then later worked with at AWS on the open source team there years later which I think is just very like picture perfect for how small this world actually is. So Jim before you go to the next question or the next point I wanted to just share something real quick.

You mentioned something earlier that I mean open source requires money it requires infrastructure. it rec there's a lot of so you can grow an open source project organically. We couldn't do that with Rocky Linux because we went from overnight having nothing to all of a sudden there was there was like a huge amount of work that we needed to get done and it required a massive amount of infrastructure. Luckily we had sponsors, we had other organizations that were helping. Um, CIQ was one of many. Um, AWS, Google, Azure. Uh, really? That's great. Like, I mean, it was it was a massive amount of companies that came up and stood behind what it was we're doing.

Um, uh, and and our sponsors and our contributors and collaborators have always been fantastic. And I also want to give a big huge shout out uh the the teams and the team leads that uh organically came forth and and and built all of this for Rocky. Uh you know I I a lot of times I get um coined as the face of all of this and I guess sure there's probably better looking faces than mine to to get coined for for Rocky Linux. It's not mine. It might be Chris. But yeah, but with that being said, that was definitely I'm sorry. Yeah, with that being said, uh the amount of work and effort done by the community and done by collaborators and contributors and sponsors and backers of Rocky.

I mean, I want to be really like we started from scratch with nothing, right? in late December of 2020 and within six months we had our first beta and seven months later well seven months after the beginning so one month later it was it was released like awesome like from nothing and the amount of infrastructure and what that we that that is leveraged the amount of people it is so massive it is so much more than just me I I know I get I get a lot of times I get uh I'm lucky enough to to um be associated with this and be recognized as the founder of this, but I really wanted to just call to attention how much effort goes into this and how much work it is to keep each one of those versions alive.

Yeah. I don't think, you know, it's Chris, you've been doing this a long time, too. Like the open or I don't think people really understand that open source communities are actually people, guys. Like it's there's there's and it takes a lot of people. I've never seen an open source project go too far where it's like one or two people. Well, I'm going to go back to that's the only one, but that's another story. Um the like that but it's it's people and like it's so funny. I've been through these like uh I've been at companies where you know there's been a license change to a piece of open source software and people freak out like oh my god like this is the end of the world.

I get I totally understand the concern. Totally with you. Uh I I don't think people always see the forest through the trees and there's a lot of people a lot of decisions and a lot of factors that go into those those conversations. It's just not easy to do because ultimately when you change a license or when you EOL something there is a there is a long-term impact or there's a short term which is this immediate hacker news crazy conversation thing or like your blog post Chris you know but there's a longer term like four to five to six months and beyond both from a from

a people point of view and a tech point of view and so just kind of bringing back to like CentOS guys like you okay so okay so we're one this is June 30th y'all we actually scrambled to do this today because We're literally one year from the EOL. Correct guys? I It's to the date. So to the date. Yeah, it's a little cinematic, I guess. So 07 came out in 2014. I looked that up just to make sure I had the right year. So So and and it's even though the EOL was announced in December and version 8 was end of life much sooner, seven was the last remaining version of Sentos still alive.

And now as of a year ago, to your point, that is now no longer maintained. There is no more CentOS as we have all come to know and love it uh existing. Yeah. So, so Greg, thankfully there hasn't been anything like insane that's happened from a CVE point of view or anything, right? But like what are the risks that if I Yeah, I like that look, Chris, because Right. Yeah. I mean, you know, guys, what what are the risks? You're still running it. Like I guess you know some people are going to run it forever because they just aren't thinking about it.

But like what where are like Chris there's tremendous risk right like yeah I've had security roles before at companies and nine times out of 10 like we you know one company I worked with in particular they would see like CVEes before they were announced kind of thing like people would start exploiting things on systems. Um, so that actually happened on a Rail 3 box. It was that same company that we had to isolate these Rail 3 servers from. So I can only imagine a year after the fact, like literally a CentOS 7 box that hasn't been updated with anything significant in a whole year. Yes. Nowadays, that is lifetimes of CVEes, right?

like hundreds of hours have gone into researching the god knows how many in the past year CVE but now that there's no one really keeping an eye on it and no formal you know owner of something that is end of life especially in the cents 7 case it's possible that we'll never hear about a large vulnerability or something that happens in Centas 7 because it's going to be behind the scenes And most people will say, "Turn those boxes off. We'll figure out what to do with them later." kind of thing, right? Um, so yeah, the the risk is huge from a security perspective, but also from like an architecture perspective, right?

Like what are you building that can use a 15-year-old OS, right? Like at some point in time, hardware compatibility becomes an issue. the libraries you're using don't know how to talk in the architecture you're using. You know, like who you know, no one back when Sentos 7 was originally released thought ARM would become a major popular platform. That's right. Now, yeah, almost everybody has an ARM chip somewhere in their home. You know, it's wild to think about when, you know, 2014 was around and, you know, the iPhone was brand new. I don't even think it was based on ARM back then. It might have been, but you know, who knows?

Um, well, who knows? The internet knows there's history there. Um, so yeah, the just, you know, maintaining documentation around how to fix things or how to migrate because it's not just simply upgrading a system at that point either, right? When you end of life something that has completely changed, it is legit a migration process to move from one version to the next. And that is just an enormous lift when you're talking about potentially hundreds or thousands of servers. And I it's interesting too, Chris, because you know, like one year is a lifetime right now. Yeah. Things that Well, look at the AI space and everything that's changing there and it's happening faster now.

That's what I mean. And it's getting it's it's getting worse and worse as the years well or better and better the way you look at it. Two sides of the same thing, guys, because like the rate of advance right now is it's exponential. It's crazy. Like the the rate at which people are actually identifying these things and using, you know, increased processing power to actually find more ways to exploit or uh new hardware that needs to actually come out. Like this is I to me. Well, case in point, we're talking about quantum safe encryption right now because it's gonna happen at some point. It's gonna happen.

Yes. And then at that point, your Centa 7 box like game over. Like the second a quantum computer enters a cloud offering, anything that's vulnerable is going to be found, right? The amount of companies that I have now talked to that are still running Sento 7 is is massive. Yeah, it's massive. Yeah. Um there there's a there's a company that uh I will not name. They will uh thank me for not naming them. Um but they have about 12,000 CentOS systems still running in production. um just one company like the amount of systems that exist out there and the amount of companies that are affected by this and it's not because they want to run this old operating system.

It's that there's too much infrastructure, there's too much debt in that infrastructure to just to just roll off of it at this point. Um, uh, I talked to one one company that I don't remember the the the version of of PHP that they're using, but they just can't upgrade. Like they can't upgrade. Yeah. Major changes between versions is real. Yeah. They would have to rebuild their entire site to to upgrade and their whole site is their business. So there's there's a lot of reasons why people are kind of stuck right now. Yeah. And uh and it's super unfortunate. I I recently just did a LinkedIn post where I had a cartoon and in the cartoon there was a system administrator who's very happily doing a DNF update on a CentOS 7 system and saying, "Oh, there's no updates available.

Everything's good." Because success in the next panel is is the hacker just like super excited because none of those things are getting patched anymore. Yeah. And a lot of a lot of system administrators are I mean it's kind of you know it's poking fun a little bit I guess at system administrators I don't mean to but um uh there's a lot of people out there that for whatever reason they're still running CentO 7 and I think most system administrators know it's end of life at this point it's not a good idea but they also they don't have a choice. It's not the CIS admin's fault though it's business decision to continue running it.

Yeah, absolutely. But I understand those decisions, but yeah, it it's it's not the system administrator's decision. I completely agree. It's a business decision, but the problem is is that we're going to put the weight of a endof life operating system, a vulnerable infrastructure on the shoulders of system administrators, right? And those poor all those system administrators that know now that they are responsible for maintaining something that is no longer maintained upstream is like this is a this is this is scary. So, Greg, what can they do, right? Like, I mean, what like you said, Chris, it's like I feel bad for them, but also kind of they're it's not their fault, you know what I mean?

So, like, yeah, what's what's the little what's what's what's a company or a CIS admin to do, you know, and it's and those are, by the way, Chris, I love that you separate those two sides is actually Yeah, those are very important distinctions. You know, what's somebody do, Greg, like rebuild or like, you know, go go use AI to re recode this whole app in Python. And you say that, but I have a different anecdote there. But like, you know, one of the big points that Amazon was hyping when I worked there when uh Generative AI first splashed onto the scene was the fact that they were able to upgrade like an end of life Java.

And if you know AWS infrastructure, everything's Java. um upgrade an end of life Java version to a current one because basically generative AI told them this is how you do it and this is how you can do it with less error and then they just said okay you go actually do it AI and it this is going to be you guys let let's do another webinar talking about AI yeah sounds like we need to based on this one because there's so much I want to talk about when we talk about that uh so much but um it is super important to recognize I agree with

both of you um the business decision and the system administrator decisions are sometimes orthogonal but they're they're doing it for the same reason they all want the success of the business and the business side is just looking at it in terms of risks right until until one of those sudden throw systems gets destroyed there won't be a strong push a strong so how how do we Jim, I I I I know because I know where this webinar is going. Um that uh that was a that was a very good uh kind of leadin question. It's my job, Greg. So, so one of the things that we have been asked to to help with actually a lot with at CIQ was not was was this this did not come out of like the business side of the house to say let's go do this for the purpose of doing this.

No, this came out because exactly like we're identifying customers have have pain points and and they're asking for help. They're asking for how do we move forward? We created a a product called Bridge. And what bridge does is it takes uh this Sento 7 end of life system and we now provide a bridge to bring you somewhere. And and while you're in this bridge, you you you have safe passage, meaning we are uh we are providing updates. We are providing all the critical CVE fixes. We are we are derisking the infrastructure. uh for you and the purpose of the bridge is knowing that but we also still have to help you get to the wherever you're going and so so bridge CIQ bridge is the solution for exactly that.

Yeah. And you know Greg you you there was a one little bit in there that I it's so funny if people are like yeah we're gonna we're gonna recognize and identif we're gonna no we're going to patch the most critical CVEs or whatever that is. I love that as like a that is not a simple process y'all. Like no like to even get informed about them is very well that's what I'm saying Chris like being informed about them all taking the like the the fire hose of stuff that's coming down analyzing them understanding the impact creating the right patch and then just like that process guys it's incredible and like I don't know it's not for the faint of heart it kind of takes people who know the inter like like the the the the internals of the kernel like I mean and Great.

Like and I want to say um being an open-source advocate, this drives me nuts because there are there are multiple vendors to be clear that are doing something similar. Yes. Great. Do you know what every vendor is doing? Well, obviously not us, but I want to say every, but do you know what generally vendors vendors are doing is they're gate they're firewalling. They're they're they're they're gatewaying um access to this open-source code. Uh CIQ does we do do the development in the open. So you can go to CIQ git repositories up in GitHub and you can literally see every single fix we've made in the operating system and we encourage every other operating system vendor go take from us.

Go leverage it. Contribute back. We'd love that. Let's collaborate. Um this is this is this is how open source should be done. And that's our choice, Greg. And I think some others that are doing this have made that choice as well. Let's be very clear. Like I think people are, you know, I think there's I think there's genuine open source people who get it and I think there's people who exploit it and I I in a leg like it and that's a whole other conversation we go through guys. Um I've been very fortunate to work with a lot of really interesting companies that have they they know how to run a community, right?

And yeah, I mean look at Chris. this is this is everything you do is kind of around that thing like again there's that balance right the yeah corporate sponsorship is becoming increasingly necessary for wide adoption of any given technology right like the math has to add up at the end of the day and when you when you say the end of the day for an open source project right like there's infrastructure needs they have there's all sorts of communications yeah things, social media management nowadays, everything is about like having code in the open is definitely and hasn't been for years enough anymore, right? You have to build like a machine around it to get it to that 50 person level kind of thing.

And the machine could be small at first, but at some point in time it has to become large enough to sustain the user base of the community. And you know and in the case of CentOS it's unfortunate but Red Hat made business decisions and now we all have to you know manage the fallout of those decisions. A company has to make business decisions dude. It's not an easy thing to do right Greg. I mean Greg you you you are literally the person doing this right like in a in a company and it's not easy right? Like it's a it's a balance. So you know we we have we have certain um things that we have put in place at CIQ to help us with this to ensure that we are doing the right thing from the community perspective from the open source perspective and um certain policies certain procedures.

Uh Chris's primary role as OSPO lead is to ensure that CIQ is operating appropriately with regards to the open source community. We have got to be nailing this and we've got to be better. That's right. Than everybody else out there. We've got to be a demonstration of this is how open source should be done at companies. Yep. Yeah. No pressure, Chris. No pressure at all. Like no pressure, Chris. But yeah, you know, like but but Greg, I think it's such an important point and like, you know, you guys, we talked about this. There was It came up a couple different times, but ultimately these things are people.

Uh there's lots of different opinions. There's lots of different things and like look at everybody has like and I I I appreciate that. I think the best communities are driven by really really great people and I'm I am fortunate to work with two really great people who are crazy open source people you guys like and uh no Greg seriously and Chris both of you like thank you because like you guys you are you know this is what makes open source it's talking open about the things and you know saying what concerns are and you know Greg in particular like you you started some pretty or or been involved early days of some pretty foundational like pieces of tech that have driven a stack of of the future, our world.

So, when I was saying earlier, I love the all the names you were saying, I I'll publicly thank you as well, Greg, because I think it's, you know, you know, you it's deserving, and I, like I said, guys, I think this stuff changes lives. Um, yeah, and you know, Spent gave folks a great way to experiment and release new things in an environment that didn't cost them extra money for a license. That's right. Yeah. And think of the things that were built, right? Think of the hundreds of thousands of instances that are running CentOS, Greg. So, thank you, buddy again. You know what I mean? Like, and and to the entire team and all the people that you're talking about because I think I I think we need to publicly acknowledge like certain people have like changed our lives.

And it's funny how like, you know, like, you know, only certain things get bubbled up into the world. I there's so many little things that happen in open source I just find incredibly important. And I'm just I feel fortunate to be part of the whole game. So, um guys, thank you uh thank you everybody for who who everybody who joined. It's that was amazing. Uh you guys seriously uh I thank you for doing this, but I thank you for your years of, you know, commitment to open source as well. How's that? Thank you. Thank you, Jim. Thank you for running this and it's a pleasure to work with you as well, both of you.

Oh, buddy. Thank you. All right, y'all. Greg, thanks everybody. Have a great day. We'll put a recording up uh as soon as possible and socials and all those different things. So, uh, please follow us and check us out at ciq.com.