Extended Life for CentOS 7 with CIQ's Bridge

bit more but I don't want to get into the two deep dark uh pits of internal knowledge at AWS but I was uh working on helping internal service teams migrate from one of our versions of Amazon Linux to Amazon Linux 2 um while at the same same time trying to help support them with Amazon Linux 2023 and then um prior to that um at a small stint at another company um where I was doing like run Ops devops engineering it was not really my forte um what I wanted to do and then prior to that I worked at cray supercomputing as a Linux kernel engineer

I was also the GPU technical lead so I dealt with the kernel and back porting uh we were soua based at cray and uh I also worked with Nvidia and our customers that ran Nvidia based gpus my two main big machines there if people are familiar with the industry uh was the Titan supercomputer at oage National Labs and pdate plus at cscs in Switzerland very cool yeah you you've seen some stuff I've seen things I G say you have not been bored no no so we had an interesting conversation yesterday Jonathan I think part of what I would like for you to do is just

give us a high level of what is Bridge first of all if you don't mind uh sure uh it is not more LTS I think that's the the important thing to distinguish there are other companies out there that are selling a more like a extra five years of you know LTS support we're we are offering a you need to make a decision we can try to help bridge you to one of our offerings be It Rocky 8 or rocky 9 UM however this isn't something that's encourag to stay on forever because it is it as it ages even more it gets more and more difficult

to keep our customer safe and secure as best we can um so I I don't know how much of the details we're allowed to talk about but the price goes up yes it does the line goes up because it gets more complicated compated get more difficult yeah exactly I think it's part of what we should talk about but I remember yesterday after we were talking I did go look and this Centos 7 came out this version on June 30th of 2013 so if you're still running that that thing it's time to move it's time to move on um we've had a lot of people ask

us to go do this and that's why we're doing it but the idea is to get you to something more modern I think that was part of what we talked about yesterday Jonathan I want to get into it with everyone is why it's a good idea to stay more current so we'll get into that but in your words because I have several stories that I've heard customers tell me but why do you think people are still stuck running this old version of CTO 7 I think there's a lot of reasons some of it is upgrading and migrating a your application stack between the various versions

of um a Dro release if depending on how if you use containers or don't use containers or whatnot um is challenging especially when like core like system D or core systems like uh GBC requires full updates of a lot of uh like linked software that you would use changes so you have to spend the time to update your application and updating your systems doesn't turn profit but it prevents you from having outages prevents you from being vulnerable it enables you to potentially uh leverage new things and that's hard to quantify if you don't have some if you don't have a team that is like their

whole job is to stay on top of it and stay out in front of it and for sufficiently large companies with sufficiently complex software Stacks keeping stuff maintained exactly the same is a benefit because it allows them to focus on those other things but but the longer you stay on that the harder and harder it becomes to migrate absolutely it does becomes painful I think one of the questions that I get asked quite a bit is why is it so hard to keep something like that going for so long and I mean not not just centel 7 but in just in general as as an

operating system gets older especially in Linux what makes it so hard so because the the the whole the Holy Grail of Open Source software and and and whatnot is that the code is is visible and maintainable by the community and we're leveraging that to build additional private or excuse me we as the community the Royal Wii are building software on top of that if we're building our own custom private whatever it is that you're doing at your company if you're trying to add your differentiation there um you want to stay and you don't want to have to think about these other things um I have

gone off topic um hold on let me bring it back um nope I've lost the topic all good no so just why is it hard to do oh to keep things like this alive for anything you actually posted a picture yesterday that I thought was I mean I'm a picture person obviously right like Co Cod is nice right letters and words and dashes like that's cool but an image really says it all and there was like this little penguin that was just just I wasn't sure if it was alive or not or like what was maybe it was just like basking in the Sun but

it was definitely looked very tired I was laying on this rock and I asked him I'm like I hope he's okay is everything all right and he was like well this is what happens Rose when you have did you say 800 cves in two months yeah so that that's that's its own unique situation where the Linux kernel became its own Central authority to cut cves um that is a discussion we can probably have for another webinar that can have some other experts in the field on to really discuss what why vendor kernels and this also I guess leads into your discussion as to why vendor

kernels and lock on to like oh Centos 7 and all the user space stuff is all the same that was 11 years ago what software were you running 11 years ago what version of your iPhone or your Android were you running 11 years ago what version of Facebook or Instagram was out do you want to go back to those versions imagine how much code has changed in those 11 years in the open source community so that requires a company like ourselves like red hat like Su like Oracle um or Deb and runtu um to be able to go and say all right this is this

fixed in time you know package we're going to keep pulling stuff back from the Upstream but it's still mostly fixed in time and they have in the past 11 years accelerated that code base into something that may not even be reflective so your code may have been um modified and whatnot but it's still technically vulnerable but you can't just go grab a piece of code that was committed upstream and bring it back and make it work just oh yep this code fixes this bug and we have this line but it's just not going to go yoink right in there you might need to have custom

engineering like your your team's working on the Kernel working on user space software Etc manipulate in and then you're still not matched up with what's Upstream so you you have published code that is not being maintained by the Upstream only maintained by that vendor and so the Deltas get real large and then nobody only that vendor is watching that specific version most of the time or security researchers then that then contact those people like hey you know you got a big old bug here yeah we've been seeing a lot of that recently so nice that's why it's challenging to to do that is the the

code drift and again think 11 years ago heck think five years ago what versions of software we were using so I imagine that unless you are kind of working at the OS or kernel level you probably don't have a need it's kind of like someone explained it to me like the plumbing of your house right like you just it's underneath the floor it's in the walls like you don't really ever see it no one really thinks about it until there's a problem right and then all of a sudden it's like everything stops and you are going to get that fixed and it becomes a priority

so what we're doing here talking about ciq offering a bridge we're looking at a you know a potential sewage backup to stay with the analogy but we've been kind of warned that it's coming right like usually it just happens when it happens with your house this is this is basically and and why I keep coming back to like what even is are we talking about is because you'd be surprised how many people really just don't understand what's happening with Centos why we are in this situation and what it means to have an end of life of an operating system that has been open source and

freely available for the last 20s something years um so do you want to maybe kind of touch on what your understanding of that is like to help educate people who are actually the ones who are making decisions because usually people who are working at your level aren't necessarily ones who are making decisions about you know purchasing something like ciq Bridge yeah um I think you're first off your your analogy about you know the colonel and kind of the low lever stuff being like equipped to your plumbing and stuff there's a conference called the Linux plumbers like it's called Linux plumbers and it's all about like

how the internal kernel stuff works and what not um um yeah so there was a lot going on there and I get distracted easily I'm sorry um so what is it the the question is what is it that you think that people are missing as to why they're not addressing it early yeah I mean like how would you tell someone who's actually a decision maker on purchasing something like ciq Bridge from ciq like what would you tell them is like how why is this necessary right like from someone who you're working on the Kernel operating system level and you're looking up going hey guys this

is an issue like you need to make some choices now I the big thing I would say is that red hat has done an amazing job but trying to support Centos as it can um but when IBM says that it's too much work to maintain I think you should maybe take that a little bit to heart um why would you choose ciqs version over um like uh the other vendors that are off also offering long-term support I think it is that we and and I don't know all the contract details that's I am not in those discussions I would imagine um our a part of

our policy is we are going to help you uh with like I believe we have a sender that helps you bring stuff out of um Centos s to s or rocky eight or um I don't know if a seven to nine exists but that would also be a really valid jump if you don't want to have to also upgrade Rocky eight and then your turn to Rocky nine um so we're trying to work as a partner to help you migrate from my understanding and we're not trying to say you can keep doing the same things and just keep paying us more cuz there's a limitation

to the amount of work that can be done if it's even technically feasible to fix everything like you'd normally expect in a in an LTS again this kernel version that's being shipped is 11 years old and just just to give a context the current two point releases on uh the Linux kernel like uh 6.7 do something and 6.7 the next is 177,000 commits and 14,000 commits and they're about two months apart from each other that's wild um imagine that over 11 years yeah that's a lot yeah a lot yeah not bored Mr Maple not bored no well lots of stuff to do but yeah so

that's that's where I see the like our value is different than the other companies is we have well excuse me I don't know all companies so I imagine that that red hat and Su and dbn are all doing something and I don't know what it is um exactly I haven't done that that technology survey but we are we are admitting that we can't providing the same level of LTS support that red hat provided is not feasible at our company size however we are going to do our best to make sure that you have a an established way of keeping your production servers as secure as

as we can provide whilst migrating to one of our more current uh more secure um and more upto-date uh offerings versions of Rocky absolutely yeah I love those words more secure that sounds motivating right as one of the reasons to do this kind of migration yeah so not not to put the cart before the horse we have some stuff being published soon in that area um and I'm not going to get into the details until it is actually published but um the thing that I would say it's more secure than Centos 7 is that the Centos 7 code is static and nefarious Nels can take

that code and analyze it and nobody's really watching the Centos 7 stack for these sorts of things independently every body's working on the upstream and again that code Delta is got the Upstream way up here and Centos 7 way back here some of the things that they have already fixed in one of the many intervening pieces in between might still be vulnerable that we just don't know it is because nobody's looking at it because it had been fixed by something that wasn't actually R listed as a bug fixer cve sure thank you joh yeah one of the other things that I wanted to get into

because I think you have a really interesting past and how you got here I know you've done a lot of back porting and a lot of keeping things alive for people pick a story I don't care which one I think you have tons of them so examples of why this is hard and why this is not a great idea in why you should just upgrade and stay current yeah sure uh I can speak from it from like a um so some of the designs of the konel source tree that we are using inside ciq is based off of several experiences I had at cray and

um to a lesser degree at Amazon because I was not um at uh I was not on the colonel team there um but one of the issues I found while working at cray was we worked with Su most of the time and we got we had special Partnerships with them and we were allowed to get easier access to their Source control and and see what was going on so that we could you know be real early in the beta stages of of their releases we did this weird thing at cig where our Mainline development where all of our code went into like you would go

oh this is Mainline I'm going to commit all my changes here and that's what we'll get released that's also where we had all of our um updates so like if you wanted to update from Su 11 service pack one to Service Pack 2 I had to take a diff of all of Linux from what Su delivered and Service Pack 2 to service pack one create that diff and make it as a single commit at the head of the mainline kernel which never works so you had a bunch of merge conflicts so I had to go all right well this code is changed how has it

changed and I need to go figure out what Su did I go figure out what the Upstream did and go all right well that option isn't being used so this patch doesn't work correctly and then I'd sort out those to get it to to to be able to be committed and then I had to get it to build so I had a bunch of build time conflicts and then I had a bunch of merge or runtime conflicts I had to sort through um what I found with that uh especially with like uh the vendor kernels and as they aged out is there's a lot of

changes that existed in the source code that were like oh um this version at used the classic kernel 2632 which was a very common kernel number for those that are in the industry um at one point during Su I think it was 10 10 or 11 they're like we can't keep backporting everything and keeping the kabi or the kernel application binary interface the thing that like drivers talk to that aren't shipped from the colel that make it keep working um it's one of the big value ads of vendor kernels um it they're like all right we can't keep doing this we're going to migrate to

Linux 3.0 from 2632 and they're like cool we're going to we're going to do this and we're going to be try to be more in line with upstream and I'm like cool this all worked I was then asked to support a bunch of Centos 6 or Centos 5 uh Colonels because our clusters used Centos our supercomputers used Su and our AI stuff used auntu and they're like my management is like we want to take all these cool all this cool work we did on the supercomputers and give it to the Clusters and give it to the AI stuff I'm like cool I start pulling them

out and I try to integrate them and then they're you have to go all right well Red Hat SL Centos at the time didn't didn't backport all of this but why and you're like sorting through the code and you have like 10 browsers open multiple versions of the code open um in different Terminals and you're trying to sort out like how and why decisions were made some of it's they don't support this option some of it is oh to maintain that that interface they had to change the code so it doesn't look like what it did in where it came from in the upstream and

it came really really difficult um that is one of those weird stories where you're like okay I can see what is going on underneath the covers and I don't like it but I get why it's happening um it just became difficult to maintain multiple versions of os's even though they said they were the same kernel version that 2632 sentos Kel looked a lot more like my SL my Su kernel than it did at actual 2632 kernel because they would just pull back whole swats of things um so that they maintain closer to Upstream but were basically lying about the version number so that the drive

so that the third party drivers would work um so we so that's the difficult part there you have mentioned back porting several times I just want to make sure that everybody understands exactly when you say back porting can you tell us exactly what that means because I think a lot of people hear it and it's like okay it do sound that hard but it's hard yeah so it's a lot of it a lot of the times there there's kind of two things that will happen either it will come in absolutely cleanly like you go I want to apply my patch for for lack of a

better term um versus what we actually do in the kernel want to apply my patch and it just cleanly goes in there's no there's no offset there's no fuzz fuzz is when you have a bunch of changes and there's stuff outside the context that is like oh this line is different but I'll allow one line to be different from my patch file you gotta go you got to go in and go is that fuzz something I can deal with like what if that in that fuzz it's setting a variable that you're that's in your patch to it's to a null pointer and then you try

to use it in the patch well that fuzz is bad because now you've introduced a a kernel panic for example by De referencing a null variable which basically means here's my piece of memory the actual data is way over here somewhere and you expect it to be used in your patch however in that fuzz in that area around it it says this is null and if you say I want to access null to get a value out it should Panic it should be like seg fault I can't do that you're accessing an invalid memory space um so it's it's it's not just bringing the patch

in it's also knowing the context around it and then it gets into even more complex when you get to an area like that's real old has been modified several times and it's different than the Upstream but it's still technically vulnerable but it's now you have to figure out like this patch does actions a b c and d we're like B is the like the the vulnerable part that gets fixed but we don't have Parts A and C but we have b and d and now how do we like integrate those together and make sure that we're not inducing other bugs that's the big thing is

how do you make sure that everything you fix doesn't induce other problems provided you know where the fix is which is subject number two in the Linux kernel and most cve that people see you can tell exactly which which piece of code fixes that cve they're very they've been very upfront about that in other in other um uh user space applications there might be nothing there just like it is fixed in this version it was introduced in this version there are 50,000 commits good luck unless you are actively a part of that community and my big example of this is patching webkit gtk which if

you didn't know that's basically Apple's web kit so rendering all the stuff but built for uh gnome uh there's several layers of ausc there and it just says we don't they don't maintain any stable branches they have the one they're working on and the previous one and they just flip-flop back and forth through the whole thing so if you need to fix something and you're like 15 versions behind and nobody tells you where that piece of code is you have to try to find it so then you have to become an expert in that in that piece of software to know where use after free

or null pointer exception or address overrun sort of thing is when that's the basically what the the the code the the cve says is like oh this sort of thing happens so you're like good great I get to go comb through 50,000 commits to maybe find the one so then you get going all right well let's take gtk for example here because I experienced this at AWS is what if we just rev it to the next version because we can't find all the cve patches and um not even red hat has published where all the red where the like they fixed it but they didn't

say which commit it was so we're like all right cool um so like what if we rev it to the most recent version that took one of my system Engineers three months to get working because it it changed dependencies and build structures in it so much that it took that long to fix because we couldn't find all the patches so then we believed that reving would be good he did all that work I did the next set of cves on that I got most of them and it should have worked and then we were having some other issues and a new one came in we

couldn't find it so we tried to go oh we've just we just recently rebuilt this let's just rev it to the next version there were four cod refactors in it wow so that's sorry that's what it was there were four Cod refactors the cve was past all of those Cod factors and we couldn't figure out we couldn't guarantee that we knew enough about the exploit to know to bring it back so we're like we'll just rev it didn't rev we it took that senior engineer again um senior uh systems engineer like another couple weeks to rebuild it because it's that complex of a [Music] package

that that's brutal and that's exactly what I was looking for so that everybody understands that this is hard this is not an easy thing to go do and the further back in time you're trying to go the harder it gets the more messy it gets so yeah move to a modern version of Rocky please quickly and then I know you and I talked about philosophies on patching I think I know where where this is going to go but what do you believe when it comes to patching how should people address that and kind of how current should they stay so when you refer to patching

are you referring to like frequency of updating their their fleets or updating the the Bas layers or the minor versions or the major versions sorry a little of an over I I I it's a little bit of both so I think people staying current on major version is something that that really gets ignored and forgotten the longest I still run across a lot of people on the minor versions that take years past when they should have done it but I know what I believe in I'm just curious what you how you feel about it I have a I have a confession I run Arch Linux

on my home machines um so I'm always at the bleeding edge and every about every third week I'm fixing my grub installation so that's not I don't recommend people doing that but most of the time I don't have a whole lot of um issues upgrading on like a daily basis but that's not really sustainable for production software so how do we find that happy medium um in my opinion you should probably try to Target uh the point 2 release of most 10-year cycle um um uh os's um that gives you that that lets some of the stuff get weeded out in the the zero and

the1 release and you should start targeting that because after you get past about 4.6 it starts slowing down in terms of the bug fixes the feature updates all that sort of stuff so that gives you a couple of years um I'm going to go back to my old company I liked the way that the Amazon Linux was starting to approach it is they were going you get two years of active support and then you get three years of LTS on Amazon Linux and you basically they'd release a new version every two years so you had two years of support then you there would be a

new version available so you had a much smaller much more minor jump but if you didn't want to take that one the LTS supported you to have onee rollover of that LTS versus the next one so you'd have two years two years two years that's six years out um or you'd be four years out and you'd still have one year of rollover um if if you really needed it um I like that faster iteration model um and one of the things that I liked was um they um they agreed kind of like with Upstream is that they were not using um for their Amazon Linux

2 you their target is I think 6.2 there right now um is as their version of um the Linux kernel so they have progressed the Linux kernel more in line with Mainline than they have with the actual uh Centos 7 kernel that Amazon Linux 2 is based off of so rather than having it like an army of hundreds of Engineers doing the back porting of all the time they kept most everybody updated by just building whatever that new kernel was for all the different versions that they had released so that's where I'm at I'd like to see the overall reduction in long-term support timelines but

I know that's not really that may be a future discussion for future generations of of CTO and stuff um because we have a current model right now and we need to ease our way into that get people accustomed to upgrading very nice there are probably some things that can help with that as well that maybe we didn't even have 10 years ago that we do now that might be a different you know conversation um but that's definitely something to look into for people is is help with that you have a question from art um such a Hot Topic I know I know I know he's

just trying to strike a match and throw it in the middle and walk off I know what he's doing thanks art think you think about live jel patching really interested in your your take on this Maple it is something that we don't don't have the staff to support right now um so basically you need to do your back porting and then you need to do additional back porting of those specific fixes because basically what it's doing is saying hey my kernel isn't trying to make a braw block here my Kel is in memory oh there's a fix that replaces like this block of code so

it goes oh instead you're over here now and you're running in this memory these are designed to be able to fast patch fix and reboot at a predetermined time that is safe for you to take down your service compounding K splice on top of K spice on top of K spice is generally a bad idea just I thought that live kernel patching you didn't have to reboot I thought that was the whole point of it is no it it's so that you don't have to immediately reboot okay so there is still a reboot necessary you just get to plan it you get to have the

patch and hopefully it works and then you do a a reboot at whatever time you need to yeah there will be some that argue that you don't ever have to reboot your kernel but that is taking a lot of a lot of um um trust that the memory space will remain stable um I the same thing happens in user space so one of my principal engineers at AWS he's very knowledgeable at uh GBC basically the library that literally everything runs on top of like PHP which is the current Hot Topic in this issue um what what happens when you upgrade your system you say I

need a a GB C update come in so basically the the base C library that talk to the kernel and talks to the hardware if that needs updated you've updated the RPM and thus every new instantiation and and and um uh Sim link to that GBC is is to the new version that you've just done your RPM upgrade to or your dnf update to that doesn't apply to any application that is actively running so if everything's built against GBC and you have a bunch of applications running that are running against a vulnerable version and you up and you update your GBC you're there's still a

subset of processes on your system using a vulnerable set of so or dynamically linked objects that are active because they are actively using it right now they don't just dynamically unload it so rebooting is good it's always been my Fe I've heard great things and horror stories about doing live kernel patching where you reboot and all of a sudden that stuff was not really committed to dis it was all in memory so you may not have the patch you think or thought you had yeah and I don't know enough about that area like it didn't really exist while I was pre in my previous Kel

roles um I've just been told how it works and how it like how you can liken it to like the the GBC updates where it's a little bit easier to understand um but usually with a case update there comes like a just a normal kernel update from my understanding so that when you load it up you get everything absolutely but the other problem is is not all updates come in case spes sometimes you just need to update sure I like Neil's comment servers are cheap buy lots of them and don't case G patch fantastic there you go fantastic oh man all right so it sounds

like there are pro and cons to just about everything and it really depends on your uh sensitivity to risk I suppose and what it is that you want to be doing so um yeah today obviously our topic was ciq Bridge which is basically ciq providing uh support and security updates for sentos while you are migrating that is the idea right did you give you a little bit more time to kind of bridge the ease of that pain and of course we'll help you do that was there anything else about ciq bridge that you um wanted them to know Maple uh not that I'm aware of

I don't know all the details of what we tell our customers on that I just I help support it um and we appreciate that thank you very much for that and for kind of giving a little bit of a a deep dive and explanation on um the work that goes into it and the challenge and value that this offer brings to our customers and how necessary it is I wish we had a little chart too we should have thought of that but maybe we'll pop it in there at some point just like of all the machines that are still running Centos right now it's terrifying

and I think there's a lot of them we don't even know about yeah terrifying there's a lot of you out there so hey pre-warning the plumbing has an end of life there's a big leak you don't know about yet actually you do know about it You' even told you have a leak you just can't see it yet and you will see it June of this year so that's only a couple months away so reach out to ciq um come to our website cq.

comom you can fill out any form there it's going to come to us we will happily chat with you about your environment what's going on best practices and if ciq bridge is the right move for you then we are here to support that for you and oh this has been awesome Mr Maple I haven't actually had conversations with you we've just kind of like you know typed a little bit in um in the the you know internal slack and stuff so it's really nice getting to know you and and hear you and all of your stories so thanks for being with us not a problem

thank you for having me glad I could talk a little bit about the difficulties of of what this is and and hopefully how we can help help not keep a sustained potential problem for for your for for the customer's business but help engage them going we'll help you get to the next stage and here's what we can TR here's what we are trying to offer to help you get there it's perfect yeah thank you very much Mel all right appreciate everyone we'll see you next week same time same place thanks for subscribing bye thanks everyone bye [Music] oh