Frozen Kernels: Vendor Kernels, Bugs, and Stability

very exciting the second thing I'm just going to barge in here real quick is if you like these kinds of topics and all the things that we're bringing to you each and every week um and you're already a subscriber if you go and you're on YouTube if you go in there and you click on the notify button then you won't miss anything it used to be that you could come in and you just subscribe and then you get notified when all the you know different um places on YouTube that you subscribe to would notify you but you actually have to click the notify button now

and if you're not a subscriber um hello subscribe we want you we need you we love you click the little subscribe button and um you'll be able to get notified click the notify button okay so now we're getting into it we're going to do a little introductions I will let you all introduce your beautiful amazing selves and just for Simplicity I'll just call on you so Jeremy you're the first one on my screen why don't you introduce yourself I'm a distinguished engineer at ciq and to be honest what that means is I get to do the stuff that nobody else wants to [Laughter] do thank

you I get drives into just about everything so anyway I'm with that oh that's awesome well that's probably what happens to all three of you right like when you really understand the lowlevel kernel stuff that we're going to be talking to today and then it's like you actually can do everything on top of it as well so thank you for that and Mr Maple uh sure uh I'm a senior software engineer and the colonel specifically um I also get dragged into everything but more from a how do we scale out kind of situation and and make sure that we can um continue to grow at

a a clip so that we're not either wasting uh our Resources with you know people have nothing to do which is never a thing at a startup and you have the most interesting background of all of us maybe maybe maybe maybe we got a lot of very interesting people here and Ronnie welcome this is Ronnie's first time on our webinar you are you've made it man you're here you live on YouTube and I think we're on LinkedIn so thank you so much and please introduce yourself yeah hi I'm Ronnie salberg I work as principal engineer in the kernel space at caq I have a long

long long background working with Network file systems and so you want to talk to me and it's not about curle or network file system yeah I'll my eyes will glaze over and I'll walk away okay well good to know what's happen I will not take it personally when you walk away from me Ronnie I promise I won't take it personal oh that's awesome okay so we are going to dive into it now um first of all I'd like to give a little little overview I mean I kind of touched on a little bit of what the kernel is some people are not familiar with it

so let's just give a little overview think Jeremy you were gonna do that right oh so I mean um yeah the Linux kernel is essentially the the piece that um mediates between the hardware and applications and it's very important and it's uh runs on just about everything if you buy a television these days it's running Linux uh it runs the cloud it runs in your car if you have a Tesla you're running Linux uh it's one of the most if not the most important pieces of software in the world and it's full of bugs and as as as it gets worked on New bugs get

added every day yes new bugs so I think you actually had some numbers didn't you Maple did you want to throw those numbers out right now that we're talking about bugs yeah well maybe not directly bugs but um for example in our white paper we the the red hat kernel or Centos kernel we're based off of is uh based off of the Linux kernel version 4.18 for the eight version uh 5.14 for the N9 version and I don't remember what the 10 is off the top of my head that's coming out soon um from when Centos first forked that there have been um a little

over 500,000 commits so lines like uh changes to the colonel since it came out um Red Hat uh sentos for the 8.8 which is the main um uh Colonel we analyzed or or did our uh graphs and stuff against has um a little about 95,000 commit in it um so it has about a fifth of the Upstream but not quite and um then I think the numbers were 5,000 known bugs in that subset so um proportionally not exactly the same but still um equivalent an interesting um so so those are the numbers for that um we can kind of talk about if we want how

we arrived at needing to look at this well I think we should mention that that's kind of interesting because essentially we are rebuilding this kernel and as part of the job of doing that uh I think it was Ronnie who first started the oh let's let's take a look at what gets fixed and what and what doesn't which was actually the Genesis behind the white paper that we published um that got a lot of uh got a lot of comments on social media and various technical websites almost almost uniformly negative might I add people people were very upset about it um but interestingly enough out

of all the comments and I I read most of them and responded to quite a few you know there were there were many people criticizing the methodology and all the fact that we didn't consider this and consider that or whatever but no one actually came out and said the numbers were wrong and that was that was kind of interesting so Ronnie do you want to talk about how you how you arrived at this stuff I mean the I I wrote the white paper but I to be honest the guys at least on my screen below Ronnie and and maple did essentially all the work so

Ronnie do you want to talk about how you got there yeah I was well I Was preparing my vacation and going going away and I thought like actually now with with all this really really good thing stuff that that Maple has built for us now with with history and everything and understanding this that huh I wonder I wonder if we can could could go and have a look at like well can we identify bug fixes and can we identify things yeah let's let's have a look yeah how how hard can it be it's just a small python script and then I ran it and this

is not right so I looked at my python script to find because it was obviously broken and I couldn't find the bug so I thought okay well maybe this this stuff Maple built was was completely broken and just yeah there were nothing wrong with his stuff either so I I was actually genuinely surprised that I did not expect expect this I had preconceived notions about yeah for example that when bugs are introduced in the kernel they are very quickly identified and fixed which sounds reasonable but not to be true that is not not true at all so many bugs live a long time before they

are but anyway that's that's that so the so the reason that we had this tooling set up is that when we provide cve fixes for our customers LTS one of the things that we want to do is when we backport um a cve fix we want to make sure that that like there are no known fixes for that for that um specific cve code so we're introd we're not introducing a a commit that also has known issues so that's where this Genesis of Ronnie's thing Ronnie's thing came from and then what we did uh he just sort of like well what happens if we run

it against all the commits we re we rebuilt so we basically tried to replay all the changes that red hat made um there are some discrepancies and some things where they made specific changes that are not reflected in um um uh by Upstream commits because we just pulled back all the Upstream commits they did and sort of rebuilt it in place and so then we know exactly which Upstream commits they used they may be modified from their original one but still the core of it still there um that's kind of how we got there and when Ronnie did this we knew that this Linux kernel

was going to be authoring cves itself and basically said that anything that was they considered a bug so effectively if a commit for a change has I fixed this commit Shaw in the past that is save some like some some um filtering all bugs are considered cve because this Colonel lives in such a privileged space at the hardware and Below user space that if you can exploit it you can exploit anything on the system effectively we should probably explain a cve is a security bug yeah sorry a cve a common vulnerability and exploit is what it stands for bugs in the C so I want

to step back a little bit and just give an overview first of all if you haven't read the white paper and the blog post go and do that or at least record this go do back go read that and then come back and and watch the recording of this because you won't understand what're talking we'll get the link for you too we'll grab the link we'll put in the comments yeah but but essentially what what Ronnie and maple were able to do was to look at the the idea behind the vendor kernels I call them Frozen kernels in the blog post because I wanted to

make Frozen jokes because I I love that Disney movie and then I was now I need to sing it's my turn to sing right go ahead Rose let it go let it go don't pass it up anymore yes so the the idea and it's a very powerful and intuitive one is that a vendor uh like red hat or or someone else takes a snapshot of the lenx kernel code base and they say this is going to be our release for this particular stream and then what they do is they only put they only add to it specifically carefully selected pieces of code and you would

imagine and and the you know the actual marketing behind that is well this makes it more secure because we start with a stable base and the true it's true about the stability what they have is very stable but their their argument is and of course because we carefully pick these commits uh for security purposes we we are slowly but surely eliminating security holes in the kernel it it it's a very intuitive model it's wrong that's the problem with it it kind of sense that oh this stable kernel you only had tiny little changes to it it must be more stable and secure yes it's more

stable it's not more secure and that was that that was basically the the r Ronnie was the first one to take his spectacles off and say hang on a minute the emperor has no clothes it's not more secure and that's that's the problem yeah I would be cautious to say that it's wrong because the idea behind it is correct I think it's the longevity of it has has yeah when it was created 25 years ago is like to to make sure that they could you know be solidified in the Enterprise and you know the Enterprise customers could reliably maintain Linux systems and drivers and and

all that sort of stuff made total sense still makes sense but maybe not for the duration that they uh that that some of the some of the vendors do we have examples of other vendors that that go this is a good idea but we need to rev more quickly like Su and AWS both do that um so I know Su is moving to 6.4 in their next Service Pack and I can't remember what AWS is at but last I saw they were on 6.1 or 6.2 um the the problem comes when you freeze a kernel and you put bug fixes into it from Upstream um

but what what Ronnie and maple were able to identify is that later on in Upstream which is where all development really occurs in Linus's particular kernel there are fixes for the changes that they have already added but those new fixers don't get added to the Frozen kernel to the vendor Kel they don't get um at least uh as Ronnie F had it's what is it 5,000 or so the white paper has the full details and those fixes don't get added now why is that a problem well um as as um Greg cro Hartman has pointed out and Jonathan corber the author of ly Weekly News

almost any bug in a lenit kernel if you're clever enough can be turned into a security vulnerability and there's some really clever people dedicated we employ one of them dedicated to uh a good old Sola uh dedicated to actually looking for and turning these what would appear to be completely irrelevant trivial bugs into security compromises um and and because the Linux Kel these days runs everything like I say my car runs Linux um my music players run Linux my high-i runs Linux my I'm watching you guys on almost certainly has a Linux kernel inside it um and bugs at that level can be incredibly dangerous

so so the the the idea is how do we get to a more secure base kernel for everybody in Enterprise Lings that's really the problem that we're trying to solve so there is the Linux kernel which is open source yes and developed by the community and then we're talking about a few different things right so there's the one that's being actively work worked on which what what do they call that just upstream or Linus's tree or Linus's tree Upstream okay so that thing has so many changes to day right constantly evolving it's probably the the the newest the best or I don't know best L

developed open source project in the world okay and then we have Upstream stable kernel describe that sure um basically um you can treat like Upstream Mainline like what the main Linux release is as the development branch and then they have a release branch which is um the minor version uh below it so like they're actively working on 6.10 right now the stable branch is 9 or 6.9 something so all the fixes that happen in in development that Greg cow Hartman um can pull into stable um become bug fixes and stable now that's when we get into the long-term discussions which is a little bit more

different and probably a good reason as to why none of the uh they're kind of inconsistent with their numbering things cuz numbering in the Linux kernel doesn't really actually mean anything yeah so it's it's makes it hard um so there's a long-term kernel for 6.6 and a long-term kernel for 6.1 and again like stable Greg C Hartman and team go what are the bug fixes that apply to this Kel and bring it back and they do that in the Upstream they provide it for everybody but what they lose out on and those two those three four things that the Linux like kernel.org the official you

know Upstream main development repository for it is they don't they don't maintain what is called the kernel application binary interface and that is for like I have an Nvidia driver or I have a melanox driver or I have a special like I scuzzy driver that their their driver code is not inside the kernel which is a very common thing to have those are usually built against specific versions like the vendor kernels where their interface into the kernel doesn't change or should have very minor changes that guarantee is not insured in anything in kernel.org currently okay so that's kind of one of the values and benefits

that you guys were talking about of having a vendor kernel is that it is I don't know if guaranteed is the right word but there it is maintained in essence to kind of have some stability and it works with your different drivers and other things that you want it to work with that's a lot less important than it used to be um because these days uh when when Linux was first started to be used in place of Solaris and ax and all the old Enterprise unix's that was a a a big problem for venders because they weren't going to open source their driver code they

you know it's like well customers are asking if to make this work on this Linux thing how can we do that it's not even stable and and so you know that that was one of the reasons behind the vendor Frozen kernel was that well we can give you that stability now that Linux runs everything um there are not that many holdouts um for getting their driver code into the lenux kernel I mean making a Wi-Fi card work in L used to be really hard now just about every single Wi-Fi card in existence has drivers already upstreamed you know the big holds out are Nvidia and

and lus gave a very famous Su gesture to invid about that which I shall not repeat he was upset about that and there are a few others but mostly most of the vendors of Enterprise Hardware they Now understand how important it is to have their code inside the lyit kernel and externally maintained now on the low end that's different when you buy these little devices and and Android particular you know has suffered from this but there's some interesting uh conversations we can have around how Android solves this problem um they you know many of the drivers for Android are are out of tree or at

least but but for Enterprise workloads you know the big um the Big Iron stuff most of that code is now Upstream not all um so there is less pressure on maintaining an in kernel uh application binary interface which is basically the the interfaces inside the kernel that allow drivers not to be changed on every release now I just want to make things very clear this has nothing to do with running an application on top of Linux you know your oracles and your you know SQL servers and you know all this stuff they use the user space uh binary interface now that is absolutely stable as

stable as the maintain can make it if you break the user space interface you get a rant and a really nasty one from lonus if you think any of us are opinionated she goes look at anything lus break user space it does still happen but it's really rare so what that means is that you can take a an application that was written for you know a Linux even three or 3.x or 4.x kernel you can run it without problems on a 6.x kernel no problem at all so do you want to talk about how how Android Sol things Maple already I don't know what what

what Android does is I mean Android back in back in the day every single Android every vendor in the embedded space had their own special special kernel that was incompatible with every everything else but that they have evolved and gotten to the point now where the entire industry in that segment are actually using the Upstream stable kernel because yeah all the vendors the vendors they don't want to sit and maintain and backport their own private Forks because there are more interesting things you can work on if you're a kernel engineer then then that so so they have pretty much United behind Upstream stable and yes

there is there is the issue with the difficulty with the kernel application binary interface not being super stable in in Linux but it's a solvable problem and what Android demonstrates or shows is that the various Android vendors have I think around 300 different out of three modules and they still get things to work even in F of the potentially unstable apis so so they all hope is not lost so they actually maintain a stable ABI or Colonel application binary interface for stable Google does that for the community or Android I I don't know the delineation at this point but they they do like when Greg

updates they pull it in if the the the interface that they have identified the pieces of the interface that they have identified that all their uh vendors like Samsung or LG or uh HTC or I'm sure there's a million other ones um the ones the the the interfaces that those vendors require to be stable Android makes stable for the thing and Greg KL Hartman has mentioned that that he helped them set that up so they're aware of how to do that now could we do that with an Enterprise Linux where everybody has the same stable um interface for their drivers and work off of a

long term and then that way we everybody benefits from all the work that Greg and friends are doing um and then we can focus on you know ciq is is focused its original in Inception was around HPC can we focus more on HPC and specialized um like really custom interfaces that you know we help our customers get their fixes upstream and and stuff like that um whereas you know maybe if Su or AWS wants to join in on that and and help support that we all help support maintain the the the interface with colonel.

org and then they make theirs more like Su makes theirs more targeted towards the general purpose and uh AWS focuses on AWS hyperscaler stuff um AR makes their database run really well yeah so let me just uh try and understand this are you saying that that everyone pulls from stable so the the Linux Mainline stable kernel instead of having all the various vendor kernels that are only maintained for six months at a time that would be great yeah that would be the goal I mean maybe open Ela can can do something I think that would be a very good solution for for everyone every single

player in the industry to to collaborate on on on that I say there is the issue with a stable kernel application binary interface but I'm not have no data about this I'm just making making up a statement here I claim maintaining a completely stable API for these modules is significantly less effort than backporting the 1,23 cves that were issued last month alone so so we can touch on that about the whole cve numbers and stuff as well so so Rose you have to understand that the whole purpose of this white paper and everything is to try and move the industry so that Ronnie and maple

can do more interesting work this may be greedy so they don't have to spend their lives backporting you know security vulnerabilities from Upstream to and and the other thing you know I want to point out is you can get the Upstream kernels for any Linux drro you can get them for Debian you can get them from Rocky you can get them I I know you can get them from Susie probably Oracle too um so this is not like oh you know we want you to use a ciq Upstream kernel no no just just use an upstream kernel I don't care it comes from it would

make our lives so much you know rather than backporting code that's already been written we'd actually be able to work on new stuff which is you know the goal of all Engineers really um but yeah do you guys want to talk about the CVA stream and how that has has turned out how that happened sure um I've done a little bit of of digging into this um so we just rebuilt the 8.10 Centos base and um Red Hat SL sentos like us will include in their change log for all of that like what what fixes specific fixes that that were introduced fix cves for the

entire life of 8 do of eight up to 8.10 there were 300 cve and that they fixed so that's 5 years right um they I'm sure a lot some of the other bugs and back parts and stuff they did were Technic cve fixes but nobody no independent security researcher cut a cve against them since um February the very beginning of February with the new way that the now the Linux kernel is the only people that can create cves against the Linux kernel there have been over 2,000 new cve in just four months and um looking at the red hat analysis of it over 1,000 of

them impact Centos 8 or red hat 8 and 600 of them impact uh Red Hat N9 I do not know how um if they fixed them yet it just right now they just have yes these are impacting um does that mean that impact like R 8.6 or R 8.8 that's you know further analysis you have to do and you have for every single cve you have to compare it against every single one of your long-term support kernels so like we have 8.6 and 8.6 RT 8.8 and 8.8 RT and then soon 8.10 and 8.10 RT and then the nine versions so there's a lot of

work and a lot of comparing code that you have to do just to fix one cve across all of your products whereas if this was in the Upstream it just gets merged in by Greg and then you suck it down and you're I'm done um so uh well unless suck it down and you're done yeah that that that is what what makes Greg's Kels and the stuff he supports or provides so attractive because if if if if someone asks do you have a fix for that cve then you don't even have to check you can just say yes because by definition every single K of

cve has already been fixed in in in his tree before the cve is even published and actually that might be a good thing to clarify for the stream uh for for those watching is that cves or common vulnerability exploits are usually made when a security researcher is either very specifically looking into an area and discover something and then they you know tell the tell somebody that they need to get that fixed or they're looking at a fix that was introduced like a like a code change that was introduced that fixes a bug and then they see how they can exploit it and then show oh

this is this is this bug is actually a cve and they get their their like the security researchers their whole like currency for their profession is how many cves that are of certain quality did you did you like recognize sound like an awes job actually like like a demolition person but in like the tech world like it requires it requ a certain destructive mindset yeah that's that's there's there's no currency in in having lots of cves attached to your name anymore because Greg has won that that that battle already he he he has 250 CVS every week no one can compete anymore it's over so

the reason the reason that they issues so many is essentially they don't there are so many bugs that get fixed they don't have time to analyze could this be exploitable the answer is usually yes but it require to actually create an exploit and prove that it's it's a flaw that can cause a security issue it's a lot of work so it's easier to just assign a cve fix the bug and move on and and so that that is true if you pick a random cve it might be a lot of effort to to fix them I I would rather just be more selective and just

pick the cdes where the commit message actually describes how you would exploit it yes there are some of those yeah yeah so to wrap it up this whole paper why we started looking at this which one so we could maintain like doing this internally ourself I've worked at other companies where we've done it a weird way we'll just leave it at that and I didn't want to repeat that and I wanted to be able to share um those fixes across the the various code branches easily and the way we designed this allowed us to do that while we rebuilt um the the the Centos kernels

and the minor release kernels and then we are on our own after that we don't have any access to what red hat is fixing in their extended support so um we have to make up all that difference and that is a lot of work and we wanted to try to minimize as much as possible but this kind of did expose you're like oh man I like my you know as bleeding edges possible systems that's all I run at home but I understand why vendors or like commercial ENT prises don't but it might be time for the industry to start changing especially now that um the

way that CES are being handled by the upstream or by the kernel.org it's just it's going to be overwhelming for most companies or possibly everybody the number of cves are just such an enormous flow but also the consideration that the process of CV are much much different in 2023 and earlier like far far back in the past people would find a vulnerability they would disclose it on responsibly on mailing lists there would be a 90-day grace period where everyone would fix it and that it would be published that doesn't happen anymore there is no grace period if if a vulnerability is Found You Are it's

published immediately it's not going to be 90 days where you can wait and then upgrade the kernel and get has have it fixed before it's published that's those days are over that means if you wanted to exploit things you would literally cherry pick from from Greg's cve stream looking older Frozen vendor kernels uh for the same bug and go oh yeah I'll use that one today to break into a system so it's yeah we really need to get customers um used to updating more regularly and updating their kernels more rapidly I think um otherwise they're going to get overwhelmed too so does an update require

it like a reboot of the system or is the idea of doing it in place update a different topic I I can speak on this so yes and no there are features like K patch that allow you to patch the kernel and go I got my I got the source code I got from the the RPM I installed the kernel with this section here is vulnerable and I'm going to replace it with this one over here MH and you can do that on a live system for certain classes of fixes but not all fixes um what that allows you to do is defer your reboot

but you should still reboot um we don't offer it CU we don't have the we don't have the resources to be able to do that because it is an exceedingly extra amount of work to be able to do that it's non-trivial um however um I used to work with someone that was very very uh intimate with how GBC works so the base compiler for almost every or base C run time which is the base of just about everything on a Linux system and when he mentioned to me he's like oh yeah you can update but what happens to those applications that are running in memory

that have the old exploitable version if they didn't get restarted after they were rebuilt with the update you're like oh that's a good point your RPM is updated but you're actively running binary is not so rebooting on updates absolutely best practice yeah now downtime is usually the argument is why you don't but um yes you should reboot Colonel especially um just due to you know firmware drivers um all that sort of stuff gets updated so I think another another aspect on it is that while K spli and K patch are really really interesting Concepts they were probably built for the scenario where you want to

patch you you want to fix a cve once qu or something and defer be defer B they they W the problem is scale when you have a cve being issued every 30 minutes you can't really patch your Kel every once every 30 minutes like 247 forever batch processing Roney batch processing or it's specifically it includes like 300 and then next week there's another 300 and can you how well can that company providing the k splicer K patch stack those on top of each other yeah because then you start like having the colel go through all sorts of different kernel spaces than what it was originally

booted with um it's also more an representative of an era where um Everybody treated their servers as pets so they didn't T shut them down they didn't get rid of them whereas the cloud has really rethought and uh Sr work not just in the cloud with instances but in physical bare metal data centers is you should just be able to you should have enough resources and enough scheduling systems to be able to shut down your server leave the other one up bring it back up and you still not impact your service customers um so your server is effectively cattle cattle it's pets and cattle are

usually the the terminology and it's it's easy to replace a cattle and not easy to replace a pet and a lot of these old a lot of these systems were designed to replace pets and not cattle and cattle you just oh there's a new image just reboot it reboot yeah yeah so that may be one of the benefits of doing the vendor six-month cycle is that you really are only doing an update once every six months if you're kind of staying current that's where it's like oh we have we have to take our server to the vet every six months to get wormed or whatever

30 seconds to clarify that six months is I think what the general increment for Red Hat stability guarantees and 6 months is the minor version so we have we have customers on 8.6 and I think we can say this we've extended 8.6 lifetime for our LTS on that we update that as we fix things so we could have one week There's an update and then the next week There's an update and then it could be oh two months from now and then you'll have three weeks in a row for example um it's not just once every six months it's as we release them um that's

not the most predictable Cadence um and there's there's the way that RPM updates and all that sort of stuff works in the industry is kind of going through a like how do we how do we deterministically do these things and there's a lot of a lot of ideas out in the industry and I won't um say anyone specific one is better than another because I haven't done a full um survey but the the kind of the older model whereas the updates show up in your repo as we fix them is kind of less ideal for some customers or they'll hold off they'll only do it

once a week or once every other week or you know maybe they never do it we can't do anything about that I mean we we have to start treating servers as as essentially not long-term um resources they're short-term resources you spin one up you do something you spin it down again pretty much the cloud model which everyone is moving to and you have the high availability means that you can have a certain percentage of them continually booting and the applications still keep running yep and so that's that's model that people are really already moving towards but the you know we need to treat the update

of the systems um in the same way so it's really easy to say oh I'm scheduling 10% of my servers to be updated this week you know it's not going to affect anything everything else will keep running that's the you know um yeah I mean occasionally if your electrical socket goes out you use another one and then you go on you know um flip the breaker in the garage or whatever flipping the breaker in the garage is updating the server and but you don't just stick a screwdriver in it and TR no you don't sort of say I I can't I can't do anything until

this particular socket Works you're just like oh okay I'll use another socket and I will reboot and fix the the socket that's down right now yeah is is the way you have to think about it um you know Computing as a as a and that that comes from the the whole sort of computing is a resource you plug into the wall and get computing power which was which is sort of the cloud model and the the high model of the cloud so yeah yeah um unfortunately I'm going to have to drop I have I have to go do uh go to another meeting for uh

some stuff um I I want to leave with just my one thing I I am very adamant that all of the engineers at all of our respective companies are doing what they can with what with what the demand have been for the past 20 25 years and there are cases of of companies like Su and AWS and I'm sure there are others that are trying to do the best they can and we are um internally grateful for all the work they've done setting that up in the industry um we just we saw this new cve wall coming and how they were going to treat bugs

and how that so that's the Genesis of this paper to maybe help create a new 26 model which is how we got into this um the stability and um I'd love to be a part of that working with all the other companies I think we can all do something really amazing but uh like I said I need to drop and you guys have fun afterwards all right talk to you later all right thank you see you yeah I mean you know even the white paper we explicitly called out were not trying to criticize the engineers and the work they did because you know that's us

too you know um we we all do terrible work and great work and great well yes yes occasionally it works yeah but but I mean also being responsive to your customers and what they're needing and wanting and not just saying well this is what I think that you should do this is this would be my best way of doing it so that's that's your only option and quite frankly our customers are are both right like we have people coming to us I mean OB like want to stay on sentos for another year or two people who want to be on 86 or you know whatever

it is like be on a stable version we've got plenty of people coming to us talking about the upstreet St stable versions and um really wanting to kind of be on The Cutting Edge and do different things and other companies that actually produce the drivers coming to us saying hey can we work on this together so I mean quite frankly it's a little bit split right now so it's interesting to have this kind of a conversation where we're really talking about the pros and the cons and where we're going and and what might be some best practices as we move together well the hope is

to move more and more people to the Upstream stable because the the burden of maintaining the older Frozen kernels is is going to get harder and and and worse uh and you know eventually it's there's going to be some points where you're going to have to have there are security fixtures that essentially you have to say for for any company you say these can't be fixed on this Cel um you know there there are some changes that go because Linux changes so much Linux kernel changes so much and so quickly there are some things where you essentially have to say no this can't be done

um you're easier just replacing the kernel with a more up-to-date Upstream kernel from whatever Source this is not just a ciq thing um and so you know it yes we must respond to what customers want but sometimes we do need to be able to lead them and say what you're doing is not sustainable in the long term and so here's here's a model we think would suit you better in the long term short term of course you know we will'll always help but long I mean it makes sense right yeah because we're hum right it's like don't touch anything don't do any I'm busy I

don't have time to deal with that you know but in terms of I I'm reminded of a great cartoon I saw recently published on many social media it's it's a young engineer rubbing a magic lamp with a genie coming out and the genie says you have one wish and the engineer says I wish to learn everything there is to know about computer security the genie says granted and the next two panels the engineer going oh no and the last oh God that's accurate every security person I ever talk to they're like yeah oh God yes it's worse than everybody thinks yes that's that's such a

great it's much worse than every everyone thought and also with a with a focus that we are starting to get on security in the kernel and CVS it I think I think how the industry deals with secuity is going to change going forward because just just right now the amount of cves being issued for the Kel has increased by a factor 50 5 Z that's there are very few processes that can deal with a workload being scaled up by a factor 50 with without month or so yeah yeah yeah so so something has to change but but like I say this isn't all we're doing

right now is pointing at the problem um there are many solutions but we can't fix this on our own uh we need to collaborate with everybody who's shipping Enterprise Linux um and if we can all work together we can make a better products our products better for all customers no matter who they purchase from and that's you know that that's the ultimate goal yeah I like that well said well said Mr Jeremy all right uh so Ronnie do you have any final thoughts you want to leave everybody with about the colonel please everyone come join us work Upstream with with a stable Branch because there

is no other give Ronnie more interesting stuff to do yeah everyone I I I hope the entire industry will just come together and and work with with with Upstream stable because yeah well the alternative is we need you we want you we love you so where um where would somebody go to get involved with the Upstream Colonel um what you call col.org is that well the Linux colonel mail list but the trouble with a Linux colel mailing list is it's a fire hose so you know it getting involved in the Linux kernel um it's it's a hard job it's it's a big program it's very

complex um start small uh like with any open source project find trivial things that you know first of all install Linux on your laptop or or desktop or whatever that's that's number one even if you're only running it in the windows uh they have a subsystem for Linux these days or whatever just start using it I mean that's that's goal number one is is to just get familiar um it's interesting there are there are many Computing science uh grads I talk to you know universities and they they're learning all these sophisticated Cloud techniques but they're not s sitting there running and using Linux on a

day-to-day basis and that's going to be their job when they graduate that is systems because it's every everywhere everywhere so yeah I I mean just just start using it and then if you are uh interested yeah again the the wonderful thing that the the change in the world uh from when I grew up I I had to join a Unix company to see Unix source code these days you can download it from a million sites on the internet and this is the code that's running your your cars this is the code that's running your everything uh all of the devices so you can actually look

at it so if you want to learn there has been no better time um to learn problem is of course the amount to learn is is very large but you know that's that shouldn't be a problem if if you start small take bite-sized pieces start start to learn um to understand this um it's it's complex but it's really satisfying when you you if you make a first kernel change and you boot that colel the feeling of I did this is is wonderful and that that's one of the beauties of Open Source is being able to make those changes even if you're only running on your

own hardware and and to the the to feel the level of control you have over the stuff you're you're running it's like I can make a change I I'm not I'm not beholding to I'm sorry Rose going into the eye store to the Apple Store purchasing a a premium iPhone that you are you know app Apple dains you to be allowed to use uh Linux is not like that um you know you can mess with the internals so that's the fun part all right I like it my family also uses iPhones I I I use an Android of course but um my family uses iPhones

and I I lo having to support those things well it's nice to have options as well right it's like using a computer with mittens on you're awesome I love it sorry well so you guys working or you know watching here thank you so much if you want to talk about um you know having a stable Colonel vendor kernel we are happy to oblig happy to chat with you and go to C iq.com check out what it is that we do and get a hold of us if you want to talk talk about um the Upstream stable Colonel and getting involved with that we also want

to talk to you about that it's kind of one of the fun things about being a a tech startup is that we are also flexible and wanting to be out there doing the the newest and the best so come along on that little journey with us um thank you guys Ronnie it is like the middle of the night for this guy so just thank you so much for waking up with us it's soon sunrise okay man yeah I really appreciate you joining us Ronnie that's great because because you know as I said I wrote this stuff but I didn't do any other the work and

I really need to point that out Maple and Ronnie did all of the real work and the analysis on this yeah so they're the ones who actually um producing the the code that you you run on your Rocky Kels um and clown TS every day so yeah yeah and it was such a great story too Ronnie thank you for sharing that with us where you're like well I just was curious I just thought huh let me just look and see what I find and that's every every gate Discovery starts with that's funny yeah yeah yeah that's right every good story starts there and all Innovation

and forward movement starts with that kind of curiosity as well otherwise we would just stay where we're at so it's a really great conversation I'm sure that we're going to have more things to talk about when it comes to the colonel but um thank you so much both of you have a wonderful day and we'll see you next week and if anyone has any questions feel free to post them on the YouTube comments or whatever and we'll answer them as as we get to it so because I'm expecting that that people who are going what are these guys talking about what what is this thing

that might actually then go and read the [Music] paper perfect oh my God what are you people talking about yeah good times all right all right well thank you guys have a blessed day bye bye bye [Music]