Rocky Linux and Security

you're booting rocky you're getting rocky uh and not something else so is there anything special that you have to do to enable secure boot whenever you're you're doing this with rocky as long as you kind of have relatively recent hardware and by relatively recent i mean since probably 2012 2013 and we know there's peop there's uh systems out there that are older and still good and still able to run stuff so uh if you have some older stuff there may be in incompatibilities with uh the the shim due to kind of hardware design issues early on uh but i think those are kind of few

and far between so most of the time especially if you have like a new laptop or something you can just kind of boot up and and secure it should be enabled by default if you're using rocky linux 8.5 i have a funny story with regards to this so i just got a new laptop um by the way the new dell xps 15 with oled display is really awesome and it works with rocky totally out of the box well almost totally apparently the audio had a little bit of an issue um and i patched the kernel rebuilt the kernel and tried to boot on this kernel

and secure boot came right out and basically did exactly what it was designed to do which is to stop viruses and people like me from injecting things into the bootloader uh and uh it actually it blocked me actually then i had to go into the bios and disable secure boot to run my kernel um to test the new the audio uh patch that i put in there and it worked uh so um yeah secure boot totally cool and it does give that validity on the operating system that you want to be running and uh and stops you from getting viruses and and have weird you

know wannabe kernel hackers like me uh playing with your system that's a great point too and something else to bring up with secure boot is if you're relying on any custom kernel modules or third-party drivers that might be coming from like el repo or some of those other places or indeed if you're using a different kernel altogether like one from yale repo those wouldn't have the rocky linux secure boot on them but if you're using those custom drivers then you may also have to disable secure boot or load the certificates that those are signed with into your into your tpm into the secure trusted uh

storage so that you can boot off those and you can do that with the kernel that you built too it just you'd have to generate your own keys and uh load them in yourself to to validate your own computer that uh you're you're building your own kernel so thankfully not a lot of us have to uh build build kernels every day except craig except correctly your kernel work did you go through and make your own keys no i didn't i didn't make my own keys i i just disabled it but the the the kernel fix for audio does work and so i'm i'm now doing

what i can to bribe um the the kernel folks on rocky to um to let my patch in it's already upstream so it's not like i created it but um but it works it's awesome and now now that laptop uh the xps 15 again uh oled display touch screen it's it's seriously awesome does the touch screen work in linux oh wow i'm rocky yeah everything works perfectly right from the install i didn't even had to make any bios changes well until i threw my own kernel in there i didn't even have to make any bios changes nothing everything just worked so have you found yourself

going and poking your other laptops as you're trying to work on them you forget that they're not all touchscreen you know how many times i've done that oh yeah i've been sitting in meetings with people who are used to using the um the microsoft surface and they'll have somebody else's machine they're sitting there poking people's monitors and knocking them over hilarious uh so we got a question uh how's the validity of a runtime kernel module validated as part of secure boot does the kernel verify them on insertion that's a great question uh i'm not 100 sure and i can circle back with uh our folks

that are more well versed in the specifics of how that works uh to get a great answer but i you know just kind of spitballing off the top of my head i would imagine that they they would have to be verified on insertion um and they would often i believe how it would work if you try to load one that isn't signed um while it's running it would essentially take the kernel in and cause possibly a recoverable error to come back but i'll have to try that out and um i think we can certainly get an answer for exactly how the um run type modules

are validated sorry i might be able to get you neil i may be able to get you a malicious kernel module or two if you want you want it to have fun sounds fun so when we start talking about and looking at rocky security i think it's important to kind of talk about from a community standpoint what is rocky security like um is there anything special that we do around that yeah uh security is something that we were very focused on from the very beginning while building out rocky linux um over a year ago now which is which is crazy um but it was foundational

to a lot of the questions uh or the answers to the questions that we were asking about how we were going to design access to systems and uh who is has access to keys for example for uh secure boot and um lots of other things that we are doing not necessarily different than anyone else but uh in what we feel is a the best practices in in the current age where we are with practice of least privilege and uh um only giving people access to the things that they absolutely need to have access to uh and that's something that we believe is also coming from

the transparency side of what the organization is doing um to that end we have you know all of our security updates that we're grabbing um are coming from rss feeds from uh upstream and notifications and we actually have a coming site that's gonna be tracking when those packages are imported and released to the the public mirrors uh so we can see you know when they're released in red hat and when they're coming out when they were imported into rocky's git and then when we're releasing them as well um as well as you know we're in the past few months we've gotten some more integration around

tooling that's that's commonly used in in the enterprise environment such as nessus for scanning uh as well as we recently had a sys workbench our assist community created around rocky linux so that we can get a workbench uh in for for validating the center for internet security's benchmarks across rocky linux as well i can jump in on that too because we also have so we've been working on fips for example so ciq is funding the the development and certification of fips 140-3 for rocky linux 8.5 that is currently underway and that will give various organizations i know federal is really interested in this uh it'll

give them the uh the confidence with our uh crypto implementation implement implementations there we go on uh what we've what we've what we've done with rocky linux and and that will be available as well um here fairly coming up coming up fairly soon there's one more aspect that i'd like to just kind of touch on real quick which is if you look at most of the critical cves that are out there one of the most common mechanisms or vectors to exploit hosts and applications is buffer overflows and in a buffer overflow attack basically what they're doing is they're taking a some sort of input data

or input stream and they're overloading a buffer that was not being properly checked within the source code and they overload that buffer and they overload it to the point where they know another function can basically come in and actually execute portion of that memory stream thinking that it's the function or uh or another piece of that software so they they basically buffer overflow it and then they put in a malicious piece of code as part of that buffer overflow uh so this is one of the most common ways that that hackers and malicious users are using to to basically run malicious code on systems we're

working with a a technology right now that was originally designed for embedded systems but we're applying this to the entire operating system and what will it will allow us to do is create a moving target defense of that memory segment of that memory of the application so basically at a high level view it will randomize all of those different entry points for arrays for variables for functions and whatnot within the um basically when the application is started it will it will randomize that which will basically it won't stop a buffer overflow uh but it will pretty much make them completely useless for you utilizing a

buffer overflow to run malicious code through through the application and we're going to be applying that to the entire operating system what's really cool about it is it changes the api it doesn't change the api at all doesn't change the abi and it doesn't add any performance impact to running this uh basically what it is is some build chain modifications that we're making on our side um and we're going to be adding value on top of the community open rocky and basically ciq is going to be offering hardened packages and hardened um capabilities on top of rocky that's going to basically mitigate and block these

buffer overflow attacks which if i were to take a guess is probably you know forty to fifty percent of all of the high severity uh cves uh that you have so this means that we're going to be able to block past present and future um exploits uh moving forward so we're really excited about this uh and again this is moving target defense so great you said one thing that's it's important is we do it for the operating system but is there the ability is a software to develop as a software developer can i extend that on to my software as well yes yes so um

we're figuring out how best to do that because that's part of our build tool chain that we're going to be using for building some of these packages and yeah we will figure out how to make that happen um it's probably going to be you know licensing of that tool chain uh to customers who want to you know compile their software with that tool chain um the other thing i actually i i didn't mention but i think it's important is this entire moving defense model exists in run space or user space so it's a run time feature it's not a kernel feature what this means is

we can actually apply it to a container and wherever that container goes and moves that container will also have this moving target defense model within it so it's it's really is it's super powerful um it's very easy to move around and do different things with uh and we're we're really looking forward to bringing this out to market and we should be people should be seeing this um in q2 so pretty soon awesome so now we talk a lot about the rocky build system and how it's different than some of the others how we're trying to make it more open but how how does the build

system we have offer additional trust to the resulting packages that come out that make up rocky linux sure yeah and uh so just to clarify we currently are using the rocky is currently using the fedora build tools so koji module build service those sorts of artifacts to build the sources for centos 8 and those tools are awesome they they take a lot of time to wrap your head around but once you get them up and running and are running them in an environment um they work great what we found when we were learning about these tools is that um they didn't integrate great into the

tool chains that we wanted to use uh and wanted to try and integrate into our stacks in a more cloud-native way um but one of the reasons that a lot of the tools like ogmps work really well is because the folks that are developing them are the same ones that are using them and they have the um the expertise and the knowledge to to make those services and tooling work really well and they have it all built into their existing stacks and services and so um what resulted from all of this was essentially a build system that we are calling peridot which mustafa one of

the software engineers here and one of the community members at uh rocky linux did a presentation on at fosdem a couple of weeks ago i think we can link to that probably uh in the description um and one of the things that we're really just trying to do with the entire build system including with peridot and the the previous build system which will continue to be used for for the eight dot um x train for rocky is provide better transparency and more transparency into everything that goes on from the sources that we're ingesting upstream from git does on dos.org to the artifacts that we publish

down at dl.rockylinux.org we want that pipeline and that process to be fully transparent energy for you to be able to validate the entire stack down from what artifact was pulled what patches were applied where it went after that and have a softer build materials essentially for the operating system and peridot allows us to integrate plugins that we can add things like security and we can integrate with ci cd pipelines and notify people and it's a very extensible operator a very sensible platform that will enable us to be more transparent and hopefully enable better security for us in the long run so peridot's really going to

be for rocky 9 moving forward right that's what we're targeting right now as well as for special interest groups that might be looking at wanting to add value on top of rocky and release packages there okay what about errata how do we handle that so errata's part of uh um kind of the same system essentially we are we're pulling those sources from upstream and validating them and and reissuing them as rocky linux security updates to rada.rockylinux.org and so you can go to rock that website and see a list of the the notification or the security vulnerabilities that have been released and what their ids are

and all of that as well as um has an api with a version two coming soon with some fixes um some bug fixes that allows anyone including security vendors like nessus and tennable to integrate with that to find out about the security build security vulnerabilities as they're released and i believe the v2 of that api is also going to come with an rss feed so that users can subscribe to that if they want to get notifications very nice so i know like a lot of people uh i just run yum update move along don't really pay attention but what's the best way to actually view

what what is available for security patches without just blindly doing what i do that is a great question so i think they're the probably the best way is uh to not do what i do well to not do it well so right this there's a few bunch of different ways there is the router.rockylinux.org website um has the api as i mentioned there's going to be an rss feed coming for that um the second way would be the rocket linux mailing list we send out updates as soon as we publish up uh publish images to the mirrors um and they can start pulling them down so

that happens maybe 24 to 48 hours after you release a patch it can take to circulate to all the mirrors in the network but it'll be available right then when we send that email um and you can it will be available right when you send that email on download.rockylinks.org and you can pull it from there and you can subscribe for notifications about those emails at lists.resf.org and we have a list there called rocky announced that has announcements made about all of the the mailing lists or all of the packages and any uh updates that might be coming in so so there's two ways in the

third way i have a short little demo here on a a container that you can use dnf to see exactly what updates you need on your system right now and i had this conversation earlier i can't get used to dnf yet i'm still quite young that's just the default it's the only way my brain works still let's see let me share my screen here while you're doing that zayn don't feel bad i know many very large enterprise sites that basically just do a a yum update in cron every night so oh boy yeah yeah so even professional big huge sites i know that are that

are doing that i'm i'm not suggesting that that's the best way of doing it um as a matter of fact i i would suggest the the opposite there there are probably better ways of doing it but uh don't feel bad very good and neil and i were talking earlier about living through going going through scanning for different different entities that need to do scanning in our pain through pci audits and faa audits fun stuff sure you've never been through one of those craig not an faa audit oh yeah that was fun every every year went through an faa audit that was those are interesting um

yeah my tenure at uh department of energy there was one point where we were also we had a training course on how to deal with um being um interrogated by intelligence officers that was fun nice um the one thing that i'll that i'll that i'll share for everybody here if anyone's curious um most of these most of these interrogations revolve around the idea of awkward silence so they'll ask the question you'll answer and then they'll keep staring at you as if they're waiting for something else and the amount of people that will just willingly give up and just keep talking and just keep sharing and

blabbing all sorts of information is actually how they get most of their info so get very used to awkward silences when you've answered you be quiet and you stare back at them i love it well i i got my screen shared that was a adventure as sometimes it can be to share screens on linux um but so i have a container here and thank you all for filling time i've got a container here running rocky alliance 8.

i just pulled this from docker hub we also have them available on quay.io o as well as i think for the amazon container registry and apples and uh docker hub images so this dnf update info command has a bunch of different options and one of them can just list out what the vulnerabilities are currently open for the system that you have installed and so it lists here the vulnerability id or the advisory id um what type it is so this one's these are moderate security vulnerabilities but uh in the future or there could be bugs that instead of security updates uh as well as the

name and this is the name version architecture just string something here and so there's a couple of different ways that we can install updates from here if you don't want to just see them the first of which is we can we can show information about all of those and get more information like the cve number and in fact let's just go ahead and grab that cv number we're gonna go dnf install cve and not install but update and then we're going to say yes and yes again and so that command there we took the cve number and passed it into dnf and it went and

grabbed that using the metadata that's signed on the download.rockylinux years and we can do that again with an advisory id if we wanted to so i know greg's laughing at the vemm one so i'll grab vim let's update film that's still my favorite text editor i can't convert too old is it still posix standard i believe i think it is too i also use yes it is it has been validated then three out of three senior webinar uh engineers choose them so as well as doing it with the the cv number you can also pass an advisory number and advisories as you can see here

have multiple packages associated with them if you had to pull in any of one of these they'd have to be pulled in anyways dependencies but the final way that you can just update your system and make sure it has all the security builder abilities patched is just saying be enough to update dash dash security and so that's going to go ahead and install the other two advisories that were available and now our system if we try to run that again dnf update.js security should say nothing to do and likewise if we do it dnf update info it is empty so we can see all of

the security vulnerabilities that are patched on our system bypassing dnf update info dash dash list dash dash all and so that is at least three ways to update make sure your system has all the rocky linux uh security updates that's better than the way i do it that's awesome thanks for sharing and one thing one thing i can go back and just touch on is um peridot the build system that we're working on actually incorporates all of this as part of that build system so it is it is uh slated to be open source here very soon if it's not already um and this means

that organizations can actually leverage that and they can actually even have so they can maintain some of this even themselves if they wish to absolutely um collaboration and people using the system to help them accomplish the the goals that they want to accomplish is is very much one of the principles we want out of it all right so we talk about sharing we talk about being open what's keeping a contributor from putting something into this that is malicious i think the the biggest thing that prevents malicious acts or malicious packages or really any sort of wrongdoing is that same transparency because all of the builds

are auditable you can see exactly what went into them you can get the artifacts we signed everything with our gpg keys there's it's obviously not impossible security is a it's a net not something that you can block completely um it would be i think difficult to to get something in we use prs as i mentioned before we uh take security very seriously in terms of not giving people access to things that they don't need access to that comes down to sanity around admin accounts and making sure that people aren't using admin accounts for daily drivers and um having approvals on code being merged in that

will affect the building of packages and everything else so we took a kind of hint from fedora and other projects like um debian open infra and other uh other linux distributions with how they perform transparency and mentorship and the auditing of resources that come out of there and um so yeah uh the in short no someone couldn't inject a nasty something into a package um without some alarms bells being raised and and without uh hopefully someone taking a a notice but again humans are are infallible and that's part of i think a bigger discussion in general um we saw that last year with uh some

commits that were being made as part of a research project to the linux kernel you know so um that's another challenge as a distribution of packages as well is you have to not only worry about the safety and security of your distribution but also what's going into it too i was going to bring up that point as well this is this is this is the same for pretty much every open source project that exists as long as you have a process for how are you verifying and maintaining every bit of code that's coming into the project through uh transparency um auditing you know merge requests

prs how whatever your policy is however you're bringing that in and also keeping accountable in terms of who are the people that are contributing who are the people that have commit access and those people that have commit access really you know how do you make sure that you have multiple people doing reviews all of these sorts of processes are designed to not only facilitate good code contributions to the software but also to catch if somebody is doing something malicious and to give people trust over time in such a way that lots of people can be part of the project and that's the goal is to

increase you know the the breadth of people um who want to be part of that project but also how are you managing and mitigating security risks to that so what i want to mention is this is not specific to uh to rocky linux it's it's it's pervasive throughout all of open source and we're doing the same sort of things that every open source project would do in terms of that transparency um auditing and and so on and so forth and the infrastructure i think is really designed in a in a very smart way um putting security as being one of the primary tenants in terms

of what we're doing and the rocky project itself has a very active security team within the project and so not only is infrastructure development contributors and whatnot thinking of security as a kind of number one facet to the project but we also have a number of people that have a high level of a high degree of say regarding that infrastructure and regarding how are we doing things and how are we creating policies around security so security is definitely kind of one of our number one priorities um as is community and and and you know creating a very large diverse inclusive you know open source project

and then balancing the two and coming up with that infrastructure necessary to be able to do that to be able to allow um contributions coming from you know even casual contributors uh and and bringing it in and including it into the project in a way that does not uh risk any sort of security um any security issues it does not bring up any security issues so that's really what we've been doing um you know as neil mentioned before uh the infrastructure that we started off building uh is is tremendously awesome um based on the fedora infrastructure koji um and and uh all of the other

pieces of that um coming from the fedora team they've done a remarkable uh job pulling all that together and creating a platform for building operating systems our uh next stab at that uh again peridot is basically taking that and turning that into a cloud native architecture where i think we're going to be able to get even more transparency and more auditability into that environment um and those and basically persisting every every artifact that we can out of that making that publicly accessible uh is again i think gonna be uh hugely important and again neil said this but just to reiterate that software bill of materials

the s-bomb is is really critical and we've actually gotten a number of um large organizations that are very security focused um asking for that and and driving for that so that's something that's a huge priority on our our shoulders very good i guess i feel like we should just sit here and stop and just do a long pause and see if people start asking questions kind of do the silence right yeah the officer exactly if you guys have any questions please shoot them over we'd love to love to answer them we've got some time here um if not we can make greg tell us more

stories come on you gotta have more security stories craig oh i've got a i've got a really good one um but only only if nobody asks any questions and i hope people ask questions because it's more fun to answer questions and tell stories none yet let's have a story i love it i don't sneeze again sorry about that so uh i i i may get in trouble um uh for this one the stories then so uh a couple a couple people from the from the rocky team and and others uh have heard this story before but um when when i was at uh doe we

had a intern that um would always leave his terminal logged in and it would leave his computer and we have given we've given this person you know lots of crap saying you you gotta lock your screen when you walk away this is a government facility you gotta lock your screen and he just wouldn't do it and it just it was it wasn't he wasn't trying to be malicious or it just it wasn't in his he just wasn't used to doing it so he'd walk away so one day he walked away and i come over to his computer and i see not only is it logged

in but he's got a root terminal open and so i quickly change his password and root's password to the system and then i lock the screen and i walk away and i go back to my desk in my office and his desk was outside outside my office and so um i hear him come back and i'm just kind of listening and i hear him trying i can hear him getting frustrated uh getting concerned and so after a little bit of struggling i i go out there and i say what's going on he says i don't know my computer's not working i can't i can't get

into it i can't log in and i said hmm you want me to try yes what are you going to do and i said let me let me just let me just try it real quick and see what i can do so i'm sitting at his login prompt and type something hit enter and it logs in and he's looking at me like what did you just do i said oh no no it's no worries i said i put a back door into every sento system in the world and and then i said yeah there you go cool now make sure you make sure you fix

your password and um and so i walked away apparently this person contacted security team and told them that i have put a back door into every center system in the world and um yeah that did not go over very well with the security group who then very quickly came into my office asking me all sorts of questions um and i was able to use that awkward silence thing to do my um uh to my benefit and just kind of maintain that but um it was uh it was it was a little nerve-wracking few few minutes and then i shared with them what i did and

that i did not in fact put a back door into every sento system in the world um but yeah yeah so um everybody lock lock your screen terminal lock your screens especially if i'm around rule that's the only word i have but this person learned the lesson yeah they did but they never walked away with it unlocked again so you have a question that came in asking us to repeat the fips compliance status uh in terms of status that is currently in progress the process of doing fips certification is is quite lengthy especially considering that we just went from 140-2 to 140-3 and there were

some changes that are going to be necessary as part of that compliance excuse me and we're expecting that we're going to have to be making some code changes um to get that cert that certification so um at best we're looking at probably another six months at worst we're probably looking at another um i'd say 10 months it's it's really hard to say we do have an organization that we're working very closely with on that um and it's in progress uh the goal is again it's going to um be for rocky 8.5 and um yeah i think that's the that's the high level gist of it

once we get that fips compliance we're also going to go after additional security accreditations that are dependent upon that fips compliance then what we're also going to be doing is having a specific point releases which we are going to target for updates on fips compliance so if probably we won't have a fips compliance on eight five eight six eight seven and so on and so forth it'll probably be eight five and then it may be eight seven we may skip one and then it may be eight nine um we're gonna have to wait and see kind of how long it takes and and um what

the lay of the land is at that point but the goal is to um have multiple versions as you may know fips compliance is a is a certification of a particular uh operating system patch set or release set so once you've updated those rpms those rpms are no longer compliant because it come it it validates a particular binary release of a package so uh we will have to continue doing fips updates long term and we aren't planning on doing that very nice i think one of the other questions that i've gotten asked quite a bit lately as people move to rocky 8 5 is when

can they start playing with a beta of rocky 9 um question i get a lot we've actually i i've gotten that one as well too um we we we are currently in the process of of building and testing um packages and build processes as as i think neil mentioned and i'll let neil give more definite and concrete information if there is any uh to be shared but we are doing this in conjunction with the new build system so we kind of have two things that we're doing simultaneously and the goal is again once once nine is out it'll be all built on the new build

system as well so um that's that's delaying the beta a little bit but we don't think that's going to delay the release at all as a matter of fact we're hoping to have the release um as soon as absolutely possible uh neil i'll hand it over to you to give more uh concrete information online sure pause awkward pause for fury well we we we're in the middle of planning it uh we're not planning but executing and planning uh a lot of it can be followed along on uh the matter most chat for rocky at chat.rockrockylinux.org there's a like planning board essentially in the development channel

that we're tracking the tasks for um rocky 9 and kind of the roadmap that's going on there excellent thank you jonathan has a question about it's more about fips uh asking about rocky linux and nist 800 171 or cmmc environments are those mostly separate concerns um you caught me i'm literally googling those right now um i'm going to need a little have a little bit more time but i will get back to you jonathan on that i don't have an answer for you specifically and google was it's not fast enough for me so or i was not fast enough in reading it but i'll get

back to you on that jonathan yeah we've got two questions we get leo jonathan answers back on perfect well guys if we don't have any more questions we are pretty much up on time so i appreciate you guys coming again uh like and subscribe stay in touch with us if you have any other questions feel free to reach out hit us up let us know what you're thinking we appreciate it thanks for the time thanks greg thanks neil i'm glad you didn't ask me for another story zane out of time sorry thanks everybody guys thank you