Rocky Linux Solutions

Webinar Synopsis:

• Repo as a Service 

• Using Packages 

• CIQ Secure Linux 

• CIS Benchmarking 

• Kernel Versions and RHEL Variance in Rocky

• Imaging as a Service - How To Handle Patching 

• Progress of FIPS 140-3

Speakers:

• Zane Hamilton, Director of Sales Engineering, CIQ

  • Linkedin

• Neil Hanlon, Solutions Architect, CIQ

  • Linkedin

• Robert Adolph, Chief Product Officer & Co-Founder, CIQ

  • Linkedin

---

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Hello everyone, and welcome back to another webcast with CIQ. Today we are going to be talking more about Rocky Linux. I think we have alluded to some of this in the past, but we are going to be diving in on some of the other things that we are doing around Rocky Linux. Again, appreciate your time, thanks for joining us. If you would go ahead and like and subscribe so you can keep up with us and also leave a comment, let us know who you are and where you are. We would like to make sure that we keep up with you and welcome you when you come back. Today, I have Neil and Robert here with me.

Let's talk some more about Rocky, I know we kind of dived into some of these topics. Robert, I know you have talked some about repo as a service in the past. I don't know that we have ever actually really dug in and talked about what those are going to be and how those are going to look. Let's go ahead and start with repo as a service and talk about what it is and what we are trying to do with it.

Repo as a Service [00:58]

Robert Adolph:

I will let Neil get into the specifics, but at the higher level, what we are really trying to do is find strategies and solutions that can help customers at scale maintain their environments and do their environments better. Over time, we’re going to have multiple different strategies to help enhance that capability. That is the overall goal of what we are doing with repo. As far as the nuts and bolts, I will let Neil maybe jump in and talk through that. But at the end of the day, our goal is to help people long-term utilize Rocky at its fullest.

Neil Hanlon:

The idea of Rocky or repo as a service here is it comes from a lot of experience, investigation, and analysis of a lot of things that are out there and available in the Enterprise Linux space for managing repositories. A common theme among that management is the need to have customized repositories, possibly with different packages that come from various sources. Also, just packages that are updated and you want to have a copy of them locally to ensure they’re valid and the right packages, and your systems are not coming out to the internet to a place that could possibly be a less secure site, like any mirror or anything else. This ensures the content you are getting is going to be not only secure but up to date, and are the current images or packages from Rocky Linux. It will enable customization around that to fit your business's needs, which you want to pin packages at specific levels in various environments and scale that out from an enterprise solutions level to manage your fleet of systems.

Using Packages [02:56]

Zane Hamilton: 

You are going to be able to actually, like you said, pin environments to the way they are. I know in the past, when I have done this, it really depended on where you had the management interface set up as to what package you were getting. A lot of times it was really difficult to even know, are they the right packages? Is this something that is going to help me with that problem?

Neil Hanlon: 

Yes, it is something that we are aiming to use and provide. That your systems can subscribe to them and you can manage them from a central location, which gets you different plugins that might be available and whatever customizations you might need for your business requirements. We are trying to get those solutions and optimize this and improve it as it comes for whatever business needs there are.

Zane Hamilton:

Am I going to be able to put my own packages or my own software in this as a part of that repo?

Neil Hanlon: 

Yes. I think Robert might be able to talk a bit more about that, but at least not in specifics, but at a high level of how that might be managed.

Robert Adolph: 

Yes, we will have multiple different ways to potentially do that. But I think we can deliver from a cloud environment in a very secure way down to your environment. Once it's in your environment, now you can do additional geographic capabilities, etc., from there and/or have multiple different ways in which you can scale that internally. Obviously, we are laying the framework in the building blocks for potentially some patching and some imaging down the road with the same trusted resource that you would be able to utilize. And Neil nailed it, I think: potentially multiple different build processes inside of an environment that is hardened specifically the way in which you want it hardened in order to execute the way in which you want to execute for your environment. It just gives you flexibility, scale, and the ability to do it in a systematic way.

Neil Hanlon:

Some of the other things that we are talking about as we grow and add on to this platform is the ability to have customized images and customized containers for your software that are installing maybe some custom applications on top or whatever that might be. Also serving those out of a similar service enables you to validate the software chain as it comes down the line from packages produced at Red Hat and are imported and sent down the distribution line as it goes into making the operating system. That is something we are focusing on to be able to provide transparency in the build process, which is crucially important, especially today with all the supply chain attacks and everything else that is going on.

Zane Hamilton: 

Absolutely. Is there something you could go ahead and show us on that, Neil?

Neil Hanlon: 

Yes, absolutely. One of the things CIQ is working on is CIQ secure Linux. Robert, why don't you give the details there while I get the demo set up here.

CIQ Secure Linux [06:23]

Robert Adolph:

Yes, that is a great question. It is a partnership we have with another company in a commercial capacity and their code is commercial. It is not available in the open source community. However, it does solve and address some of the security requirements a lot of our customers are asking for: protection in the memory space and protection in real-time access to the environment. It is a partnership that theoretically can block 40-50% of the CVEs we see come in in real time and give you the ability to really patch and manage your environment in a better way. It is definitely a commercial offering. It is definitely a solution that needs to be built into the build process in order to execute well, and that is where Neil will show you what that looks like.

Neil Hanlon:

I have a container here that is running Rocky Linux. We have pre-installed a tool called CIQ. This is the entry point to repo as a service. From here, the first thing you have to do is enroll the system to repo as a service. That would be done throughout a provisioning process in the environment, or otherwise set up in a way that is not a barrier for administrators to use the system and enroll their systems into repo as a service. We will have multiple provisioning options and ways to get this tool and get your systems enrolled into a particular site. The way we do that is they just do a CIQ enroll. We provide a site token. This will be a customized thing for the product per environment. 

You can even think of a site token as potentially a way to segment out which environments are getting what packages. In a way, it’s a subscription to get access to all of the repo as a service features. We can see all the things we have access to by just typing CIQ list. We currently have access with that site token to a product essentially called ‘secure8.’ This is the CIQ secure Linux that Robert was just talking about. The next step after here is we have to enroll and we have to enable that key. We can type CIQ enable secure8, which will go and talk to repo as a service. It will validate your subscription and modify the system by, in this case, installing a package and enabling the subscription in the package manager. The best way to look at this first is to look at the linked libraries for a couple of packages. We can look at Bash, we can also look at libcrypto. 

In both of these, this is just the standard linkages of the packages as you would get them from Rocky Linux. There is just the standard linkings here, so now that we have enrolled in the site and we have enabled the secure8 repository, we can do a DNF distro-sync. That is going to go ahead and talk to the repositories of repo as a service and download the packages that need to be replaced because they have been updated and overridden by that repository using priorities to overwrite the packages as required. 

We can see that here in the disk tag on the package, we have an el8_5_ciq_secure. Now if we go back and we look at what pack or what is being linked into Bash, we see liblfr here. By having liblfr linked is what enables Alkemist to run safe and CIQ secure Linux to do its work here and to provide the security aspects. We can also see that in libcrypto, and if we quickly go and install NGINX, which we have in this repository built with CIQ secure Linux. Some of these packages are coming from Rocky Linux repository and some of them are coming from CIQ secure Linux. We are able to merge those packages together and structure them in a way that enables CIQ secure Linux to work. It also keeps any updates that are coming down upstream from Rocky Linux enabled on here as well. Any updates for packages that are built with customizations, such as CIQ secure Linux would be updated as soon as they are available and made available into these repositories for all customers.

Zane Hamilton:

This is interesting. I have not seen it done this way, so I am extremely interested now. I can take an existing Rocky install, and if I am  a CIQ subscriber, I can actually go against this repository and make that already existing distro have this secure version running on it without having to do anything else, other than just subscribe it, update the packages, with no rebuild. I don't have to reinstall the kernel, nothing.

Neil Hanlon:

Right. Exactly.

Zane Hamilton:

Awesome.

Robert Adolph:

Yes. The other thing that is big, and a lot of folks have asked us for is what you are seeing here is our friends at RunSafe, the build there’s specifically for them, but in the same repository, you could also have open source Rocky. You could have five other variations of that for your specific applications and environment. Then you can have this on-prem with a mirror from our solution that we maintain for you and do it as a service and provide you now additional layers, which you can go do some interesting stuff in your environment for yourself.

Zane Hamilton: 

Excellent, thank you. Oh, Greg is jumping in here: “This is what enables us to block buffer overflow based attacks in the secured versions of these packages. So awesome and love the Repo as a Service functionality!” This goes back to being able to block the buffer overflows like Robert was talking about earlier for, I think, 40 to 50% of in memory attacks, which are attack vectors that come in, this is what gets blocked. It is all those shared libraries within Rocky. Thanks, Greg. One of the other things I wanted to ask you about Neil is the CIS benchmarking. I know it's something that comes up quite a bit. We have been talking about it internally for a while. Where are we with that? What does it look like?

CIS Benchmarking [13:52]

Neil Hanlon:

Yes, the benchmark actually was approved, I think just the other day for Rocky Linux 8.5. That is, I believe available. I have been checking into it a little bit because they have not produced what looks like a finalized PDF. But the status has changed to approved, so it looks like the benchmark from that perspective is complete.

Robert Adolph:

Neil, in your mind, how does that play into the repo as well? And how can folks take advantage of something like that?

Neil Hanlon: 

That is an interesting thing that could be implemented as well into repo as a service to help with free hardened images and infrastructure that are set up to meet certain security standards. So you can think, CIS 1, 2, all the different benchmarks from the US government as well as from abroad and from just independent organizations. All of those sorts of variants are possible to be created and delivered as part of this.

Zane Hamilton: 

What you are suggesting is to have a different repo for each one of those, obviously you have a collection of them, but if you have a specific thing you need, you have a repo that goes that route. If I had a PCI environment, I’d have MATE in one place and I’d have… pick another… you’d have several repos for each.

Neil Hanlon: 

It is a combination of both things because many of these security profiles are applied by specific tools that go and run and remove packages. One thing we can do with repo as a service is create crafted repositories, which are already lacking the packages, for example, that are excluded by certain benchmarks. Not only are the packages not installed, but also cannot be installed because they are just not available there. That is one way that can be delivered to lock down certain environments, as you mentioned, like PCI DSS, CIS.

Zane Hamilton:

Excellent. Thank you. We have already hit on the RunSafe and security. We have also talked a lot about kernel versions. That question comes up quite often. I know if we look at the kernel version in Rocky and the RHEL variance, it can be a little bit older. What are we looking at there? What are we doing?

Kernel Versions and RHEL Variance In Rocky [16:24]

Robert Adolph: 

Yes, absolutely. Obviously, Rocky Linux is always going to be bit for bit with Enterprise Linux. There was always going to be that solution available for the community. Also, what we want is to do additional enhancements to that existing capability. In that example you just gave, we would like to move toward having a mainline stable kernel variant of Rocky Linux, for example, and potentially long-term stable kernel variants as well. Then there is also going to be, potentially, a universal-based image for containers. There are also multiple different types of variants, as an example, that customers and people in the community have been asking us for. This is somewhere CIQ wants to make an investment and also provide additional capabilities to the community long-term.

Zane Hamilton:

You can also plug that back into the repo as a service, right? Specific customers want specific kernels, so as it goes to the build process, we can put it in the right place?

Neil Hanlon:

Yes, absolutely.

Zane Hamilton:

Excellent. When we talk about supportability, whenever I see the word enroll, it reminds me  of that old way of, I have to enroll every node and it becomes painful because then I think you're counting. One of the things that really interested me and excited me about CIQ was the model for support. It is not like most places. Instead of having to do a model where you're having to count and keep track, we actually have the option to support people. I know Robert has been passionate about that the whole time. We do not want to support nodes or host, we want to support people. It is his mantra. That is  the route we have gone. I will let you talk about that more Robert, if you would like to.

Robert Adolph:

You know, at the end of the day we believe if we help the people that are managing the environments be successful and we are working specifically with them to be successful, then we will be successful and so will they. I actually do not really care necessarily about how big or how complex or what that environment looks like. Our model is designed to support those different folks at the different levels of the support that they need them. It gives us and them the freedom to then architect something based on what they need and what they want, instead of having to worry about their licensing model and how that is going to affect what they are doing. It also gives them a lot more flexibility in how they want to go about patching, or flexibility around imaging, and  flexibility about whether I want to put this workload into the cloud or do I want it on-prem? We don't want to hamstrung them in any way when it comes to that so our model is based on our ethos of how we want to support people. It is why and how we want to interact with the open source communities that we are involved in. It is about the people and the communities and the goals and outcomes, which they are all trying to produce.

Neil Hanlon:

Yes, we saw the enrollment there with the tokens. I admit the first time I was seeing us implement that I was like, I do not know, but it is something that is evolving and continues to evolve as we work with customers to fit those needs. We never want someone not to use the product because it is a barrier. We never want to have that barrier of entry for a product for our customers.

Robert Adolph:

To Neil's point, it also gives us the ability to really engage with commercial entities, like RunSafe and offer their enhancements to the community in a way which can then satisfy them as well. Again, we love our open source communities. We love interacting and supporting them, but there are commercial entities out there that can add value back to the open source. This does give us a mechanism to help them interact with the community as a whole. I hope that makes sense.

Zane Hamilton: 

It does. The other part that I probably forgot to say is we can also do that more, what I like to call the legacy model of, if it makes sense to count nodes for you, we can do that. It follows the same path and the same levels of advanced standard basic, but it can be for those per node. We just think that most people are going to be interested in a per person model that gives you that ability to grow without having to count. That is always something that has been painful. I have lived through those many times where you are trying to figure out right before renewal, what do you have? And then you end up realizing you grew by 500 servers and, oh my gosh, who is going to pay for that. This  enables you to go be successful.

Neil Hanlon:

For CI environments, testing environments, all that sort of thing, where you want to be able to have the flexibility of spinning things up and down.

Zane Hamilton:

Sure. The other part about this is not every person that subscribes has to be at the same level. If you have a team of people, maybe you have some junior engineers that have a different requirement or a different amount of involvement in the process, they can subscribe at a different level, not necessarily that 24 by 7 level, that you might have your higher level architects working in. It is a very flexible model, to me, it is very exciting. I think it is going to change the way people look at doing software support. Anything specific you want to talk about on this one, Robert, or anything else?

Robert Adolph: 

The only thing I would add is, and I've given an example, which is probably the best way to explain it. We have had many organizations come to us in the test dev situation for Rocky, for example. Just to be clear, this model applies to Singularity, now Apptainer, and Warewulf as well. There is no difference in the way in which we approach any of the open source communities we are involved in. It actually is also how we are going to approach our next generation high performance computing platform. Again, always focused on the people that we are supporting. However, just keep in the back of your mind that in a test dev environment, maybe you just need one or two people in the beginning. Maybe you just need a standard instead of an advance in the beginning. Then as we prove out the applications being successful, as we prove out the solution can now be put into production, now you maybe beef up the support staff that is actually engaged there. Maybe, we also then have a couple licenses for your development team as they develop your application further. The model gives us the ability to really help different organizations inside your organization in different ways that maybe a traditional per node model would not. However, like you said, we can do a per node or per instance, very simplified per node, per instance model, or by the person that is needing that support. Does that make sense?

Zane Hamilton: 

Absolutely. Or by the team? I think I forgot to mention by the team. If you have a team of people, like I said, we can mix and match however we need to, but we can also subscribe just by the team. One of the other things that, Robert, you and I have gone back and forth many times on is the topic of CentOS 7, since it had a longer runway than 8. I think it is an important topic; we get asked about it a lot. What do I do with that CentOS 7 environment? How do I move if I am looking for support; how do I deal with that environment? I think it is exciting and interesting that we all agreed to what we are going to do with that. I would love for you to tell us what we are going to do?

Robert Adolph: 

Yes, it is a great question. We have been asked over and over again to support CentOS 7. We have been asked even to help potentially create a Rocky 7, and we are interested in both to be clear. But in the immediate future, we definitely want to support our current customers and folks that need it with their CentOS 7 environments, which is actually included in our support model now, so not just Rocky support and the variance of Rocky. To be clear, every and any variant of Rocky that will be in the community or for different customers will be included in our model as well. We are not segregating out the different variants, and CentOS 7 is obviously a first class citizen in that now.

Zane Hamilton: 

I just want to make sure I clarify, if you want to come to us and do a subscription, like a per person model subscription to cover Rocky Linux, that also would cover your CentOS 7 environment? It is not like I am having to do one for CentOS 7, one for Rocky. You get a team supported by CIQ, you are getting supported for CentOS 7 and Rocky 8 plus.

Neil Hanlon:

Enterprise Linux. Yes.

Zane Hamilton:

Perfect. I know there are other things, Neil, you alluded to earlier, that are coming as a part of what we are doing, especially with repo as a service. Imaging always comes up. We talk about imaging as a service. The other question we get asked quite often is how do we handle patching? What is coming for those types of solutions?

Imaging as a Service - How to Handle Patching [26:27]

Neil Hanlon: 

Right. You can imagine full end-to-end security advisories and integrations with chat tools and whatever else to enable those workflows so that administrators are able to help update their environments and see what is going on in their environments from a single pane of glass. A centralized workflow allows you to see your environment as you want to see it, group it how you want to group it, and in stages update and push them through as they make sense for you and for your business. There are a trillion different ways organizations update systems and have systems siloed from one another. They may have dev test environments. They may have several dev test environments that might need different requirements for different packages levels and security versions. 

But specifically with the security stuff, enabling all of those systems from a top down model to say, I don't really care about whatever package versions are in these specific groups. I always want to make sure that we are not vulnerable to CVEs because the security team wants me to patch these. Also, just working with customers from a customer user perspective to identify those CVEs and look at what risks they do impose, or don't impose. Maybe there is a false positive or something like that. All the things down the line from there. As well as on imaging: we are super hyper-focused on security and trying to make sure where the packages come from, where the images come from, that create an image you are using, whether it is booting into a cloud environment or using it on a container in a Kubernetes environment or a cluster. Growing the service to provide those sorts of service repositories and container registries with those customizations as well so that they can be built on the fly and given to you in this environment. 

If you're running Kubernetes, for example, you can just go ahead and schedule your deployments to grab the latest image and pull it down. When you are pulling it down, you are ensuring that it is going to have the latest updates, which have the security patches available.

Robert Adolph:

Yes. The only thing I would add is, we do take security very seriously. Multiple cryptographic signatures over time, maybe a signature for a vulnerability scan at the repo level before you deploy a patch. There are multiple different ways in which we have strategies for those multiple signatures to ensure what you are putting out there is what you want to be put out there. That goes to universal base images for containers. That goes to all sorts of different capabilities that customers have been asking us for. But that software supply chain security we take extremely seriously. We even have a couple patents around that. At the end of the day, that is something that will be built into our products and our thought process as we deliver this and help others deliver to their servers.

Zane Hamilton:

That is great. Thanks, Robert. I know you saw these emails today asking on FIPS 140-3. I know it is in process, we have talked about it many times, but is there an update on where that is or is it still scheduled for later this year?

Progress of FIPS 140-3 [30:32]

Robert Adolph:

Yes. We are going through the process; we are engaged in the process; we have multiple updates that will have different timelines over the next six months to nine months. We will keep the community up to date at every step of progress. It is moving very well and we are enjoying the progress so far based on that.

Zane Hamilton:

Excellent, thank you. That was all the questions I had. I was hoping we would have questions from the community. I know there are several people watching, if you guys have questions go ahead and ask them. If we do not have any more questions, I will let Neil and Robert go. I appreciate you joining again. Go like, subscribe, let us know you are here. Reach out. If you have any more questions, we are always here to help you in this journey. Appreciate your time.