Maximizing Your GPUs with a Tuned OS with CIQ

At the OpenShift, at the Rancher Prime level. So, we were the first ones to define and build a bare-metal OS initially and specifically targeting production inference. You guys are probably aware with the stats that the prediction in early 2025 was, or the measurement was, that inference was costing three to four times as much as training. Training tends to be occasionally, but inference is constantly, and it scales with your success. And some predictions said that inference would outpace training cost by up to 17x by the time we get to 2030. So, what we built is RLC Pro AI.

Now, sort of bottom line up first. Some of the highlights of what we were able to recognize, and we'll get into this, is in certain situations, we were able to lay down the OS and get 32% more throughput. And I'm going to stop on this because it's a big number. Like, okay, you can get 32% more input. Thing is, every workload is different, every hardware profile is different. You may have a different bus, you may have different memory, you may have different processors, you may have different resource usage patterns. So, this is one of our most impressive numbers we got in a particular condition or situation, but it shows what the art of the possible is.

Yes, question. >> Processing what you're running in containers or out of containers. So, do you still get that throughput when you're running the container like a Kata container or some type of container? >> Yes, depending on the shape of the profile. And we'll talk about how the key >> run Rocky Linux on the Kata container then to get that performance? >> I'm not familiar with a Kata container. >> Well, it's a micro VM. >> Yes, okay. >> Micro VM. >> Okay, right. Kind of like Bottlerocket. Yes. >> Yeah. >> So, yes, we cannot, because the container, unless you're running bootc, you have an OCI compatible, we cannot tune the kernel.

We could do some things in the tuning of the stack in the user space, but tapping performance heading in this direction requires coordination between the containerized environment and the bare metal OS, which I'll talk about. >> That more throughput number is tokens per second and per second, prompts per second? >> This right here is tokens per second for image segmentation, and this is where we can get to be specific. It's tokens per second for image segmentation on Ampere generation hardware. >> So, it's MLPerf types of numbers thing. >> Yes, this is a slimmed-down version of the MLPerf. I just ran official open category MLPerf benchmarks a couple of weeks ago, but this is a little older when we did it.

We took MLPerf and we slimmed it down. What we also deliver is a conformal kernel advantage. We'll talk more about this. At the most extreme, it's an 18-month kernel advantage. What does that mean? That means that we are delivering a vendor-backed kernel that has up to 18 months of vendor feature commits and improvements in the kernel that won't make it to the traditional enterprise Linux. >> This is not a managed service sub basis more or less, right? I mean, no. How's the enterprise paying for this? >> How they pay on a per node basis. We sell a OF >> Per node per >> Is that a perpetual license?

What's the >> A per node per year. >> Per node per year. Okay. >> Per node per year. And we'll get more into this. And then we found, of course, happy path scripted, some certain people, in fact Tech Field Day people, when I brief them, told me that I've underestimated this number. But we found that you can move 9x, certain people I won't name now, 9x times faster on what is behind this number, because we won't, I think, go into detail, is that from a clean OS install, right? You've booted to an ISO or a QCOW or a VM to your first token being output from Ollama in this case.

We were able to do it in 3 minutes and 43 seconds when both upstream Rocky and Ubuntu took 30 minutes best case. And simply largely because, as we'll talk about, we pre-baked and tuned. But with the other OSs, you have to download and compile commercial modules, set your FS or adjust your grub parameters, and reboot. And then lastly is this, again, hybrid or kind of this heterogeneous, to continue to use that word, compute. We deliver a like environment for bare metal, on premise, Kubernetes, AWS, Azure, and GCP. All right. Yes. >> So, your definition of validated? Is it including the security hardening or >> In this case, validated is validating that it works together and there's no performance regression.

All right, because one of the things you hear a lot about, even in just getting an image up, is going to be the dependency doom loop. So, validated is just to say, we've taken this stack, we've took versions, we tuned them, we pinned those versions. And then we went through our test and validation suite to make sure it works, and we've benchmarked them to make sure there's no degradation. So, let's get into the details of what this is. So, RLC Pro AI is, as I said, vendor-backed Rocky Linux tuned for GPUs and performance. We've built it to support inference from first boot, and we'll go into detail on this, by taking pre-tuned validated frameworks.

So, when you get your GPUs and you have to stand them up, you don't have to spend time identifying the right drivers and the right stack configuration that for at least some of the lowest common denominator workflows and workloads, you can boot those. You can have PyTorch, TorchVision, your CUDA stack, and your GPU drivers all running and working. So, that's all NVIDIA language. And ROCm and the ROCm >> Okay, good. And is this, feel like I'm going to expose myself here, but why not? Is this auto discover, or are you clicking in a wizard to say, "Here's my GPUs"?

Like, how are you tuned? >> Currently, so, auto discover is on the plan. Currently, you have to get an AMD image or a NVIDIA image. Now, if you're standing up yourself, you also have access to both of those repos from the images with pre-compiled modules. So, you can pull them from the repo if you have a truly diverse fleet and you need to dynamically discover. And then what we do is it's you. Do you have PS for this sort of thing as well or something so that you all can do that for the >> Yes.

Okay. >> Yeah. All right. Which goes to just what I'm finding, a lot of people don't just want a product or a bit. Hit back to that AI factory thing. They want people to help them understand how to do it. And then, of course, what we do is part of that tuning is that you can set parameters when you compile CUDA. You can set parameters when you compile the GPU drivers to determine what the range of support is. So, we kind of slide that window. So, we look to support the last three generations and keep them performing. You hear a lot about NVIDIA and you hear a lot about NVIDIA authorized because we very early on, prior to AMD, were actually the first Linux distribution to go to NVIDIA and get rights to redistribute their CUDA stack and their GPU drivers.

And then just a key thing, like we've talked about, is one stack, same stack for every environment, really trying to capture and support this move from bare metal and cloud to sovereign on prem. We see that the key differentiators are is that with Ubuntu, we'll call out, you can right now go pull, I think, 24.04. I think you can get the 7.0 kernel. You can actually go get NVIDIA DGX OS or NVIDIA OS on your DGX. I think they just bumped that to the 7.0 kernel. And that thing performs. But there's a couple of things.

That's a Debian distribution, and 7.0 kernel has already been deprecated upstream. So, you're starting to build your production delivery on something that has no guarantee of support in the future. So, for us it's taking an upstream kernel that is ahead of everybody else, upstream long term latest stable, if you're familiar with kernel.org, and providing vendor backing to ensure that you can count on how long you'll be able to run production workloads on that. A kernel tuned for AI workloads, and we're continuing to tweak and tune and learn here. We're pretty transparent with the settings. They're actually their settings anybody can tune.

But what we've done is we've took the time to tune memory allocation, transparent huge pages, and to tune those in a way that they work with the stack. Yeah. >> Devil's advocate, I mean, you can use configuration management tools to do all that too, though, right? >> Like Puppet, Chef, Ansible. >> You can. Now, there's going to be a couple of things. There's going to be build time parameters, a couple of which we've tuned, which, sure, you can use any automation to drive your build time. Then there's going to be run time parameters. You could use automation to tune your run time parameters.

But again, then we're talking about the difference between I stand up a system, I run automation against it, I run it against it multiple times to kind of configure it, versus I lay something down. Yeah, from the start. And >> So, just to be clear, you're tuning the kernel parameters to optimize it for AI workloads. >> Yes, for inference workloads. And this particular case is our default. Then on top of that, and to your point for automation, this is what you would trigger. We actually have TuneD tuning profiles that can detect what your workload is or be triggered to be tuned for a specific workload.

But the other thing I say here is you are correct, but then when we go to validated stack, driver, CUDA toolkit, and framework, there are also key settings within the CUDA stack and the PyTorch stack that can unlock performance. All tunable, but most people aren't tuning them. One common thing we saw was TorchVision. When you're doing vision-based workloads, by default, and in fact we did this by accident once ourselves in a build, by default, finding people have it set to CPU. So, you're losing, I don't know what the order of magnitude is, but underneath this complex stack, when people are thinking at containers and they're not thinking about OS's, right?

Or they're just sort of outsourcing their stack, you can end up in a world where a small thing like a single binary flag on TorchVision means you're not even using your GPUs for that process. So, what does the validated stack look like? We're continuing to grow it. Right now it is PyTorch, TorchVision, Ollama, vLLM, and we have others in the works. And each of these are, as we said earlier, validated with a specific version or versions of the driver and kernel. We'll talk about that. In this case, right now on our 10.2 and our Rocky Linux 9, or Pro AI 9 and Pro AI 10.2, we are shipping CUDA 13.

And then on the NVIDIA drivers, we have, it's roughly what I'll call, an n plus two strategy. In other words, to hold for production stability, we ship the latest long-term data center driver, but then we also ship the latest feature branch and a version in between that. It'll come up later here that we discovered that on a DGX or on a GB architecture, we were able to see up to 9% improvement by running the 595 driver versus the 580. >> Not necessarily here now today, but do you have a flavor of this for our AMD friends in ROCm?

>> We do. >> CUDA-compatible, so like they'll, the driver side though, obviously. >> We already said it. >> We absolutely do. There're reasons we could discuss later as to kind of why it's not headlining here, but the short answer is yes. >> I think the reason it's not headlining here is obvious. But I would just say that the AMD customer tends to be on a much bigger scale, and a little bit more DIY oriented. So, I just want to support this. It makes a lot of sense. >> Right.

Yes. >> See, just, we're already in the world of the AI kit, which I hope you get back to soon, as opposed to the AI factory. That's a little more modest, >> Right. >> even if it's still significant in implementation. And so, that makes sense here. So, it's not a critique. I just wanted >> Oh, yeah. No. But I'd love to have this discussion, but I will say that the, so, short answer is yes. The reference that I'll have here that I am proud of is that AMD came to us to build Rocky Linux as in terms of what they distribute by default with their HPC clusters.

So, that's sort of the extent of the relationship, and then actually, or not the extent and an indication of the relationship, and sort of speaks to the size of workloads that running a lot of HPC. So, what are we looking to solve with the validated stack? No post-installation driver assembly, which could get you in the dependency doom loop when you go to compile and a package isn't there or you find you compiled so it doesn't match. No per instance driver installation across fleet. As we talk about diverse fleets, what we see with a lot of HPC administrators that will move into enterprise with AI is that they spend most of their time building these bespoke images to either support the specific hardware config or to support the scientist request for what's in there.

Mitigating compatibility drift between nodes, which, another thing I probably shouldn't hit on, actually can turn into behavioral drift. And then not having to build and maintain custom gold images. An argument that I didn't hear, right? But you can make it as, "Okay, you guys did all this. If I have a smart person, I can do this once and I can stamp it out across my organization." But the reality is all these things are rolling constantly. So, it's not just going to be once. You're going to have to continually build and maintain these golden images. So, that's where we're looking to overall in this is get you using your GPUs faster and reduce your overall FTE or full-time engineer cost.

So, ideally they can go focus on something else. They can focus higher up the stack. And honestly, I'm going to say getting engineers that have a deep experience to do things like this, or to build out what is our CIQ Linux Kernel 6.18, they're hard to find. So, what this is is it's an enterprise build of the long-term upstream kernel. As we've talked about, AI specific tuning. Its goal is to provide both performance and stability together.

And when I talk about the expertise that is hard to find, what I'm proud of is around me in the organization, I have an organization built of people that have spent decades in HPC and performance intensive computing, all the way from Cray to Beowulf clusters, which is now what we're trying to take out and do, to deliver an enterprise production environment. And people that have spent decades building Linux. People that have spent 16 years building Amazon Linux, building CentOS Linux. And this is what allows us to take this kernel, build it, deliver a bit in a reliable way.

And so, people may find interesting to a discussion here that in a lot of cases, we have people saying, "I'm not necessarily wanting AI workloads, but I want the newer kernel vendor backed." So, what we're starting to do is to offer what we call CLK, the newer kernel, on its own. So, people can plug and play. Yes. >> Brian, can you dive a little deeper into that, that's by your experts? I can very easily imagine a customer who's like, "Ooh, I want this." >> Yes. >> I don't have the folks who can put this together and deploy this.

Can you talk a little bit about how you would support a deployment in a medium or a large enterprise? And then the second question, kind of related to that, is once I've got this deployed, how do I take care of it? Can you talk to us a little bit about how would you help that medium or large enterprise customer support now their shiny new >> Absolutely. >> deployment. >> And I think the key thing I'm going to, yeah, I was going to say I'm going to extend what I probably could make a simple answer, because I'm known for making simple things complex.

But >> We're an engineer. >> We're in a room together because >> [laughter] >> Yeah. >> So, how do they take care of it? The reality is they don't. We take care of >> Okay. >> And that's what we're solving for. >> Okay. >> So, this is distributed with, ultimately it's possible to be distributed with any one of our distributions. So, back to the validated, is validated to work with all your standard packages and any workload upstream Rocky runs. So that, I think, is going to be the key thing there. I'm going to go back to the other side and say, what is the use case I'm saying?

I think you've already kind of called it out. We have people like quant trading organizations, right? Where milliseconds are millions of dollars, to be a little pithy. And they have to hire people to go tune kernels, but then because milliseconds are millions of dollars, people don't want you to bring their systems down to patch kernels and >> Yeah. >> and bring them up, right? And so that's a cost and it's a risk of that they started to come to us for and say, "Hey, we want this." And really, I think the short answer is going to be we package it and deliver it just like you would deliver 5.14 kernel.

It's in our repos. You download it, you run it. If a CVE or an RHBA bug advisory comes up against it, we patch it, your repository gets updated. So, I don't know, does that answer the question? >> Yes. >> Okay. >> Thank you. >> Okay. And I'll get a chance to brag. I'm probably going to run out of time. I got myself behind here is that >> Oh, just an extension of what Mish is asking in terms of the continuous validation. How do you show it, that the hardening has take place with all your tunings, or that we've not, and after, with the new version or revision of baselining, that we're not losing the initial GPU optimization that you've >> Yeah.

>> You >> That's interesting cuz customers haven't really asked that, but it's something I'm building for today. So, I think the probably less than impressive answer is they're buying based on a relationship with us, trusted vendor backing, the fact that we have SLAs across support as well as security. All right. So, there's a throat to choke if we don't deliver on the promise. Now, I am specifically for both hardened and will be different. So, we've talked about Pro Hardened. With Pro Hardened, we have DISA STIG, PCI DSS, NIST 800-171. With every build, we harden it.

We run the scan and we deliver you a scan results with what you'd call a plan of action and milestones on things that you need to remediate yourself. For Pro AI, it is different because largely we're proving performance or lack of performance degradation. We deliver periodic benchmarks. What I am moving to over the next couple of months is continuous benchmarking. So, every time you get a build, you'll get an official benchmark across a family of products. >> Okay. >> Is that something that if you were buying, you'd be concerned with, right? >> Well, yes. I mean, I also sit in front of governance committees and auditors as well.

So, it's like rather than say, "I trust this vendor," how do I show it? >> Yeah. And the other thing you'll find with this is going off on hardened. AI doesn't have this. >> Yeah. >> While we're submitting against public benchmarks, Artificial Analysis, the ones NVIDIA uses, MLPerf, the latest, and then we're looking to benchmark against their new agentic benchmark. I can't remember what it's called. But then when we go for our Pro Hardened, where you have more official bodies, we actually have submissions to CIS, as well as to DISA, for our STIG.

So, ultimately, what a lot of our customers want is they only want us to prove it. They want DISA to say, they want some other body to say, "Yes, this is, we'll accept this." So, problems solved by the kernel. You get hardware and enabling it right away without waiting. I'll give a, like, what is one of the anecdotes? And I'll call out Red Hat, right? And other distributions that deliver stable enterprise Linux may backport from long-term upstream. So, you can get a 5.14 kernel that has some of the newer hardware support. Obviously, they need to support GPUs.

But you're not guaranteed it's all going to come forward. So, I recently benchmarked, and I'm just going to jump ahead because we may not get to it. Right? So, this is a benchmark against the 5.14 kernel. Hidden in here is actually a benchmark against the 7.2 kernel. And the NVMe sequential read throughput, as well as mean read latency, was materially better, not because I can claim that I or my colleagues are geniuses, but because the scheduler was improved upstream. Right? So, our magic is being able to pull the scheduler forward, and being able to tell people that we are going to support it, and we're going to keep it patched and secure.

Okay. What we talked about earlier, what we get a lot of the quant orgs. You don't have to hand build a kernel, or an HPC organizations. You can get a performant kernel from us. And then you get that performance from that kernel without giving up your turfs. That's really the magic of it. Now, we deliver those kernels with secure boot. I marked FIPS because it's available. We deliver it for a large customer. So, if someone needed today, they can have it, but I'm working on delivering FIPS with the PQ ready algorithms as part of the Pro AI stack shortly.

We have CVE patching and remediation with an SLA. One of the things I like bragging about is that one of our best cases, gosh, I can't remember the name of the CVE, came out in May. We were able to deliver a patch for it the same day, 8 days ahead of upstream being able to deliver a patch for that. All right. So, then another value prop we talked about was the container family. Cuz the end of the day, a lot of people, the OS is invisible. That's why I'm talking about it. If I ask people, I've literally asked people, what OS are you running your inference on?

Kubernetes. All right? So, as far as they're concerned, and this is kind of, this is my kids' generation, how do I deploy? Docker pull, right? I'm not thinking about the underlying OS anymore. So, we have to deal with the reality that distributing workloads is happening at the container level, but also recognizing that at the end of the day, the container, containerd, etcd, all of those are running kind of at the behest of this thin layer running between you and the hardware. In fact, one of the main things the neoclouds do that not everybody has the ability to do, especially across a heterogeneous fleet, is just slim that OS down to as small as it can.

To your point, deliver flat back containers, etc. So, we distribute these to support flexibility. So, we deliver a base that just gives you the drivers if you don't want to deal with those. We deliver a runtime built to support production inference with the CUDA stack as well as with the CUDA stack. We deliver a development version that actually brings forward AMD and NVIDIA software development kits. They're big. You don't want to send them to production, but this importantly speaks to what I've been building for, and that's kind of from lap to rack, right? From desktop to data center. Being able to take a container that you run on your desktop, build out a workload on it, and know that that stack and that environment is going to be the same and supported in that middle layer when you go to production.

And then we're building out a family of workload specific containers. So, llama.cpp. Now, the key thing with these is going to be that no container rebuild when the host driver updates. That we're maintaining continuity and co-versioning between what we deliver for the container and what we deliver in the bare metal OS. Right? And this remediates drift that sometimes can be the cause of hidden problems or very silent performance drift. Right? And also what we are finding is kind of fragmented. Look, if you want to deploy an RLC Pro AI container on another Linux distribution, you can. It will run. We're not trying. This is kind of that flexible stack.

If you want the performance benefits of our compiled kernel, come to us. In inverse, if you want to deploy the bare metal OS and deploy NVIDIA Ubuntu base containers on top, you can do that as well. And then key thing, no assembly reassembly of your containers when moving from laptop to a fleet. >> I'm a little confused. You surprised me. >> Yes. >> This is taking turnkey like another layer up. >> Yes. >> And I don't know, like, what is it, the cattle versus the >> Pet versus cattle. >> versus cattle, right?

I don't feel like I've encountered a lot of customers who want to treat their containers as >> Well, >> cattle exactly. >> I guess I'd say it's not creating your containers as cattle. And we may have to talk, as I race against time, and more about this. But I'll try to say, as people, various, especially agentic, as someone talked about earlier, are kind of hosting, managing, and distributing their AI compute stack in containers, right? You make one, you stamp it all out. We're not changing that. It's back to the, we're going to get you 80% of the way there.

We're going to give you 80% of the layer that you need in terms of your driver and inference stack. You describe another layer on top of that. Now you have your containers, you can go stamp out. >> Is this under management by CIQ as well? Bart? >> In terms of, yes, it's a product. It's delivered as a product. And we can support how you actually deploy this. >> So, who's the typical customer buying this? I mean, I guess I'm sure much older than your son or your son, but I'm kind of like, get Docker pull, like, I'm not even, I'm even worried about this.

I'm just like, my stuff's going to work. Like, so who's like buying this, and like, I'm really worried about that. >> So, people with a, either couple of things. Extensive GPU investment. >> Okay. >> Right? I am watching the dollars and cents, and I want to get the most out of our investment. Even in HPC, I've had people come to say I can do more with it. I'm going to rush. You got me with 11 minutes, I'll race through the rest. Then I'm going to tell the story about, I was at NVIDIA GTC.

I was talking to a guy from NOAA. They have to take images from a very expensive satellite. They have to run fast Fourier transforms on them. If they can't run that fast Fourier transform enough, that signal that was captured is lost forever. All right? So, when we were talking about even Rocky, community Rocky alone, having a measurable performance benefit over Ubuntu, he was out of his wits. He's like, "If I can run these transforms 3% faster, there's, and I don't know the number, that's material." So, this is somebody where units matter. GPU time, processing time, and then to the extent, like a pithy thing I used earlier, where milliseconds matter.

I'm a quant, I'm doing very high-frequency trading. Everything I could squeeze out of it matters. >> So, a lot of the optimization is focused on drivers and hardware. What if a customer wants to use RHEL and they want to also optimize their own application? So, what kind of optimization capabilities do they have towards the OS? >> So, let's find that you talk about that as we move up the stack. So, that's not in our wheelhouse today, right? You have a specific application that you're going to optimize. What we're finding we're supporting, though, is I have a specific application, I'm optimizing it, and I want to figure out how it works with the OS.

There's sort of different layers of optimization, and I'm going to push through in the 10 minutes cuz I think some of these things may be spoken to here. I'll just quickly hit a couple of sort of top-line numbers. Our time to first inference was 3.7 or 3 minutes and 43 seconds compared to 30 minutes for community Rocky and Ubuntu. Our overall inference throughput on this particular hardware, but identical hardware, on image segmentation, we got 32% faster. On non-VL accelerated, in this particular case text inference, we got 10%, I'm sorry, not faster, more tokens per second or more or faster throughput.

We saw this earlier. We benchmarked the kernel. I wanted to validate or evidence that the upstream kernel had benefits. These are some of the benefits we've seen, and I think this particularly important cuz despite everything I'm saying here about throughput, a number of people are saying they're in a point where they're not compute bound anymore, they're network bound and they're storage bound, right? So, being able to provide access, a lot of people are moving from single node GPU setups to, again, more like traditional HPC multi-node distributed jobs. Won't spend a lot of time on this, but just to say, what we take pride in that we deliver is a current kernel with support, AI specific tuning of the overall stack, pre-built and validated frameworks, support across bare metal, cloud, VMs, containers, and bootc.

You're going to find most people offer one of these, but not all of these, and then the really key thing is it's an enterprise Linux with enterprise-grade commercial backing and SLA. So, next I want to talk about how do we run workloads, which is another type of sort of optimization on utilization of the compute, right? And this is simplifying the ability to run AI workloads in a compute agnostic way.