CIQ Presents at AI Infrastructure Field Day 6

Then the role of the OS in maximizing the throughput and utilization of your GPUs. And then we'll talk about, well, once you have that platform established and you have this composable kit with the OS in place, how do you actually deliver your workloads in a reliable, consistent, and trustworthy way? So, why an AI kit versus a factory? I'm going to pause on this title slide for a sec just to talk about some of what I'm observing here. And that is that we're in a technical evolution cycle that is moving faster than I've ever seen in my career, and I think that most of us have seen in our careers.

Usually, I'd say, when we look at layers to any technology we deliver, one or two of those layers are kind of evolving at a time. We establish a foundation, we move on it. We're in this world where every layer of the AI stack is evolving and moving, and in some cases on one to three month cycles. So, even the hardware is revving inside of a year. And so, we're in a world where, under that pressure to deliver, go from research to production in three months, everybody jumped to: somebody just give me a stack, a factory, and I'm going to go deliver AI and I'm going to deliver inference.

But as we'll talk a bit about, the other side of that rapid evolution is when we tie ourselves into a stack, we might lock ourselves into something that is going to be obsolete or antiquated in the next three months, in the next 12 months. So, we have to build for flexibility. Now, a bit more about myself. I'm Brian Dawson. I'm here with CIQ. I have decades, I say decades instead of the actual years, in software and technology delivery. I started as an engineer, moved on. A lot of my time was spent in DevOps. I'm excited that where I started, at Sony PlayStation working on single chips, rapid matrix math, the geometry transformation engine, is now what we're representing in these GPUs, orders of magnitude more.

I remember when I used to get excited about transforming one polygon on a screen. So now, at CIQ, I help lead product strategy and I oversee product management for a few of our products, couple of which we'll talk about today. One is Rocky Linux from CIQ Pro AI, Rocky Linux from CIQ Pro Hardened, and then a just announced product called the CIQ Stratum Kubernetes Platform, which is a Kubernetes engine as well as a fleet management solution. So, I want to set the stage for CIQ because it's a lot of the underpinning of this thesis, and this will explain a lot about why I'm excited to be here at CIQ and excited to have this conversation with you.

So, our company is the company behind Rocky Linux. It was founded on years of expertise in HPC and Linux, and I think that is sort of the unique value that we offer in assembling this stack. It was founded by Gregory Kurtzer. He was one of the co-founders of CentOS. I'm going to assume most of the people in this room are familiar with CentOS. Don't go into much detail, but if you are familiar with it, then you understand how widely adopted it was and how important it was in terms of giving a free and open source option for running production-grade Linux.

And you would actually find in the number of shops that about 10% would be paid RHEL licenses and about 90% would be CentOS. Greg is also the creator of a couple of our products, driven by his 30 years in the Department of Energy and Berkeley National Labs. He created a product called Fuzzball, which we'll talk about, another product called Warewulf, and Apptainer, formerly known as Singularity. And so, he's sort of driven building this foundation in our organization based on delivering performant, resilient, secure infrastructure in order to drive some of the world's most intensive computing and simulations. So, our thesis that I'm going to build the rest of this discussion on is that existing infrastructure is not prepared for the demands of the age of AI.

So, what we believe is that we need a new modern IT stack that needs to be rethought now, versus trying to incrementally iterate when, as I said, the whole layer is rapidly iterating. And some of the problems we're seeing is that enterprise Linux cost are rising. While Linux is now pretty much commodity for AI, at the last stat I checked, Linux runs 95% of AI workloads. Ubuntu as a Debian distribution holds about 53% of that, and then enterprise Linux distributions hold the rest of it. But we're also seeing that a lot of vendors are increasing their prices. They're moving to compute-based pricing, they're moving to consumption-based pricing, and that is starting to squeeze those budgets when people have additional areas where they need to spend money.

Security is more and more critical. It is critical across the board with AI accelerated threats. But then when we also talk about sort of the crown jewels of an organization that we are handling, or as a neocloud or a hosting provider we're handling for our customers, this data is critical. It's valuable. It needs to be protected. People are spending an immense amounts of money on GPU hardware, and oftentimes they haven't figured out the stack, so it's sitting idle. I did a calculation with a customer that we're doing a build out for. God, I can't remember exact cost of the hardware, but the hardware was purchased, sat on a dock while they waited to figure out networking, cooling in the data center, and then get the infrastructure built up.

Well, in this case, I think they might have mounted in the data center. We're probably not going to start the full build out until they're fully ready in 2 months. My calculations have it at, that is $15 million in GPU hours. This is roughly 9,000 nodes, current generation NVIDIA cards, and for every 30 days that that hardware is not producing value, it's costing them $15.6 million, which was astounding. But I'm hearing that there's a lot of this shortage of GPUs. We can finally get a GPU. We got a GPU. We're writing off huge capital expenditures, but we haven't figured out how to tap them.

And then we're finding that operations are fragmented. We go through the cycle of centralization, decentralization. We are in sort of this explosion of Kubernetes where people are managing traditional fleets. Then we have GPU nodes standing up with different solutions for those. So, we're starting to see kind of refragmentation of IT that eventually is going to have to be federated again and brought back. Any questions so far? Okay. So, what do we believe the solution is? It's a new hybrid first integrated stack. All right? We believe that performant workloads, i.e., this performance-intensive computing that we have to deliver on these GPUs to power cutting-edge workloads, requires a new infrastructure stack.

But not only that, requires a new infrastructure stack. The whole environment has a cascading impact on your entire fleet. Even, yes. >> Do you really think that one stack is going to cover AI kits and enterprise solutions? That's the challenge. You talk about the fragmentation of IT, but I don't see a solution here. Is your product a solution? Maybe that's... >> Well, it's a product suite. So, the thing is, is this where you go to a factory. Yes, I believe there's a solution. Right? It is, and I believe we have it or are contributing to it.

One is it requires, as we'll talk about, independent products. Two is it requires flexibility in the way that you distribute. And a lot of that flexibility is going to be building on open source and open cores, which makes it easier to integrate with your stack. You avoid vendor lock-in. But then our other core thesis, one of the reasons I joined the company for, is then having variants, as we call them, that we'll talk about, that are specifically tuned for a given workload. They get you 80% of the way there, but yet they have a common basis. Right? We'll talk another thing being about being enterprise ready and production ready.

In our mind, is using an enterprise-grade, production-grade Linux distribution that your teams are skilled in. >> So one of the things we've been talking about generally, I think, those of us observing and talking about what's going on with AI and its impact on infrastructure, is increasing diversity and parts the two to infrastructure down to the system and subsystem level. Really down at the silicon. Right. Down to the on-chip. In other words, multiple, the way I think of it is multiple vendors, but the way it's usually talked about is all these XPUs, all these new processing units.

It's not just about CPU plus GPU. You often have DPUs on the networking cards. You'll have FPGAs or ASICs like blah blah blah, but that's just extend that farther out to more heterogeneous architectures. >> Yes. >> Don't want to go further on that. I feel like you're suggesting a similar heterogeneity. >> I am. >> In OS. >> Yes. >> And then what you just described, though, is this sort of like commonality baseline with these different connected islands of workload-tuned OSs. I love this slide in part because it doesn't have AI on it.

>> Right. >> Right. So we're going to talk about this... >> Well, I'm going to AI wash this baby. Don't get it wrong. >> Yeah. So first of all, you're sort of validating this concept of mine that I hadn't really seen before, which is at many layers, in the OS layer is what we're talking about in particularly, diversity is helpful and required in that sort of, okay. >> And I'm going to illustrate some of this. So I'll move on in a minute, but first I want to say yes, that's the way we're seeing it.

That's the way I'm seeing it. And I would qualify that there's a world. Keep in mind that one of the things, the power of Linux is that it has every package in the world. You can build for practically every processor architecture. Every vendor that has meaningful hardware commits upstream support for it, right? So... >> But an operator wants to put all those things in and manage it in one place and as if it's monolithic. >> But right. And I think this is where I'll say there's a case. But that is a lot of fat.

Now, I think what it is, is it's a homogeneous foundation that can be deployed to support heterogeneous use cases. Those above, the workloads are running, and those that are below it. Does that make sense? So let's talk about this, right? And unfortunately this got a bit squished as I was working on it. But what we are looking about is, let's just first talk about diversity of performance-intensive computing workloads. All right? And yes, there are what I wouldn't even call database architecture being boring. But, right? Stuff that needs to just be stable and reliable, but at the same time, there are high-performance things demanding, driving needs of that.

Up top, you have your workloads that you need diverse support for. Down below, you have diverse compute and infrastructure. Even before we get down to processor architectures, NPUs versus DPUs versus TPUs versus GPUs, NVIDIA versus AMD, et cetera, you have on-prem, you have cloud, you have VPC. And AI in particular, you guys may have seen this. People rushed to the cloud because they didn't have GPUs or infrastructure support. Got to be really expensive. I call it the AI hangover. Christmas, you got the bill. You went out and bought GPUs, and now you need to move these workloads over, right?

So, this hybrid sort of heterogeneous compute that you call out is one of the things. And then the heterogeneous workloads. And the question becomes, how do we deliver orchestration, operation, automation, and overall management of that stack in a way that it supports that diversity? Okay. So, I'll pause here. We'll sit on this for a second. I'm going to come back to this, but I'm going to go to individual product layers in a minute. So, sort of the simplified view and simplified theory is that an AI kit is something that you build from the layers you need that you don't have today, and you run them alongside your existing infrastructure or the components that you want to build and deliver yourself or buy from another vendor.

But what it should be able to do is evolve, right? As your compute underneath changes, you should have a stable layer that doesn't need to change. As you move from bare metal up to Kubernetes-orchestrated workloads, it should be able to support that. And my concern is that an AI factory is one vendor's fixed stack based on their opinion at this particular point in time, and on the centralization-decentralization curve, that thing's going to break back apart in a couple of months. Okay? So, next I want to talk about the host layer under the GPUs. I'll talk about the fact that it's the OS that controls access to all these expensive resources we've talked about, enables that workload layer up top to tap the compute layer below, but frankly it's a forgotten layer.

I don't know how many AI conferences I've been into in the past few years. How many AI layer cakes we've looked at that completely ignore there's this little thing called the kernel that is responsible for turning your devices on and off or configuring them. Even when we have direct memory access, you're still engaging with the kernel. >> Brian, a lot of that control is starting to move from the kernel to the GPU. >> Yes. And this is where I was going to say, even as we get, and I don't know about being able to get into benchmarks, but I recently was benchmarking against GB10s, right?

And so GB10s and GB in general, they have the Grace Blackwell, they have the unified memory. Yes, it pulls a lot of that memory sort of thrashing down. But at the end of the day, currently, at minimum, the kernel's turning the switches on and off, let the thing run, right? But also oftentimes it's still controlling the flow of streaming from storage and controlling the flow of streaming from other places. We'll come talk back about that. >> Apparently. >> So, and I think you'll have some things to comment on here. So, what I want to talk about is how you can use the OS to maximize your GPUs.

Again, I'm Brian Dawson. I've spent a number of years in delivery of software and technology. A percentage of my career in DevOps. I am now representing CIQ, helping lead product strategy as well as product management for our Rocky Linux and Kubernetes products: Rocky Linux from CIQ Pro AI, Rocky Linux from CIQ Pro Hardened, and our newly announced Kubernetes engine and platform called CIQ Stratum. I want to talk to you about important layer in the AI stack. And if I were to just oversimplify the AI layer cake, I would break down the major categories, at least the major categories that we're ready to address, as your workloads, your hardware on the bottom, then the orchestration of those workloads across that compute, the operation or OS that gives access to that compute, the automation that manages it, and then the management.

How do I provision? How do I keep packages up to date? How do I deliver patches not only to the OS but up that stack? This is the stack that represents CIQ's products. We're going to focus right now on the RLC Pro family of products. So RLC Pro is basically a vendor-backed version of upstream community Rocky Linux, where we had CVE remediation, support SLAs, secure build, secure boot. We then roll that into what we call workload optimized variants. And then RLC Pro is what we have kind of called sort of our homogeneous, well-understood foundation. Now, for instance, if you have a particular AI workload that you're running on top of accelerators, we deliver Pro AI.

We'll go into more detail, but what we do is strip the packages and tune that to get the most out of that hardware and to make it fast and easy to deploy. Now, if you're in a secure environment, we have RLC Pro Hardened. And oftentimes these things aren't mutually exclusive. Right? I'm dealing with AI diffusion export controls, which requires effectively compliance with what is called NIST 800-53 or CMMC. And there's particular requirements for that that will throttle and kill your GPU. So, what people are having to do is harden their accelerator node to a certain level and then architect their overall system so that they deploy something like our RLC Pro Hardened to have out-of-the-box compliance on the other nodes.

Okay? So, what Pro AI is, and I stripped it out, but I'll say it was really as some other Linux distros were making their bets as to what AI looked like, they were focusing at the Kubernetes and container orchestration level, at the OpenShift, at the Rancher Prime level. So, we were the first ones to define and build a bare metal OS initially and specifically targeting production inference. You guys are probably aware with the stats that the prediction in early 2025 was, or the measurement was, that inference was costing three to four times as much as training. Training tends to be occasionally, but inference is constantly, and it scales with your success.

And some predictions said that inference would outpace training cost by up to 17x by the time we get to 2030. So, what we built is RLC Pro AI. Now, sort of bottom line up first. Some of the highlights of what we were able to recognize, and we'll get into this, is in certain situations we were able to lay down the OS and get 32% more throughput. And I'm going to stop on this because it's a big number. Like, okay, you can get 32% more input. Yeah, thing is, every workload is different, every hardware profile is different. You may have a different bus, you may have different memory, you may have different processors, you may have different resource usage patterns.

So, this is one of our most impressive numbers we got in a particular condition or situation, but it shows what the art of the possible is. Yes, question. >> A lot of workloads in the cloud are processing which you're running in containers or out of containers. So, do you still get that throughput when you're running the container like a Kata container or some type of container? >> On the shape of the profile. And we'll talk about how the key... >> Run Rocky Linux on the Kata container then to get that performance? >> Not familiar with a Kata container.

>> Well, it's a micro VM. >> Yes, okay. Okay, right. Kind of like Bottlerocket. Yes. So yes, we cannot, because the container, unless you're running bootc, you have an OCI compatible, we cannot tune the kernel. You can do some things in the tuning of the stack in the user space, but tapping performance heading in this direction requires coordination between the containerized environment and the bare metal OS, which I'll talk about. >> That more throughput number is tokens per second, inference per second, prompts per second? >> This right here is tokens per second for image segmentation.

And this is where we can get to be specific. It's tokens per second for image segmentation on Ampere generation hardware. >> So, it's MLPerf type of numbers, then? >> Yes, this is a slimmed-down version of the MLPerf. I just ran official open category MLPerf benchmarks a couple of weeks ago. But this is a little older, when we took MLPerf and we slimmed it down. What we also deliver is a conformal kernel advantage, and we'll talk more about this. At the most extreme, it's an 18-month kernel advantage. What does that mean? That means that we are delivering a vendor-backed kernel that has up to 18 months of vendor feature commits and improvements in the kernel that won't make it to the traditional enterprise Linux.

>> Is that a managed service sub basis, more or less, right? Or no? How's the enterprise paying for this? >> They pay on a per node basis. We sell a OF... >> Per... >> Per node. >> Is that a perpetual license? What's the... >> A per node per year. >> Per node per year. >> Okay. Per node per year. And we'll get more into this. And then we found, of course, happy path scripted, some certain people, in fact, Tech Field Day people, when I brief them, told me that I've underestimated this number, but we found that you can move nine X. Certain people, I won't name them.

Nine X times faster. On what is behind this number, cuz we won't, I think, go into detail, is that from a clean OS install, right? You've booted to an ISO or QCOW or VM, to your first token being output from Ollama in this case. We were able to do it in 3 minutes and 43 seconds when both upstream Rocky and Ubuntu took 30 minutes best case. And simply, largely because, as we'll talk about, we pre-baked and tuned. But with other OSs, you have to download and compile commercial modules, set your infra FS or adjust your grub parameters, and reboot.

And then lastly is this, again, hybrid or kind of this heterogeneous, to continue to use that word, compute. We deliver a like environment for bare metal, on-premise, Kubernetes, AWS, Azure, and GCP. All right? Yes. >> So, what's your definition of validated? Is it including the security hardening, or... >> In this case, validated is validating that it works together and there's no performance regression. All right? Because one of the things you hear a lot about, even in just getting an image up, is going to be the dependency doom loop. So, validated is just to say we've taken this stack, we've took versions, we tuned them, we pinned those versions, and then we went through our test and validation suite to make sure it works, and we've benchmarked them to make sure there's no degradation.

So, let's get into the details of what this is. So, RLC Pro AI is, as I said, vendor-backed Rocky Linux tuned for GPUs and performance. We've built it to support inference from first boot, and we'll go into detail on this, by taking pre-tuned validated frameworks. So, when you get your GPUs and you have to stand them up, you don't have to spend time identifying the right drivers and the right stack configuration that, for at least some of the lowest common denominator workflows and workloads, you can boot the OS, you can have PyTorch, TorchVision, your CUDA stack, and your GPU drivers all running and working.

>> So, that's all NVIDIA language. >> And ROCm, and the ROCm... >> Etc. Okay, good. And is this, feel like I'm going to expose myself here, but why not? Is this auto discover, or are you clicking in a wizard to say here's my GPUs, or like how are you tuning... >> Currently, so auto discover is on the plan. Currently, you have to get an AMD image or an NVIDIA image. Now, if you're standing up yourself, you also have access to both of those repos for the images with precompiled modules.

So, you can pull them from the repo if you have a truly diverse fleet you need to dynamically discover. And then what we do is it's... >> Do you have PS for this sort of thing as well, or something so that you all can do that for the... Okay. >> All right. >> Which goes to just what I'm finding, a lot of people don't just want a product or a bit. Hit back to that AI factory thing. They want people to help them understand how to do it. And then of course what we do as part of that tuning is that we know you can set parameters when you compile CUDA.

You can set parameters when you compile the GPU drivers to determine what the range of support is. So we kind of slide that window. So we look to support the last three generations and keep them performing. You hear a lot about NVIDIA and you hear a lot about NVIDIA authorized, because we, very early on, prior to AMD, were actually the first Linux distribution to go to NVIDIA and get rights to redistribute their CUDA stack and their GPU drivers. And then just a key thing, like we've talked about, is one stack, same stack for every environment, really trying to capture and support this move from bare metal and cloud to sovereign on prem.

We see that the key differentiators are that, with Ubuntu, we'll call out, you can right now go pull, I think, 24.04. I think you can get the 7.0 kernel. You can actually go get NVIDIA DGX OS or NVIDIA OS on your DGX. I think they just bumped that to the 7.0 kernel, and that thing performs. But there's a couple of things. That's a Debian distribution, and 7.0 kernel has already been deprecated upstream. So you're starting to build your production delivery on something that has no guarantee of support in the future. So for us it's taking an upstream kernel that is ahead of everybody else.

Upstream long term latest stable, if you're familiar with kernel.org, and providing vendor backing to ensure that you can count on how long you'll be able to run production workloads on that. A kernel tuned for AI workloads, and we're continuing to tweak and tune and learn here. These are actually, we're pretty transparent with the settings. They're actually settings anybody can tune. But what we've done is we've took the time to tune memory allocation, transparent huge pages, and to tune those in a way that they work with the stack. Yes. >> Devil's advocate, you can use configuration management tools to do all that, too, though, right?

>> Puppet, Chef, Ansible. >> You can. Now, there's going to be a couple of things. There's going to be build time parameters, a couple of which we've tuned, which, sure, you can use any automation to drive your build time. Then, there's going to be run time parameters. You could use automation to tune your run time parameters. But again, then we're talking about the difference between I stand up a system, I run automation against it, I run it against it multiple times to kind of configure it, versus I lay something down from the start.

>> And so, just to be clear, you're tuning the kernel parameters to optimize it for AI workloads. >> Yes, for inference workloads. And this particular case is our default. Then, on top of that, and to your point for automation, this is what you would trigger. We actually have TuneD tuning profiles that can detect what your workload is or be triggered to be tuned for a specific workload. But the other thing I'd say here is you are correct, but then when we go to validated stack, driver, CUDA toolkit, and framework, there are also key settings within the CUDA stack and the PyTorch stack that can unlock performance.

All tunable, but most people aren't tuning them. One common thing we saw was TorchVision. When you're doing vision-based workloads, by default, and in fact, we did this by accident once ourselves in a build, by default, I'm finding people have it set to CPU. So, you're losing, I don't know what the order of magnitude is, but underneath this complex stack, when people are thinking at containers and they're not thinking about OSes, right? Or they're just sort of outsourcing their stack, you can end up in a world where a small thing, like a single binary flag on TorchVision, means you're not even using your GPUs for that process.

So, what does the validated stack look like? We're continuing to grow it. Right now it is PyTorch, TorchVision, Ollama, vLLM, and we have others in the works. And each of these, or as we said earlier, validated with a specific version or versions of the driver and kernel. We'll talk about that. In this case right now, on our 10.2 and our Rocky Linux 9 or Pro AI 9 and Pro AI 10.2, we are shipping CUDA 13. And then on the NVIDIA drivers, we have, it's roughly what I'll call, an n plus two strategy. In other words, to hold for production stability, we ship the latest long-term data center driver, but then we also ship the latest feature branch and a version in between that.

It'll come up later here that we discovered that on a DGX or on a GB architecture, we were able to see up to 9% improvement by running the 595 driver versus the 580. >> Not necessarily here now today, but do you have a flavor of this for our AMD friends and ROCm? >> We do. >> ROCm's CUDA compatible, so like they'll, the driver side though, obviously. >> Wait, we absolutely do. There reasons we could discuss later as to kind of why it's not headlining here, but the short answer is yes. >> I think the reason it's not headlining here is obvious.

But I would just say that the AMD customer tends to be on a much bigger scale and a little bit more DIY oriented, so I just want to support this. It makes a lot of sense. >> Right. Yes. >> So just, we're already in the world of the AI kit, which I hope you get back to soon, as opposed to the AI factory. That's a little more modest, even if it's still significant in implementation. And so that makes sense here. So it's not a critique. I just wanted to say... >> But I'd love to have the discussion, but I will say that the short answer is yes.

The reference that I'll have here that I am proud of is that AMD came to us to build Rocky Linux as in terms of what they distribute by default with their HPC clusters. So that's sort of the extent of the relationship, and then actually, or not the extent, an indication of the relationship, and sort of speaks to the size of workloads that are running a lot of HPC. So what are we looking to solve with the validated stack? No post-installation driver assembly, which could get you in the dependency doom loop when you go to compile and a package isn't there, or you find you compiled so it doesn't match.

No per-instance driver installation across fleet. As we talk about diverse fleets and what we see with a lot of HPC administrators that will move into enterprise with AI is that they spend most of their time building these bespoke images to either support the specific hardware config or to support the scientist request for what's in there. Mitigating compatibility drift between nodes, which, another thing I probably shouldn't hit on, actually can turn into behavioral drift. And then not having to build and maintain custom gold images. An argument that I didn't hear, right, but you can make it as, okay, you guys did all this.

If I have a smart guy or smart person, I can do this once and I can stamp it out across my organization, but the reality is all these things are rolling constantly. So it's not just going to be once. You're going to have to continually build and maintain these golden images. So that's where we're looking to overall in this, is get you using your GPUs faster and reduce your overall FTE or full-time engineer cost. So ideally they can go focus on something else. They can focus higher up the stack. And honestly, I'm going to say getting engineers that have a deep experience to do things like this, or to build out what is our CIQ Linux Kernel 6.18, they're hard to find.

So, what this is, is it's an enterprise build of the long-term upstream kernel. As we've talked about, AI-specific tuning. Its goal is to provide both performance and stability together. And where I talk about the expertise that is hard to find, what I'm proud of is around here in the organization, I have an organization built of people that have spent decades in HPC and performance-intensive computing, all the way from Cray to Beowulf clusters, which is now what we're trying to take out and do to deliver an enterprise production environments. And people that have spent decades building Linux.

People that have spent 16 years building Amazon Linux, building CentOS Linux. And this is what allows us to take this kernel, build it, deliver a bit in a reliable way. And people may find interesting to a discussion here that in a lot of cases, we have people saying, "I'm not necessarily wanting AI workloads, but I want the newer kernel vendor-backed." So what we're starting to do is to offer what we call CLK, the newer kernel, on its own so people can plug and play. Yes. >> Brian, can you dive a little deeper into that? That's why you're experts.

>> I can very easily imagine customer who's like, "Ooh, that I want this." >> Yes. >> "I don't have the folks who can put this together and deploy this." Can you talk a little bit about how you would support a deployment in a medium or a large enterprise? And then the second question, kind of related to that, is once I've got this deployed, how do I take care of it? Can you talk to us a little bit about how you would help that medium or large enterprise customer, now they're shiny new >> Absolutely. >> deployment. >> And I think the key thing, yeah, I was going to say I'm going to extend what I probably could make a simple answer, because I'm known for making simple things complex.

But >> We're engineers. >> We're in a room together. We can... >> [laughter] >> Yeah. >> So, how do they take care of it? The reality is they don't. We take care of... >> Okay. >> And that's what we're solving for. So, this is distributed with, ultimately it's possible to be distributed with any one of our distributions. So, back to the validated to work with all your standard packages and any workload upstream Rocky runs. So that, I think, is going to be the key thing there. I'm going to go back to the other side and say, what is the use case I'm saying?

I think you've already kind of called it out. We have people like quant trading organizations, right? Where milliseconds are millions of dollars, to be a little pithy. And they have to hire people to go tune kernels, but then because milliseconds are millions of dollars, people don't want you to bring their systems down to patch kernels and >> Yep. >> and bring them up, right? And so that's a cost and it's a risk of that they started to come to us for and say, "Hey, we want this." And really I think the short answer is going to be we package it and deliver it just like you would deliver a 5.14 kernel.

It's in our repos, you download it, you run it. If a CVE or an RHBA bug advisory comes up against it, we patch it, your repository gets updated. So, I don't know, does that answer the question? >> Yes. >> Okay. >> Thank you. >> Okay. And I'll get a chance to brag. I'm probably going to run out of time. I got myself behind here, is that... Yes. >> Just an extension of what Michi's asking in terms of the continuous validation. How do you show that the hardening has taken place with all your tunings, or that we've not, and after with the new version or revision of baselining, that we're not losing the initial GPU optimization that you've >> Yeah.

That's interesting, 'cause customers haven't really asked that, but it's something I'm building for today. So, I think the probably less than impressive answer is they're buying based on a relationship with us, trusted vendor backing, the fact that we have SLAs across support as well as security. All right, so there's a throat to choke if we don't deliver on the promise. Now, I am specifically for both Hardened and will be different. So, we've talked about Pro Hardened. With Pro Hardened, we have DISA STIG, PCI DSS, NIST 800-171. With every build, we harden it.

We run the scan, and we deliver you a scan results with what you'd call a plan of action and milestones on things that you need to remediate yourself. For Pro AI, it is different because largely we're proving performance or lack of performance degradation. We deliver periodic benchmarks. What I am moving to over the next couple of months is continuous benchmarking. So, every time you get a build, you'll get an official benchmark across a family of products. >> Okay. >> Is that something that if you were buying you'd be concerned with, right? >> Well, yes. I mean, I also sit in front of governance committees and auditors as well.

So, it's like, rather than say, I trust this vendor, how do I show it? >> Yeah. And the other thing you'll find with this is, going off on Hardened, AI doesn't have this. >> Yeah. >> While we're submitting against public benchmarks, Artificial Analysis, the ones NVIDIA uses, MLPerf, the latest, and then we're looking to benchmark against their new agentic benchmark. I can't remember what it's called. But then when we go for our Pro Hardened, where you have more official bodies, we actually have submissions to CIS as well as to DISA for our STIG.

So ultimately what a lot of our customers want is they only want us to prove it. They want DISA to say, they want some other body to say, yes, this is, we'll accept this. So problem solved by the kernel. You get hardware and enabling it right away without waiting. I'll give it, like, what is one of the anecdotes? And I'll call out Red Hat right now, the distributions that deliver stable enterprise Linux may backport from long-term upstream. So you can get a 5.14 kernel that has some of the newer hardware support. Obviously they need to support GPUs, but you're not guaranteed it's all going to come forward.

Recently benchmarked, and I'm just going to jump ahead because we may not get to it. Right? So this is a benchmark against the 5.14 kernel. Hidden in here is actually a benchmark against the 7.2 kernel. And the NVMe sequential read throughput as well as mean read latency was materially better, not because I can claim that I or my colleagues are geniuses, but because the scheduler was improved upstream. All right? So our magic has been able to pull the scheduler forward and being able to tell people that we are going to support it and we're going to keep it patched and secure. Okay? What we talked about earlier, what we get a lot of the quant orgs, you don't have to hand build a kernel, or an HPC organizations.

You can get a performant kernel from us. And then you get that performance from that kernel without giving up your terms. That's really the magic of it. Now we deliver those kernels with secure boot. I marked FIPS because it's available. We deliver it for a large customer. So if someone needed today they can have it, but I'm working on delivering FIPS with the PQC-ready algorithms as part of the Pro AI stack shortly. We have CVE patching and remediation with an SLA. One of the things I like bragging about is that one of our best cases, gosh, I can't remember the name of the CVE, came out in May.

We were able to deliver a patch for it the same day, eight days ahead of upstream being able to deliver a patch for that. All right. So then, another value prop we talked about was the container family. Because at the end of the day, a lot of people, the OS is invisible. That's why I'm talking about it. If I ask people, I've literally asked people, what OS are you running your inference on? Kubernetes. All right. So as far as they're concerned, and this is kind of, this is my kids' generation, how do I deploy? Docker pull, right? I'm not thinking about the underlying OS anymore.

So we have to deal with the reality that distributing workloads is happening at the container level, but also recognizing that at the end of the day the container, containerd, etcd, all of those are running kind of at the behest of this thin layer running between you and the hardware. In fact, one of the main things the neoclouds do that not everybody has the ability to do, especially across a heterogeneous fleet, is just slim that OS down to as small as it can. To your point, deliver flat back containers, etc. So we distribute these to support flexibility. So we deliver a base that just gives you the drivers if you don't want to deal with those.

We deliver a runtime built to support production inference with the CUDA stack as well as with the CUDA stack. We deliver a development version that actually brings forward AMD and NVIDIA software development kits. They're big. You don't want to send them to production, but this importantly speaks to what I've been building for, and that's kind of from lap to rack, right? From desktop to data center. Being able to take a container that you run on your desktop, build out a workload on it, and know that that stack in that environment is going to be the same and supported in that middle layer when you go to production.

And then we're building out a family of workload-specific containers. So, llama.cpp. Now, the key thing with these is going to be that no container rebuild when the host driver updates. That we're maintaining continuity and co-versioning between what we deliver for the container and what we deliver in the bare metal OS. Right? And this remediates drift that sometimes can be the cause of hidden problems or very silent performance drift. All right? And also what we are finding is kind of fragmented. Look, if you want to deploy an RLC Pro AI container on another Linux distribution, you can.

It will run. We're not trying, this is kind of that flexible stack. If you want the performance benefits of our compiled kernel, come to us. Inversely, if you want to deploy the bare metal OS and deploy NVIDIA Ubuntu-based containers on top, you can do that as well. And then key thing, no reassembly of your containers when moving from laptop to a fleet. >> I'm a little confused. You surprised me. >> Yes. >> I mean, this is taking turnkey, like, another layer up. >> Yes. >> And I don't know, like, what is it?

The cattle versus the >> Pet versus cattle? >> Pet versus cattle, right? I don't feel like I've encountered a lot of customers who want to treat their containers as, well, cattle, exactly. >> I guess I'd say it's not creating your containers as cattle. And we may have to talk, as I race against time, and more about this, but I'll try to say is, people, various, especially agentic, as someone talked about earlier, are kind of hosting, managing, and distributing their AI compute stack and containers, right? You make one, you stamp it all out. We're not changing that.

It's back to that we're going to get you 80% of the way there. We're going to give you 80% of the layer that you need in terms of your driver and inference stack. You describe another layer on top of that. Now you have your containers you can go stamp out. >> Under managed by CIQ as well? Or >> In terms of, yes, it's a product. It's delivered as a product and we can support how you actually deploy this. >> Who's the typical customer buying this? I'm guessing, I'm sure much older than your son or your, but I'm kind of like, get Docker pull, let's, I'm not even, I'm even worried about this.

I'm just like, my stuff's going to work. So who's like buying this and like, I'm really worried about it? >> So, people with either couple of things. Extensive GPU investment. >> Okay. >> Right? I am watching the dollars and cents and I want to get the most out of our investment. Even in HPC I've had people come to say I can do more with it. I'm going to rush, you got me with 11 minutes, I'll race through the rest. I'm going to tell the story about, I was at NVIDIA GTC.

I was talking to a guy from NOAA. They have to take images from a very expensive satellite. They have to run fast Fourier transforms on them. If they can't run that fast Fourier transform enough, that signal that was captured is lost forever. All right? And so when we were talking about even Rocky community, Rocky alone, having a measurable performance benefit over Ubuntu, he was out of his wits. He's like, if I can run these transforms 3% faster, there's, and I don't know the number, that's material. So this is somebody where units matter. GPU time, processing time, and then to the extent, like a pithy thing I used earlier, where milliseconds matter.

I'm a quant, I'm doing very high frequency trading. Everything I could squeeze out of it matters. >> So a lot of the optimization is focused on drivers and hardware. What if a customer wants to use RHEL and they want to also optimize their own application? What kind of optimization capabilities do they have towards the OS? >> So that's why you talk about that as we move up the stack. So that's not in our wheelhouse today, right? You have a specific application that you're going to optimize. What we're finding we're supporting, though, is I have a specific application, I'm optimizing it, and I want to figure out how it works with the OS.

There's sort of different layers of optimization, and I'm going to push through in the 10 minutes 'cause I think some of these things may be spoken to here. I'll just quickly hit a couple of sort of top line numbers. Our time to first inference was 3.7, or 3 minutes and 43 seconds, compared to 30 minutes for community Rocky and Ubuntu. Our overall inference throughput on this particular hardware, but identical hardware, on image segmentation, we got 32% faster. On non-vLLM-accelerated text inference in this particular case, we got 10%, I'm sorry, not faster, more tokens per second, or more or faster throughput.

We saw this earlier. We benchmarked the kernel. I wanted to validate or evidence that the upstream kernel had benefits. These are some of the benefits we've seen, and I think this particularly important 'cause despite everything I'm saying here about throughput, a number of people are saying they're at a point where they're not compute bound anymore. They're network bound and they're storage bound, right? So being able to provide access, a lot of people are moving from single-node GPU setups to, again, more like traditional HPC multi-node distributed jobs. Won't spend a lot of time on this, but just to say, what we take pride in that we deliver is a current kernel, web support, AI-specific tuning of the overall stack, pre-built and validated frameworks, support across bare metal, cloud, VMs, containers, and bootc.

You're going to find most people offer one of these, but not all of these. And then the really key thing is it's an enterprise Linux with enterprise-grade commercial backing and SLA. And I'm going to skip this and push through. So next I want to talk about how do we run workloads, which is another type of sort of optimization on utilization of the compute, right? And this is simplifying the ability to run AI workloads in a compute-agnostic way. So, oops, I had two versions of the slide. So, I'm going to talk to you about distributed AI workflows with Fuzzball.

I'm Brian Dawson from CIQ. I've spent a few years in software development and technology with a lot of focus on DevOps, working across the spectrum, as an engineer in the early days at PlayStation, to product where I function now. I help oversee product strategy as well as product management at CIQ around our Rocky Linux from CIQ Pro AI product, our Rocky Linux from CIQ Pro Hardened security product, and our recently announced CIQ Stratum Kubernetes Engine and platform. So, we've been talking about this idea of a stack. When you encounter establishing any modern compute stack, but in particular an AI compute stack, you have multiple layers that you have to address, all the way to capitalize on your hardware while ensuring that you could support what your respective workload is reliably.

And that goes all the way from orchestration of those workloads, operation of the underlying hardware and systems, automation for managing your day-to-day operations, and then continued management, I mean, management either around continued setup or maintenance. And so what I'm going to talk about here is Fuzzball. So Fuzzball sits at our orchestration layer. It runs on top of Kubernetes, whether it's Stratum Pro or it's a generic Kubernetes distribution. It leverages etcd, and the initial goal was, we talked about customers and we're moving to Docker containers, is traditional HPC organizations needed to modernize their stacks and service the ability to support their students, their researchers, faster so that they can deploy more.

So we created, or Greg created, Fuzzball. Fuzzball uses Kubernetes and some other tooling to replace a traditional Slurm or MPI scheduler to enable you to stand up a distributed workload and run that workload across any... Let me see if this is actually going to work the way I want it to work. So you can run the same workflow on premise, in public, or on a GPU cloud, all without rewriting it. And if time allows, we'll pop into a quick demo. All right. So we go from a control plane that allows you to orchestrate, and then that same control plane that will allow you to federate across clusters and across compute.

All right. Let's see if my... So this is one platform, and it's actually, I have a typo here. I shouldn't put that there. It's one platform for batch HPC pause and AI. So, supporting traditional batch jobs and traditional stateless jobs, but what we've done over the past few years, realizing the enterprise need, is that we've extended this to support stateful as well as stateless AI workloads that aren't running on traditional infrastructure. And what we are seeing in diverse fleets today, I'm running into this, having this conversation like every couple of weeks, is, hey, I have to run my HPC cluster. I have to support general Kubernetes, but we are also getting demands for support for gen AI.

And we didn't build out our HPC clusters for inferencing and gen AI. So, now they're standing up these different stacks. And Fuzzball is one of those things that helps unify it. We allow you to describe a job in one workflow file. That's the compute. That's the images that it's going to deploy. That's the storage, all in one. And then you can then run that anywhere. >> Do you support checkpointing as a feature? >> Yes. And then, but we're also working on, and you're probably seeing or heard solutions about, I'll just call them sub checkpoints. I don't want to name, but continuous tracking. Yes, we support checkpointing.

We are also building out solutions ourselves and with partners that give you finer, that will call GPU resiliency. >> Right. >> So, the sort of pride and joy of this, and we probably won't get time to pull it up, is sort of the workflow catalog. Right? So, this allows me to take a job, describe it in YAML. Offer it to my customers, my clients. We have people white labeling and repackaging this to allow them to then run that job with just pushing that green run button without worrying about the compute or location. To support movement to production, we've enabled agentic integration.

So there is an MCP to your question. Fault-tolerant scheduling here, traditional checkpointing versus kind of managing it in contiguous fabric, but we are working on that. We can currently restart failed jobs elsewhere. This goes back to someone asked, not exactly what you're asking about optimizing an app, but this other layer of utilization and optimization. How do I keep a job running and use as much of my available compute as I can? Supports multi-tenancy, resource accounting, AMD ROCm and NVIDIA GPUs under a single control plane. This is a theme with us, and then overall operational hardening, right? So, federation, auto scaling, managing your certificates is the big pain with Kubernetes, is continually refreshing certificates.

>> Just zooming in on the operational hardening. Can you expand further in terms of, across cloud, how is the secrets or the identity federation handled, or the per-job isolation handled? >> Well, we can probably take some time afterwards. I couldn't speak to it in detail, but I'd be happy also to get the lead engineer on this to... I can say very inadequately, because I've been on calls where that was answered. They are handled, but I can't answer it myself. >> Follow on, it is like signatures stuff. Is that, if you're using Sigstore or >> I do not know if we're using Sigstore.

>> Okay. >> I know access and authentication, very inadequate to access and authentication. It's managed by a Keycloak. We are also separately integrating this into a single control plane across all of our products. It can deploy with all of our products, but you have different control planes. So, this is just an example of the dashboard, right? If I was to just quickly here go hello world, which I won't, I'll show it and I'll hit play, but then we're going to run out of time. You can see all of these lamps.

I'm just going to hit her, I can send out a Hermes agent, deploy that agent, expose the endpoints and offer that as a service, or use those endpoints for telemetry, and in this particular case it's very simple, but this will run a hello world job, deploy it on available compute, and print that out to the console. See, oops, I can't jump forward when I'm not on the slide. So what I look to show you here is that we can, that there's a need to think about the complete infrastructure stack. We can deliver the complete infrastructure stack, but yet we don't look to deliver it in a way where we require that you take on an AI factory.

I would hope that all of our products fit a need and you can deploy them, but in a world where things are evolving and you need to move fast, expecting that someone can take the whole stack is unreasonable. What we look to do is solve the problems of modern infrastructure, enable you to get the most out of Linux, access your GPUs from day one, not have to worry about security because we built it in for you, and deliver an automated stack across your diverse fleet. We have a webinar on CIQ Stratum Pro tomorrow, our Kubernetes engine. People, please check it out.

You can try our products at our customer portal. We have free downloads and trials. We have developer licenses, as well as we have self-service support, and to the professional services and support, we have a first-class support team. So thank you. >> Thank you, Brian, and remember to breathe now. >> [laughter] >> That was fast, that was heavy, that was hot in there. We could have spent a lot longer, and that's a good sign when you're at Tech Field Day. Thank you all for joining us for the CIQ presentation. It is the first time we've had CIQ at Tech Field Day. Suspect it won't be the last.

We will be back at 1:00 p.m. We're going to take ourselves a bit of a lunch break, and we'll see you back for our next session at 1:00 p.m. So, Scott, would you be so kind as to take down the stream for me?