Build a Composable AI Infrastructure Stack with RLC Pro AI and CIQ
Opening the CIQ session at AI Infrastructure Field Day 6, Brian Dawson, Director of Product Management at CIQ, makes the case for composable AI infrastructure. Every layer of the AI stack is now evolving, some on one-to-three-month cycles, and even hardware turns over within a year. That pace pushes organizations to buy a fixed AI factory so they can get from research to production quickly. Dawson argues that locking into a fixed stack risks owning something obsolete within a year, and proposes an AI kit instead: layers chosen for the job, run alongside what an organization already has, and swapped as the job changes.
Dawson introduces CIQ as the company behind Rocky Linux, founded by Gregory Kurtzer, who co-founded CentOS and created Warewulf, Apptainer, and Fuzzball. That HPC and Linux lineage shapes how CIQ assembles the stack. He lays out the pressures he sees across customers: enterprise Linux vendors moving to compute-based and consumption-based pricing, security getting more critical as AI data gets more valuable, expensive GPUs sitting idle while the surrounding networking, cooling, and software stack catch up, and what he calls the AI hangover, where teams that rushed to the cloud for GPUs now need to move those workloads onto hardware they bought. Operations are fragmented too, with one set of tools for the traditional fleet and another for GPU nodes.
A delegate asks whether one stack can really cover all of that, and whether CIQ is selling the solution. Dawson answers that it is a product suite, not a single product. It is made of independent products, and much of its flexibility comes from building on open source, which makes it easier to integrate with an existing stack and avoid vendor lock-in. Workload-tuned variants get a team most of the way there on a common base. He calls it a homogeneous foundation deployed to support heterogeneous use cases, above it in the workloads and below it in processors from NVIDIA, AMD, and other accelerator vendors, across on-prem, cloud, and VPC. The talk closes on the layer Dawson says gets forgotten: the operating system under the GPU. The kernel turns devices on and off, configures them, and controls data streaming from storage, and even as unified-memory designs move some control toward the GPU, the kernel still governs access to every expensive resource on the node.
Key takeaways
- Every layer of the AI stack, including hardware, now changes on cycles of a few months to a year, so a fixed stack bought today can be stale within a year.
- An AI kit assembled from swappable layers, run alongside existing infrastructure, keeps the flexibility a fixed AI factory gives up.
- CIQ comes to AI infrastructure from HPC and Linux: Rocky Linux, Warewulf, Apptainer, and Fuzzball all trace back to founder Gregory Kurtzer.
- The pressures Dawson sees are enterprise Linux moving to consumption pricing, higher security stakes for AI data, idle GPUs waiting on the rest of the stack, and operations split between fleet and GPU nodes.
- CIQ offers a product suite rather than one product: independent products, built largely on open source, with workload-tuned variants on a common base.
- A homogeneous foundation that supports heterogeneous use cases covers diverse workloads above and diverse accelerators and environments below.
- The OS, and the kernel in particular, still governs access to GPUs, devices, and storage, so it belongs in the design rather than as an afterthought.
Questions this video answers
What is the difference between an AI factory and an AI kit?
An AI factory is one vendor's fixed stack, based on that vendor's opinion at one point in time, bought as a unit to get to production fast. An AI kit is a set of layers chosen for the workload, run alongside what an organization already has, and swapped independently as hardware, frameworks, or requirements change. Dawson argues the kit is the safer bet when every layer is evolving on short cycles.
Is CIQ selling one product that covers the whole stack?
No. Dawson describes a product suite of independent products. The ones he names are RLC Pro AI and RLC Pro Hardened, both variants of Rocky Linux from CIQ Pro (RLC Pro); Fuzzball; and CIQ Stratum Kubernetes Engine (SKE), with CIQ Stratum Pro as the commercial version built on it. He says much of the suite's flexibility comes from building on open source.
Why does CIQ focus on the operating system for AI infrastructure?
The kernel controls access to every expensive resource on the node: it turns devices on and off, configures them, and controls data streaming from storage. Dawson says the OS is what lets the workload layer tap the compute layer, and that AI architecture diagrams often leave it out.
Is control moving from the kernel to the GPU?
Some of it. A delegate raises the point and Dawson agrees that unified-memory designs such as Grace Blackwell pull memory handling down toward the GPU. He argues the kernel still turns the devices on and off and often still controls streaming from storage.
About this video
Brian Dawson from CIQ discusses the need for a composable AI infrastructure stack, advocating for an "AI kit" approach over fixed "AI factories." He highlights that the AI technical evolution cycle is moving faster than ever, with every layer of the stack, including hardware, evolving on rapid cycles (one to three months or within a year). This rapid pace pressures organizations to quickly move from research to production, often leading them to adopt rigid "AI factory" solutions. However, Dawson warns that tying into a fixed stack risks locking into something that could become obsolete or antiquated in a matter of months, stressing the importance of building for flexibility.
Brian Dawson, who helps lead product strategy at CIQ, explains that the company is behind Rocky Linux, founded by Gregory Kurtzer, who also co-founded CentOS and created products like Fuzzball and Apptainer. CIQ's expertise is rooted in HPC and Linux, providing a unique approach to assembling the AI stack. Their core thesis is that existing infrastructure is unprepared for AI's demands, necessitating a modern IT stack. Challenges include rising enterprise Linux costs, increased security criticality for valuable AI data, and immense spending on GPU hardware that often sits idle due to difficulties in setting up the surrounding stack. Operations are fragmented, with separate solutions for traditional fleets and GPU nodes. CIQ proposes a new hybrid-first integrated stack, built on a "homogeneous foundation that can be deployed to support heterogeneous use cases," to address these issues, allowing flexibility and avoiding vendor lock-in through open-source components and workload-tuned variants.
The presentation emphasizes the critical, often forgotten, role of the OS layer beneath the GPUs. Dawson explains that the OS, particularly the kernel, controls access to all expensive resources, enabling the workload layer to tap into the compute layer. While some control is shifting towards the GPU, the kernel remains fundamental for tasks like turning devices on and off, configuring them, and managing data streaming from storage. Dawson describes an "AI kit" as something built from the layers an organization needs and run alongside its existing infrastructure, evolving with changing compute (bare metal to Kubernetes) and supporting diverse performance-intensive computing workloads and heterogeneous infrastructure, including on-prem, cloud, VPC, and various processor architectures (XPUs, NPUs, DPUs, TPUs, NVIDIA, AMD). This flexible approach ensures that the infrastructure can adapt to the fast-paced, diverse demands of the AI landscape, providing a stable base while supporting underlying changes.
This video is part of the RLC Pro AI playlist. Browse every CIQ video by product and topic.
Transcript
Hello everybody. I'm Brian Dawson. I'll tell you a little bit more about myself in a moment. Today I'm going to talk to you about the need for a composable AI infrastructure stack, and I'll talk about how we provide for that at CIQ with RLC Pro AI, Fuzzball, as well as our other products. So covering this session, we're going to start with what I say is why people need an AI kit versus an AI factory when everybody today is selling, and to an extent buying, AI factories. Then the role of the OS in maximizing the throughput and utilization of your GPUs. And then we'll talk about, well, once you have that platform established and you have this composable kit with a OS in place, how do you actually deliver your workloads in a reliable, consistent, and trustworthy way?
So why an AI kit versus a factory? I'm going to pause on this title slide for a sec just to talk about some of what I'm observing here. And that is that we're in a technical evolution cycle that is moving faster than I've ever seen in my career, and I think most of us have seen in our careers. Usually, I'd say, there's layers to any technology we deliver, like one or two of those layers are kind of evolving at a time. We establish a foundation, we move on it. We're in this world where every layer of the AI stack is evolving and moving, and in some cases on one to three month cycles.
So that even the hardware is revving inside of a year. And so we're in a world where, under that pressure to deliver, go from research to production in three months, everybody jumped to: somebody just give me a stack, a factory, and I'm going to go deliver AI, and I'm going to deliver inference. But as we'll talk a bit about, the other side of that rapid evolution is when we tie ourselves into a stack, we might lock ourselves into something that is going to be obsolete or antiquated in the next 3 months, in the next 12 months. So, we have to build for flexibility. Now, a bit more about myself. I'm Brian Dawson.
I'm here with CIQ. I have decades, I say decades instead of the actual years, in software and technology delivery. I started as an engineer, moved on. A lot of my time was spent in DevOps. I'm excited that where I started at Sony PlayStation, working on single chips, rapid matrix math software, the geometry transformation engine, is now what we're representing in these GPUs, orders of magnitude more. I remember when I used to get excited about transforming one polygon on a screen. So, now at CIQ, I help lead product strategy, and I oversee product management for a few of our products, couple of which we'll talk about today.
View full transcriptHide full transcript
One is Rocky Linux from CIQ Pro AI, Rocky Linux from CIQ Pro Hardened, and then a just-announced product called the CIQ Stratum Kubernetes Platform, which is a Kubernetes engine as well as a fleet management solution. So, I want to set the stage for CIQ, 'cause it's a lot of the underpinning of this thesis, and this will explain a lot about why I'm excited to be here at CIQ, and excited to have this conversation with you. So, our company is the company behind Rocky Linux. It was founded on years of expertise in HPC and Linux, and I think that is sort of the unique value that we offer in assembling this stack.
It was founded by Gregory Kurtzer. He was one of the co-founders of CentOS. I'm going to assume most of the people in this room are familiar with CentOS. Don't go into much detail, but if you are familiar with it, then you understand how widely adopted it was and how important it was in terms of giving a free and open-source option for running production-grade Linux. And you would actually find in a number of shops that about 10% would be paid RHEL licenses and about 90% would be CentOS. Greg is also the creator of a couple of our products, driven by his 30 years in the Department of Energy and Berkeley National Labs.
He created a product called Fuzzball, which we'll talk about, another product called Warewulf, and Apptainer, formerly known as Singularity. And so he's sort of driven building this foundation in our organization based on delivering performant, resilient, secure infrastructure in order to drive some of the world's most intensive computing and simulations. So our thesis that I'm going to build the rest of this discussion on us is that existing infrastructure is not prepared for the demands of the age of AI. So what we believe is that we need a new modern IT stack that needs to be rethought now versus trying to incrementally iterate when, as I said, the whole layer is rapidly iterating.
And some of the problems we're seeing is that enterprise Linux costs are rising. While Linux is now pretty much commodity for AI, at the last stat I checked, Linux runs 95% of AI workloads. Ubuntu is a Debian distribution that holds about 53% of that, and then enterprise Linux distributions hold the rest of it. But we're also seeing that a lot of vendors are increasing their prices. They're moving to compute-based pricing. They're moving to consumption-based pricing. And that is starting to squeeze those budgets when people have additional areas where they need to spend money. Security is more and more critical. It is critical across the board with AI-accelerated threats.
But then when we also talk about sort of the crown jewels of an organization that we are handling, or as a neocloud or a hosting provider we're handling for our customers, this data is critical. It's valuable. It needs to be protected. People are spending an immense amounts of money on GPU hardware. And oftentimes they haven't figured out the stack, so it's sitting idle. I did a calculation with a customer that we're doing a buildout for. God, I can't remember exact cost of the hardware. But the hardware was purchased, sat on a dock while they waited to figure out networking, cooling in the data center, and then get the infrastructure built up.
Well, in this case, I think they might have mounted in the data center. We're probably not going to start the full buildout until they're fully ready in 2 months. My calculations have it at that is $15 million in GPU hours. This is roughly 9,000 nodes, current generation NVIDIA cards. And for every 30 days that that hardware is not producing value, it's costing them $15.6 million, which was astounding. But I'm hearing that there's a lot of this shortage of GPUs. We can finally get a GPU. We got a GPU. We're writing off huge capital expenditures, but we haven't figured out how to tap them.
And then we're finding that operations are fragmented. We go through this cycle of centralization, decentralization. We are in sort of this explosion of Kubernetes where people are managing traditional fleets. Then we have GPU nodes standing up with different solutions for those. So we're starting to see kind of re-fragmentation of IT that eventually is going to have to be federated again and brought back. So, what do we believe the solution is? It's a new hybrid-first integrated stack. All right, we believe that performant workloads, i.e., this performance-intensive computing that we have to deliver on these GPUs to power cutting-edge workloads, requires a new infrastructure stack, but not only that, the whole environment has a cascading impact on your entire fleet.
Yes. >> Do you really think that one stack is going to cover AI kits and enterprise solutions? That's the challenge. You're talking about the fragmentation of IT, but I don't see a solution here. Is your product a solution? Maybe that's... >> Well, it's a product suite. So, the thing is, is this ready to go to a factory? Yes, I believe there's a solution, right? It is... and I believe we have it or are contributing to it. One is it requires, as we'll talk about, independent products. Two is it requires flexibility in the way that you distribute, and a lot of that flexibility is going to be building on open source and open cores, which makes it easier to integrate with your stack.
You avoid vendor lock-in, but then our other core thesis, one of the reasons I joined the company, is then having variants, as we call them, that we'll talk about, that are specifically tuned for a given workload. They get to 80% of the way there, but yet they have a common basis, right? We'll talk another thing being about being enterprise ready and production ready. In our mind, is using an enterprise-grade, production-grade Linux distribution that your teams are skilled in. >> So one of the things we've been talking about generally, I think we, those of us observing and talking about what's going on with AI and its impact on infrastructure, is increasing diversity and parts of the two to infrastructure down to the system and subsystem level.
Really down at the silicon. >> Right. >> Down to the on-chip. In other words, the way I think of it is multiple vendors, but the way it's usually talked about is all these XPUs, all these new processing units. It's not just about CPU plus GPU. You often have DPUs on the networking cards. You'll have FPGAs or ASICs, like blah blah blah, but that's just extend that farther out to more heterogeneous architectures. >> Yes. >> Don't want to go further on that. I feel like you're suggesting a similar heterogeneity. >> I am. >> In OS. >> Yes.
>> And then what you just described, though, is this sort of like... >> For... >> commonality baseline with these different connected islands of workload-tuned OSes. Is that the... but I love this slide in part because it doesn't have AI on it. >> Right. >> Right. >> So we're going to talk about this... >> Well, I'm going to AI-wash this baby. Don't get it wrong. >> Yeah. So first of all, you're sort of validating this concept of mine that I hadn't really seen before, which is had many layers in the OS layer is what we're talking about, in particularly diversity is helpful and required, and that sort of thing.
>> Yes. >> Okay. >> And I'm going to illustrate some of this. So I'll move on in a minute, but first I want to say yes, that's the way we're seeing it. That's the way I'm seeing it, and I would qualify that there's a world. Keep in mind that one of the things of the power of Linux is that it has every package in the world. You can build for practically every processor architecture. Every vendor that has meaningful hardware commits upstream support for it, right? So, but... >> Yeah, but an operator wants to hook that up and put all those things in and manage it in one place as if it's monolithic.
>> But right, and I think this is where I'll say there's a case. But that is a lot of fat. Now, I think what it is is it's a homogeneous foundation that can be deployed to support heterogeneous use cases. Those above the workloads are running and those that are below it. Does that make sense? So let's talk a little bit about this, right? And unfortunately this got a bit squished as I was working on it. But what we are looking about is, let's just first talk about diversity of performance-intensive computing workloads. Right? And yes, there are, what I would even call database architecture being boring, but right, stuff that needs to just be stable and reliable, but at the same time there are high-performance things demanding, driving needs of that.
Up top, you have your workloads that you need diverse support for. Down below, you have diverse compute and infrastructure. Even before we get down to processor architectures, NPUs versus DPUs versus TPUs versus GPUs, NVIDIA versus AMD, etc., you have on-prem, you have cloud, you have VPC. In AI in particular, you guys may have seen this. People rushed to the cloud because they didn't have GPUs or infrastructure support. Got to be really expensive. I call it the AI hangover. Christmas, you got the bill. You went out and bought GPUs, and now you need to move these workloads over, right? So this hybrid sort of heterogeneous compute that you call out is one of the things.
And then the heterogeneous workloads. And the question becomes how do we deliver orchestration, operation, automation, and overall management of that stack in a way that it supports that diversity? So I'll pause here. We'll sit on this for a second. I'm going to come back to this, but I'm going to go to individual product layers in a minute. So sort of the simplified view and simplified theory is that an AI kit is something that you build from the layers you need that you don't have today, and you run them alongside your existing infrastructure or the components that you want to build and deliver yourself or buy from another vendor.
But what it should be able to do is evolve, right? As your compute underneath changes, you should have a stable layer that doesn't need to change. As you move from bare metal up to Kubernetes-orchestrated workloads, it should be able to support that. And my concern is that an AI factory is one vendor's fixed stack based on their opinion at this particular point in time and on the centralization-decentralization curve, that thing's going to break back apart in a couple of months. Okay? So next I want to talk about the host layer under the GPUs. I'll talk about the fact that it's the OS that controls access to all these expensive resources we've talked about, enables that workload layer up top to tap the compute layer below, but frankly it's a forgotten layer.
I don't know how many AI conferences I've been into in the past few years, how many AI layer cakes we've looked at that completely ignore there's this little thing called the kernel that is responsible for turning your devices on and off, for configuring them. Even when we have direct memory access, you're still engaging with the kernel. Brian, a lot of that control is starting to move from the kernel to the GPU. >> Yes. And this is where I was going to say, even as we get... and I don't know if I'll be able to get into benchmarks, but I recently, and we was benchmarking against GB10s, right?
And so GB10s and GB in general, they have the Grace Blackwell, they have the unified memory. Yes, it pulls a lot of that memory sort of thrashing down, but at the end of the day currently, at minimum, the kernel's turning the switches on and off, let the thing run, right? But also oftentimes, it's still controlling the flow of streaming from storage, and controlling the flow of streaming from other places.
Built for scale. Chosen by the world’s best.
2.75M+
Rocky Linux instances
Being used world wide
90%
Of fortune 100 companies
Use CIQ supported technologies
250k
Avg. monthly downloads
Rocky Linux
9
Enterprise products
Spanning the kernel to the orchestrator
Have questions about your infrastructure?
Talk to a CIQ engineer about Rocky Linux, HPC, and AI infrastructure.