GPUs cost you money every day they wait for the rest of the stack. I ran the numbers with a customer whose hardware arrived before the networking, the cooling, and the software were ready. By my calculation, roughly 9,000 nodes of current-generation NVIDIA hardware cost about $15.6 million in GPU hours for every 30 days they produce nothing [VERIFY: Brian's calculation from the session; no claim file exists for this figure].
Pressure like that is why so many teams ask a vendor for a complete AI factory. I understand the request. I also think it creates the next problem, because the stack you buy whole today is built from layers that will each change within the year.
Build your AI infrastructure as a kit instead. Add independent layers where you have a gap, keep the layers that already work, and make sure you can replace any one of them when the hardware or the workload changes.
I made this argument in the first part of my session at AI Infrastructure Field Day. You can watch that segment.
Plan for every layer to change within a year
In most technology changes I've worked through, one or two layers change at a time. You settle the foundation and build on it. AI infrastructure isn't working that way. Every layer is changing at once, some on one-month to three-month cycles, and the hardware itself is revised within a year. I haven't seen a cycle this fast in my career.
At the same time, teams are asked to go from research to production in about three months. So they ask for one stack from one vendor and start serving inference.
An AI factory answers that request well. It gets you running. The cost shows up later, when one layer changes and the rest of the factory was built around the old one. In three months, or in 12, you can be locked into parts that are already out of date.
An AI factory is one vendor's fixed stack, built on that vendor's opinion of AI infrastructure at a single point in time.
Buy the layers you're missing and keep the ones that work
A kit is built from the layers you need and don't have today. You run those layers next to the infrastructure you already operate, the components you'd rather build yourself, and the ones you buy from someone else.
A kit has to keep working when the hardware and the workloads change. When the compute your workloads run on changes, the layer your team operates every day should stay the same. When you go from bare metal to workloads orchestrated on Kubernetes, that layer should support both.
A delegate at Field Day asked me directly whether I think one stack can cover all of this, and whether CIQ has a solution or a product. My answer was that it takes a suite of independent products. Each one has to be useful without the others, and you have to be able to integrate it with what you run and replace it if you need to.
Keep one foundation for mixed hardware and mixed workloads
The hardware AI workloads run on is getting more varied. It's no longer only a CPU and a GPU. You have DPUs on the network cards, NPUs and TPUs, FPGAs and ASICs, and accelerators from more than one vendor. You also have more than one place to run: on-premises, cloud, and virtual private cloud.
I call one version of this the AI hangover. Teams rushed to the cloud because they had no GPUs of their own. The bill arrived. They bought GPUs, and now the same workloads have to run on-premises too.
The workloads vary as much as the hardware. Some need to be stable above everything else. Others need every bit of performance the node can give.
Another delegate pointed out that an operator wants to manage all of that from one place, as if it were a single system. I agree with the goal. I don't think one system that contains every package for every case is the way to get there, because that's a lot of extra software on every node.
A common Enterprise Linux foundation your team knows, with versions tuned for each workload, supports mixed hardware and mixed workloads.
Start with the layer most AI diagrams leave out
I've sat through a lot of AI conference talks in the past few years, and most of the stack diagrams skip the operating system. The kernel still turns your devices on and off, configures them, and controls how data streams in from storage. Newer architectures with unified memory reduce some of that work. They don't remove the kernel from the path.
That's why the kit I'd build starts at the operating system and goes up from there:
- The operating system. Rocky Linux from CIQ Pro (RLC Pro) is the supported foundation. Rocky Linux from CIQ Pro AI (RLC Pro AI) is the version tuned for GPU workloads, and Rocky Linux from CIQ Pro Hardened (RLC Pro Hardened) is the version for regulated environments.
- Kubernetes. CIQ Stratum Kubernetes Engine (SKE) is an open source Kubernetes engine, and CIQ Stratum Pro adds fleet management across SKE clusters.
- Workflow orchestration. Fuzzball runs high-performance computing (HPC) and AI workflows across on-premises clusters and cloud.
A tuned version of the operating system gets you most of the way to a working node for a given workload. You add the layer that's specific to your application on top. You can take one of these products and none of the others, and I'd rather you did that than buy a layer you have in place.
If you want to see how the operating system layer fits into a kit, the second segment of the session covers it in detail.
In a market that changes this fast, expecting anyone to replace their whole stack at once is unreasonable. Pick the layer where you're losing the most time or the most GPU hours, and start there.
See how RLC Pro AI handles the operating system layer for GPU nodes.
