Distributed AI Workflows with Fuzzball and CIQ
In this segment of the CIQ session at AI Infrastructure Field Day 6, Brian Dawson, Director of Product Management at CIQ, covers the orchestration layer of the stack: Fuzzball. A modern compute stack for AI has to be addressed at several layers, from getting value out of expensive hardware to running workloads reliably, and orchestration, operation, and automation sit in the middle. Fuzzball runs on top of Kubernetes in that layer, on CIQ Stratum Pro or a generic Kubernetes distribution, and it grew out of traditional HPC organizations needing to modernize their stacks so students and researchers could deploy more, faster.
The core idea is that a user describes a job once, including the compute it needs, the container images it runs, and the storage it reads and writes, in a single workflow file, and that file runs unchanged on premises, in public cloud, or on a GPU cloud. The same control plane that orchestrates a job federates across clusters and compute resources, so one definition lands wherever capacity is. Fuzzball began as a platform for batch HPC and stateless jobs, and now supports stateful AI workloads too, which matters for the fleets Dawson hears about every few weeks: an HPC cluster, general Kubernetes, and new demand for generative AI that the HPC cluster was never built for. He also shows the workflow catalog, where a job described in YAML runs from a run button, and some customers white-label the catalog for their own users.
A delegate asks about checkpointing, and Dawson confirms Fuzzball supports it. On the operations side, he describes fault-tolerant scheduling that restarts failed jobs elsewhere, multi-tenancy, resource accounting, and AMD and NVIDIA GPUs under a single control plane, along with the hardening production needs: federation, autoscaling, and managing the Kubernetes certificates that otherwise need constant refreshing. Asked how secrets, identity federation, and per-job isolation work across clouds, Dawson defers the detail to the lead engineer. He closes by placing Fuzzball in the wider CIQ stack, which he describes as parts that can be adopted one at a time rather than as an AI factory, and points viewers to a CIQ Stratum Pro webinar.
Key takeaways
- Fuzzball sits in the orchestration layer and runs on top of Kubernetes, on CIQ Stratum Pro or a generic distribution, to modernize traditional HPC stacks and speed up distributed workload deployment.
- A single workflow file describes compute, images, and storage, and runs unchanged on premises, in public cloud, or on a GPU cloud.
- The control plane that orchestrates a job also federates across clusters and compute resources, so one definition runs wherever capacity is available.
- Fuzzball started with batch HPC and stateless jobs and now supports stateful AI workloads as well.
- Checkpointing is supported, and a workflow catalog starts a job described in YAML from a run button.
- Operations get fault-tolerant scheduling that restarts failed jobs elsewhere, multi-tenancy, resource accounting, and AMD and NVIDIA GPUs under one control plane.
- Production hardening covers federation, autoscaling, and Kubernetes certificate management.
Questions this video answers
What is Fuzzball?
Fuzzball is the CIQ workflow orchestration platform for HPC and AI. It runs on top of Kubernetes, takes a job described once in a workflow file, and runs it on premises, in public cloud, or on a GPU cloud without rewriting the workflow.
How does Fuzzball handle both HPC and AI workloads?
It was built for batch HPC and stateless jobs and has grown to support stateful AI workloads, so one platform can serve a fleet that mixes HPC, general Kubernetes, and generative AI demand.
Does Fuzzball support checkpointing?
Yes. A delegate asks directly and Dawson confirms it. Fuzzball can also restart a failed job on other available compute, and CIQ is building finer-grained GPU resiliency, on its own and with partners.
Does Fuzzball support AMD and NVIDIA GPUs?
Yes. Dawson describes both AMD and NVIDIA GPUs running under a single Fuzzball control plane, alongside multi-tenancy and resource accounting.
How does Fuzzball fit with the rest of the CIQ stack?
Fuzzball is the orchestration layer. In his introduction, Dawson names RLC Pro AI, RLC Pro Hardened, and CIQ Stratum Kubernetes Engine (SKE) as the products he oversees, with CIQ Stratum Pro as the commercial version built on SKE. He presents them as composable parts rather than a single packaged AI factory.
About this video
This presentation by CIQ at AI Infrastructure Field Day 6 introduces Fuzzball as a solution for orchestrating distributed AI workflows. Brian Dawson from CIQ explains that modern compute stacks, especially for AI, require addressing multiple layers from hardware capitalization to reliable workload support, including orchestration, operation, and automation. Fuzzball sits in the orchestration layer, running on top of Kubernetes, to modernize traditional HPC organizations' stacks and accelerate distributed workload deployment.
A core Fuzzball capability is to let users run the same workflow on-premises, in public cloud, or on a GPU cloud without rewriting it, enabling federation across clusters and compute resources. Initially designed for batch HPC and stateless jobs, it has evolved to support both stateful and stateless AI workloads, addressing the challenge of unifying diverse fleets that may include HPC, general Kubernetes, and Gen AI demands. A key feature is the ability to describe a job, including compute, images, and storage, in a single workflow file, making it portable. The solution also supports checkpointing, and offers a workflow catalog for easy deployment of jobs via a "run" button.
Operationally, Fuzzball provides fault-tolerant scheduling, the ability to restart failed jobs, multi-tenancy, resource accounting, and support for both AMD and NVIDIA GPUs under a single control plane. It emphasizes operational hardening, including federation, auto-scaling, and managing Kubernetes certificates. CIQ aims to deliver a complete infrastructure stack that allows organizations to maximize Linux and GPU access, ensures built-in security, and provides an automated stack across diverse fleets, without demanding a full "AI factory" adoption. They encourage exploring their products, including CIQ Stratum Pro, and point to free downloads, trials, and developer licenses.
This video is part of the Fuzzball playlist. Browse every CIQ video by product and topic.
Transcript
I'm going to talk to you about distributed AI workflows with Fuzzball. I'm Brian Dawson from CIQ. I've spent a few years in software development and technology with a lot of focus on DevOps, working across the spectrum, as an engineer in the early days at PlayStation to product, where I function now. I help oversee product strategy as well as product management at CIQ around our Rocky Linux from CIQ Pro AI product, our Rocky Linux from CIQ Pro Hardened security product, and our recently announced CIQ Stratum Kubernetes Engine and platform.
So we've been talking about this idea of a stack. When you encounter establishing any modern compute stack, but in particular an AI compute stack, you have multiple layers that you have to address, all the way to capitalize on your hardware while ensuring that you could support what your respective workload is reliably. And that goes all the way from orchestration of those workloads, operation of the underlying hardware and systems, automation for managing your day-to-day operations, and then continued management, management either around continued setup or maintenance. And so what I'm going to talk about here is Fuzzball. So Fuzzball sits at our orchestration layer.
It runs on top of Kubernetes, whether it's Stratum Pro or it's a generic Kubernetes distribution. It leverages etcd. And the initial goal was, as we talked about customers and we're moving to Docker containers, traditional HPC organizations needed to modernize their stacks and service the ability to support their students, their researchers, faster so that they can deploy more. That we created, or Greg created, Fuzzball. Fuzzball uses Kubernetes and some other tooling to replace a traditional Slurm or MPI scheduler to enable you to stand up a distributed workload and run that workload across any, let me see if this is actually going to work the way I want it to work.
So you can run the same workflow on premise, in public, or on a GPU cloud, all without rewriting it. And if time allows, we'll pop into a quick demo. Right? So we go from a control plane that allows you to orchestrate, and then that same control plane that will allow you to federate across clusters and across compute. Let's see if my, So, this is one platform, and it's actually, I have a typo here. I shouldn't put that there. That's one platform for batch HPC pause and AI. So, supporting traditional batch jobs and traditional stateless jobs. But what we've done over the past few years, realizing the enterprise need, is that we've extended this to support stateful as well as stateless AI workloads that aren't running on traditional infrastructure.
View full transcriptHide full transcript
And what we are seeing in diverse fleets today, I'm running into this, having this conversation like every couple of weeks, is, hey, I have to run my HPC cluster. I have to support general Kubernetes, but we were also getting demands for support for Gen AI. And we didn't build out our HPC clusters for inferencing and Gen AI. So now they're standing up these different stacks. And Fuzzball is one of those things that helps unify it. We allow you to describe a job in one workflow file. That's the compute. That's the images that it's going to deploy. That's the storage, all in one. And then you can then run that anywhere.
>> Do you support checkpointing as a feature? >> Yes. And then, but we're also working, I know you're probably seeing or heard solutions about, I'll just call them subcheckpoints. I don't know the name, but continuous tracking. Yes, we support checkpointing. We are also building out solutions ourselves and with partners that give you finer, that we'll call it, GPU resiliency, right? >> So the sort of pride and joy of this, and we probably won't get time to pull it up, is sort of the workflow catalog, right? So this allows me to take a job, describe it in YAML, offer it to my customers, my clients. We have people white labeling and repackaging this to allow them to then run that job with just pushing that green run button without worrying about the compute or location.
To support movement to production, we've enabled agentic integration. So there is an MCP. To your question, fault-tolerant scheduling here, traditional checkpointing versus kind of managing in contiguous fabric. But we are working on that. We can currently restart failed jobs elsewhere. This goes back to someone asked, not exactly what you were asking about, optimizing an app, but this other layer of utilization and optimization. How do I keep a job running and use as much as my available compute as I can? Supports multi-tenancy, resource accounting, AMD ROCm and NVIDIA GPUs under a single control plane.
This is a theme with us. And then overall operational hardening, right? So federation, autoscaling, managing your certificates. The big pain with Kubernetes is continually refreshing certificates. >> Just zooming in on the operational hardening, can you expand further in terms of, across cloud, how is the secrets or the identity federation >> are handled, or the per job isolation are handled? >> Well, I couldn't, we can probably take some time afterwards. I couldn't speak to it in detail, but I'd be happy also to get the lead engineer on this to, I can say very inadequately because I've been on calls where that was answered.
They are handled, but I can't answer it myself. Follow on that, like signature stuff, is that if you're using Sigstore or, >> I do not know if we're using Sigstore. >> Okay. I know access and authentication, very inadequate on access and authentication. It's managed by Keycloak. We are also separately integrating this into a single control plane across all of our products. It can deploy with all of our products, but you have different control planes. So, this is just an example of the dashboard, right? If I was to just quickly here go hello world, which I won't, I'll show it and I'll hit play, but then we're going to run out of time.
You can see all of these lamps. I'm just going to hit her, I can send out a Hermes agent, deploy that agent, expose the endpoints, and offer that as a service or use those endpoints for telemetry. And in this particular case, it's very simple, but this will run a hello world job, deploy it on available compute, and print that out to the console. See, oops, I can't jump forward when I'm not on the slide. So what I look to show you here is that there's a need to think about the complete infrastructure stack.
We can deliver the complete infrastructure stack, but yet we don't look to deliver it in a way where we require that you take on an AI factory. I would hope that all of our products fit a need and you can deploy them. But in a world where things are evolving and you need to move fast, expecting that someone can take the whole stack is unreasonable. What we look to do is solve the problems of modern infrastructure: enable you to get the most out of Linux, access your GPUs from day one, not have to worry about security because we built it in for you, and deliver an automated stack across your diverse fleet.
We have a webinar on CIQ Stratum Pro tomorrow, our Kubernetes engine. People, please check it out. You can try our products at our customer portal. We have free downloads and trials. We have developer licenses, as well as we have self-service support, and to the professional services and support, we have a first-class support team. So thank you.
Built for scale. Chosen by the world’s best.
2.75M+
Rocky Linux instances
Being used world wide
90%
Of fortune 100 companies
Use CIQ supported technologies
250k
Avg. monthly downloads
Rocky Linux
9
Enterprise products
Spanning the kernel to the orchestrator
Have questions about your infrastructure?
Talk to a CIQ engineer about Rocky Linux, HPC, and AI infrastructure.