Fuzzball + Slurm/PBS: Container orchestration meets traditional HPC schedulers

At SC25 in St. Louis, I had the opportunity to showcase one of the most significant new additions to Fuzzball's capabilities: native integration with PBS Professional and Slurm workload managers. This isn't just another feature. It's a bridge between decades of HPC operational expertise and the infrastructure flexibility that modern AI and containerized workloads demand.

But what is Fuzzball?

CIQ has a long and successful history bringing Linux containers to the world of HPC. Fuzzball is a container orchestration platform built by HPC engineers specifically for HPC users and admins. But Fuzzball is a lot more than that. Once you realize that you can orchestrate containers within HPC, you realize that's all you need to run and use your cluster. Fuzzball has grown from just container orchestration in HPC to an entire cluster management and usage paradigm. To learn more about Fuzzball, check the whitepaper here.

The constant question

I’ve been attending SC with CIQ since 2022 and talking to people about Fuzzball. Every time we start those conversations, the same question surfaces: "How does Fuzzball integrate with my existing batch scheduler?"

That’s interesting. Nobody asks if Fuzzball integrates with their scheduler. They just ask how. It's assumed that integration is going to work.

Unfortunately, for the past several years, our answer has been: it doesn't. We expected organizations to run Fuzzball by itself to manage their scheduling queue. And as you can probably imagine, that's been a rather unsatisfying answer to a lot of folks. You're telling people they have to pull out Slurm, tell all their users to throw away their batch submission scripts, and start from scratch with Fuzzball. Huh?!

We got the message. So we asked ourselves: why can't Fuzzball just use Slurm or PBS to provision nodes? On AWS, Fuzzzball just calls out to the API and says, "Give me some nodes." Why can't we treat Slurm and PBS like cloud providers?

That's exactly what we're doing with the native integration between Fuzzball and Slurm / PBS!

The solution: Fuzzball provisioner configuration

Our new integration allows PBS and Slurm to act as backend provisioners for Fuzzball workloads. Think of Fuzzball as an interface that sits above your existing scheduler, giving you modern container orchestration and AI-ready workflow capabilities while preserving everything your users already know.

The yaml examples in this section are snippets of full yaml files. For more information, see the provisioner configuration documentation.

Choose your provisioner(s)

We've been working on a new provisioner configuration system that allows you to choose between different provisioner backends:

• Static provisioner: Use Fuzzball itself to provision nodes
• AWS: Provision cloud resources
• Slurm or PBS: Use your existing scheduler as the backend

Here's what that looks like in practice. In this example, there is a static provisioner defined for bare metal nodes (with optional annotations), an AWS provisioner available for spot instances, and a third provisioner for a high memory cluster that is managed by Slurm.

definitions:
  # Static bare-metal nodes
  - id: compute-standard
    provisioner: static
    annotations:
      node.type: "compute"
      performance.tier: "standard"
    
  # AWS for cloud burst
  - id: aws-spot
    provisioner: aws
    
  # Your existing Slurm cluster
  - id: slurm-highmem
    provisioner: slurm

You can mix and match different provisioners or allow a single provisioner to allocate all of your hardware. The system is flexible and very powerful. In a real configuration, details about hardware (node pools) and provisioner policies (policy expressions) would follow. We’ll cover those concepts next.

Specify your node pools

Within these yaml files, you can fully define your compute resources by specifying things like CPU and GPU count, and you can attach a cost to track usage. You can say, "I've got a pool of CPU nodes, I've got a pool of GPU nodes, I've got a pool of high-memory nodes," and specifying the cost per node will ultimately allow you to track how much your users, labs, or groups are consuming at the end of the month.

Here's an example for statically provisioned nodes and PBS nodes, both standard compute and GPU nodes.

Note the conditions under the provisionerSpec yaml.

definitions:
  # Standard compute through PBS
  - id: compute
    provisioner: static
    annotations:
      node.type: "compute"
      performance.tier: "standard"
    provisionerSpec:
      condition: |-
        hostname() matches "compute-[0-9]{3}" &&
        cpuinfo.vendor_id == "GenuineIntel" &&
        cpuinfo.cpu_cores >= 16 &&
        osrelease.id == "ubuntu"
      costPerHour: 0.40
  
  # GPU nodes for ML workloads
  - id: gpu
    provisioner: static
    annotations:
      node.type: "gpu"
      gpu.type: "nvidia"
    provisionerSpec:
      condition: |-
        hostname() matches "gpu-[0-9]{2}" &&
        modalias.match("pci:v000010DEd*sv*sd*bc03sc*i*")
      costPerHour: 3.20

  # Standard PBS compute through PBS
  - id: pbs-standard
    provisioner: pbs
    annotations:
      node.type: "compute"
      performance.tier: "standard"
    provisionerSpec:
      queue: "standard"
      costPerHour: 0.45
  
  # GPU nodes for ML workloads
  - id: pbs-gpu-ml
    provisioner: pbs
    annotations:
      node.type: "gpu"
      gpu.type: "nvidia"
    provisionerSpec:
      queue: "gpu"
      gpus: 1
      costPerHour: 3.30

You can get very specific about the types of nodes in these pools. You can match hostname patterns, but you can also match the vendor of the CPU, the number of cores, even the operating system. For GPUs, you can even query the PCI bus and specify exactly what kind of GPU is attached to specific node pools.

Set policy expressions

Policy expressions allow you to implement sophisticated scheduling logic. You can say things like: "This user or this group is allowed to use this type of node under these conditions. When they've made this type of request, we're going to allow them to access this node pool." In the following example, GPU nodes can be requested by users in the AI/ML organization when they specify one or more GPUs are needed as devices, high-memory nodes can be allocated to users when they request greater than 128GBs memory, spot instances will be allocated to users who request cost optimized services, and a job annotation can be used to request older nodes in a legacy cluster.

definitions:
  # GPU nodes only for AI/ML org with GPU requests
  - id: gpu-ml
    provisioner: pbs
    policy: |-
      request.job_resource.devices["nvidia.com/gpu"] > 0 &&
      request.owner.organization_id == "ai-ml"
  
  # High-memory jobs to specialized nodes  
  - id: highmem
    provisioner: slurm
    policy: |-
      request.job_resource.mem.bytes >= (128 * 1024 * 1024 * 1024)
  
  # Cost-optimized workloads to spot instances
  - id: aws-spot
    provisioner: aws
    policy: |-
      request.job_annotations["cost.optimization"] == "enabled" &&
      request.job_kind == "service"
  
  # Legacy workloads stay on traditional cluster
  - id: legacy
    provisioner: pbs
    policy: |-
      request.job_annotations["workload.type"] == "legacy"

This policy-driven approach means you can route containerized AI workloads to GPU resources, burst overflow to the cloud, and keep traditional HPC jobs on existing infrastructure, all without users needing to understand the underlying complexity.

Keep your scheduler

This new integration means you can keep your existing scheduler! If you're happy with Slurm or PBS but you want to start modernizing as well, your users can continue to submit jobs via Slurm. Then new users can start playing with Fuzzball, and existing users can mix and match. They can start using Fuzzball when it's convenient to do so.

Instead of managing a big, painful cutover, you can integrate and start leaning into Fuzzball gradually.

AI-ready HPC (without rip-and-replace)

Organizations are increasingly expected to support AI workloads, but organizations can't justify the disruption and cost of replacing proven HPC infrastructure.

With Fuzzball's PBS/Slurm integration, you can:

• Scale GPU-intensive AI/ML workloads using modern container orchestration
• Maintain operational continuity for traditional research computing users
• Implement hybrid cloud strategies without wholesale infrastructure changes
• Gain unified visibility across all workload types and resource pools
• Lower adoption risk by eliminating retraining costs and workflow rewrites

This isn't about choosing between HPC heritage and AI-era capabilities. It's about having both!

Other improvements over the past year

Beyond the Slurm/PBS integration, we've been very busy! Stay tuned for more announcements and blog posts detailing the following:

Nextflow integration: Because of a plugin we've written, Nextflow can now use Fuzzball as a backend to run its jobs.

Service endpoints: Want to spin up a Jupyter notebook instance, a remote desktop, or an AI inference server? It's basically a push-button. You spin it up, a button pops up in your browser, you connect to it, and you're good to go.

Scheduler improvements: We've improved our own scheduling system so it will reuse resources intelligently and not have to bring them up and bring them down every time.

User experience improvements: A lot of user-facing improvements that make the platform easier to use and more intuitive.

AWS marketplace: We've released Fuzzball in the AWS marketplace, making it easier to spin up and test the platform.

Getting started

The PBS and Slurm provisioner integration is available now for Fuzzball deployments and is fully documented. We're also creating a user interface that allows you to configure your provisioner more easily. Organizations can configure their existing schedulers as backend resource providers while adding Fuzzball's workflow catalog, container orchestration, and hybrid cloud capabilities, all without disrupting current user workflows.

We built this integration because HPC centers shouldn't have to choose between innovation and stability. Your infrastructure investments aren't obsolete. They're the foundation for what comes next!

Want to learn more about bringing AI-era infrastructure capabilities to your traditional HPC cluster? Visit ciq.com/products/fuzzball or reach out to our team at info@ciq.com.

---

David Godlove is a Senior HPC Engineer at CIQ, where he works on Fuzzball's container orchestration platform and contributes to the Apptainer project.