A university research computing team recovers a failed node in 30 minutes with Warewulf Pro

The team reached that point after standardizing its flagship cluster on three products from CIQ: RLC Pro, Warewulf Pro, and Apptainer. The cluster was originally built on CentOS 7 and managed by Bright Cluster Manager. Moving to an open source foundation meant untangling a live scheduler and re-provisioning a cluster that hundreds of researchers depend on every day.

Today the team runs its whole stack from a single golden image process, so compute nodes stay online for weeks and version drift disappears.

The flagship cluster carries 222 compute nodes and several thousand cores, with a typical node at 40 cores and larger nodes above that. A few hundred researchers run on it, from across the university: digital art, economics, engineering, computer science, bioinformatics, genomics, and the wider life sciences.

The workloads span the same range. Some researchers submit heavy MPI multi-node jobs. Others run interactive sessions or scripted batch pipelines. Some treat a 40-core node as a very large workstation. AI workloads are the fastest-growing category, and the GPU inventory backs that up: dozens of NVIDIA V100s in single, dual, and quad-GPU configurations, plus A100, L40S, and GH200 Grace Hopper GPUs. Researchers run a large volume of open-weight LLM workloads on top of that hardware. Outside the shared clusters, the team also supports standalone servers that individual labs buy and run on their own, mostly on Rocky Linux 9 with a mix of GPUs.

A team of just over a dozen people supports all of it, including network engineers, HPC engineers, and cyberinfrastructure specialists who work across the software stack, image builds, and day-to-day administration.

The team migrated from Bright Cluster Manager to Warewulf Pro around 2022. Three factors drove the decision. The subscription cost had risen to a level the team could not justify. Support during an urgent incident did not match what the team expected. And the team wanted to move toward an open source foundation it could understand and shape.

The hardest part was removing Bright, not deploying Warewulf Pro. Bright was entangled with the cluster’s Slurm workload manager, and Slurm integration is specific to the version in use and how it is wired in. An engineer on the team did the work to disentangle Bright from Slurm and stand Warewulf Pro up in its place.

That migration changed how the team relates to its own infrastructure. Bright ran as a white-glove system the team was told to leave alone. Warewulf Pro gave them direct insight into how provisioning works, so when a small issue appears now, they can find and fix it quickly.

The choice of Warewulf also rested on its community and its track record. The team’s research computing lead read the Warewulf documentation and decided it was the right fit. The project’s activity on Slack and GitHub, its role in the OpenHPC stack, and its adoption across peer institutions gave the team confidence. The team pairs the open source project with Warewulf Pro and a CIQ support subscription, which adds a web UI option on top of the command-line workflow the team already trusts.

Apptainer ends the dependency chases that used to stall researchers, and it turns fraught software stacks into a single import step. An economics professor at the university works in R, where library and dependency management can be difficult. The Rocker community maintains a Docker image for R that resolves those dependencies. With Apptainer, the team imports that image directly from its Docker Hub repository and the environment is ready to run.

Before Apptainer, some of those software stacks weren’t attempted by hand at all. The more complicated dependency chains could eat the better part of a week. The same environment now imports through Apptainer in minutes to an hour or two. The team uses the same approach for tools from NVIDIA’s NGC container registry, importing them with Apptainer so researchers can get to work.

The team moved to Apptainer from a small amount of Singularity. A handful of research groups now build their own containers, and the team is actively encouraging more groups to make the move.

The team chose Rocky Linux as its operating system foundation and found the path straightforward. It started the cluster on CentOS 7, ran early Warewulf images on CentOS 7, then migrated those images to Rocky Linux 8. The transition was smooth, with different libraries but nothing insurmountable. Concern about the direction of CentOS Stream confirmed the decision to move to Rocky Linux.

Today the team runs a mix of Rocky Linux 8 and Rocky Linux 9 across head, compute, and login nodes, with most user-facing nodes on Rocky Linux 8. RLC Pro has been stable, compatible with the team’s prior CentOS environment, and easy to manage with DNF. The team is now evaluating RLC Pro’s additional kernel builds: by enabling the CIQ repository on an existing Rocky Linux 9 instance, it can pull CIQ’s kernel builds. The interest there is specific, patches that keep pace with the growing number of kernel vulnerabilities coming out of AI-assisted security research. For head nodes, a support subscription is an easy call.

Running RLC Pro, Warewulf Pro, and Apptainer together changed daily operations for the team. Compute nodes stay online far longer, because a golden image provisioned by Warewulf Pro removes the version drift and reinstalls that used to consume time. When a node does hit a problem, the team reboots it from the golden image and restores service in about 30 minutes, rather than spending hours tracking down the cause. That reliability translates directly into research output, because more uptime means more jobs completed.

Researchers feel the container side most directly. Software that once took the better part of a week to set up by hand, or wasn’t attempted at all, now imports through Apptainer in minutes to an hour or two. Warewulf Pro, by design, stays invisible to them, which is exactly what the team wants. On cost, the university removed its Bright support spend and moved to CIQ support, and the complaints about support quality that prompted the change have not returned.

The team now runs all three products as one workflow, and expects that combination to open the door to faster, more automated deployments. The next step is image management as code. The team is looking at the Warewulf Pro API to automate golden image builds and push kernel updates and refreshed images more often, matching the pace of kernel patching the team is tracking in 2026.

Asked what they would tell peers, the team led with the result: the move has been a success across RLC Pro, Warewulf Pro, and Apptainer, and standing up a Warewulf cluster from scratch was straightforward. The team didn’t have a development cluster when it first dropped Bright Cluster Manager. It has built one since, and the advice its lead HPC engineer gives peers is to build a small development cluster first, even on Raspberry Pis if that’s what’s available, before making changes on a production system.

See how a university research computing team cut node recovery to about 30 minutes and ended dependency chases for its researchers on RLC Pro, Warewulf Pro, and Apptainer.