Setting up Apptainer in a Warewulf cluster with CIQ's Brian Phan
CIQ's Brian Phan walks through a small but common problem when running Apptainer on Warewulf-provisioned compute nodes: fakeroot fails because the user has no entry in /etc/subuid and /etc/subgid on the node. The demo starts with the lab architecture, a head node on an external network, an internal cluster network where Warewulf provisions nodes and Slurm services run, and the typical IPMI network for remote power control, plus a quick explanation of Slurm as a workload scheduler and slurmdbd for job accounting reports.
As a test user, Phan launches an interactive job, shells into a Rocky Linux container, and reproduces the warning about the missing user mapping. He then shows the fix in Warewulf: an overlay containing subuid and subgid files with the .ww suffix, which marks them as templates that include the matching files from the controller node. Adding that overlay to the default profile's runtime overlays with wwctl profile set, then rebuilding, pushes the files to the node without a reboot, and a fresh interactive job runs fakeroot cleanly.
This is a hands-on walkthrough for HPC administrators who want consistent user namespace mappings across a cluster.
Key takeaways
- Apptainer fakeroot needs the user mapped in /etc/subuid and /etc/subgid on the compute node, not just on the head node.
- Warewulf files ending in .ww are templates; the demo template includes the controller's subuid file so nodes stay consistent with the head node.
- Adding the overlay to the default profile's runtime overlays with wwctl profile set delivers it to nodes without a reboot.
- The wwctl node status command shows which runtime overlays each compute node is currently pulling.
- Running slurmdbd stores job accounting data so admins can report cluster usage and demonstrate value to management.
Questions this video answers
Why does Apptainer fakeroot fail on a Warewulf compute node?
Fakeroot relies on user namespace mappings in /etc/subuid and /etc/subgid. The head node has a mapping for the user, but a freshly provisioned compute node does not, so Apptainer warns that the user is not mapped and rootless features are limited. Distributing those files through a Warewulf overlay resolves it.
How do you distribute /etc/subuid to Warewulf nodes?
Create an overlay containing etc/subuid.ww and etc/subgid.ww templates that include the corresponding files from the Warewulf controller. Add the overlay to the runtime overlays of the profile your nodes use with wwctl profile set, rebuild the overlay, and the nodes pick it up as part of their regular runtime overlay refresh.
This video is part of the Warewulf Pro playlist. Browse every CIQ video by product and topic.
Transcript
let's jump into uh I guess a basic uh just cluster architecture so uh this is kind of the architecture that we're working with uh in our HPC lab uh so uh how I've set up this environment is uh we have a head node that's connected to some external network that I'm going to be sshing into uh this head node is connected to an internal cluster Network which um Warewulf will be doing um provisioning on and uh slur the various slurm services are also running on this internal cluster network uh addition uh not in this environment but this is very typical with on Prem environments as
you'd also have an ipmi network uh what this is meant uh to do is this enables you to control your compute nodes uh from your head node and be able to power them on and off uh without actually going into your data center and uh hitting a button uh which makes it really convenient uh and I I've you know if you're assist admin you know the feeling of going having to go into the data center and actually hit a button is very annoying and you know if this is a way to prevent you from having to do that uh so uh and then lastly uh
as part of this environment we just have a compute node that is running slurm d uh this is basically the slurm client which will will talk to the controller to either uh basically accept work and uh communicate with the controller um and there's one more service that I have on here that I didn't mention slurm DVD is basically the slurm database statement uh this database statement is basically responsible for uh writing accounting job accounting data to a MySQL or badb database uh and it also uh is can store job completion data as well so uh you would want to run this in your environment if
you want to uh do some reporting on the usage of your cluster stuff like that and uh by being able to generate these reports you can take these reports show them to your management and show them the value that your equipment is actually delivering uh for your organization so real quick Brian I know we the title of this is Rocky ater Warewulf I know we have stuff up here about slurm do you want to give just a real quick and then talked about what from a reporting perspective you want just kind of real quick high level what is slurm oh yeah so slurm is basically
View full transcriptHide full transcript
a uh workflow um workflow scheduler so basically it offers an interface for users to submit workflows either in batch mode or interactively and uh have these workflows run on your uh compute cluster and uh basically uh slurm has like a lot of features where you could optimize uh you can optimize slurm configuration for your workflows whether it's from uh setting resource limits uh implementing uh access policies all that type of stuff can be implemented in slurm and also tracked as well on this bottom prompt uh I've basically just set up a test user and uh within this cluster I have slur running so if I
do an S info here you can see I have one node up HPC lab compute zero uh so uh as a user um on a cluster some something I would try to do is uh create a uh set up a interactive job which lands me onto a compute node so that I could try you know maybe building my software and testing it out on some of my data uh and this is just for testing purposes before uh I start launching a bunch of workflows into the cluster and I just want to make sure my stuff works and that my software is working so uh the
typical workflow that I would do is I would start off with a interactive job uh and this should put me on a CL put me on compute node zero uh and right now I am going to try to do and if we take a look at my home directory I basically have this Warewulf Rocky Linux a uh container uh that I'm just going to try to Shell into uh with fake R so sh add a fake rout here or first we can we can start with just showing going into it normally if we do who am I you can see that I'm the same user
so if I exit out of this and I try to use fake rout it gives me um a warning that um I basically don't my user uh I basically don't have a user mapping for this user in uh /c sub uid basically and the purpose of this demo is to show you how to get these files um basically onto your compute nodes uh through Warewulf uh so uh and because I don't have this user mapping uh I don't have the basically in a the fake root context that I'm in right now it has doesn't have the full capability of um Apptainer's rootless um I guess
C feature basically uh so uh let's back out of this I'm just going to exit and let's get out of this Interactive job and let's let's see how I'm actually going to implement this so taking a look um at into Warewulf now um I basically have this overlay set up called Apptainer fake group and within this overlay I basically have uh SLC Sub uid sub uid and subg uh with this ww suffix uh on it and you might be wondering what is the suffix uh on this file uh the suffix basically denotes that this file is a uh Warewulf template so uh let's actually take
a look at what this uh template actually is so if I C this file and let's just take a look at some and basically catting this file uh within this overlay you can see that um I have this line which includes uh the file on my this specific file on my controller node and if we take a look at that actual file on my controller node we can see that we have a mapping for um test user um that's already here so by having this file on my compute node um I should be able to use the full featured fake gr feature um so to
do that let's first take a so I also want to show you uh the node that I am the compute node within Warewulf as well so as you can see here um what I would like to point out um in this line here is the runtime overlays um are coming from this default uh node profile which currently just contains the generic overlay which contains things like uh your users your groups and all that type of stuff uh um so how do we actually get this overlay onto the node so to do that uh I am going to add this overlay as a runtime overlay on
the default node profile so to do that uh we are going to do set and then I'm going to add group so I'm going to run this uh Warewulf control profile set command which will set um the runtime overlays of the default profile to generic and Apptainer gr so do that it'll ask me to confirm yes so uh I also want to show you uh you can also see what um runtime overlays your uh compute nodes are pulling by running the Warewulf node status command and as you can see HPC lab compute node zero is pulling a runtime overlay which just contains this generic um
overlay um so after I've set um this overlay runtime overlay on the profile I can quickly rebuild that overlay for my compute nodes and we can and because this is part of a um runtime overlay we can actually watch uh oh okay so as you can see um because this is a runtime overlay we can see that um this has already been pulled and on this compute node now it's pulled the generic and Apptainer fake rout overlay uh so if we jump back to our test user now if we do another uh interact we just start off another interactive job and then we can hit
it with that fake root command again and now you can see um I no longer have that warning of UI sub uid not being mapped um and without I'm not seeing that warning with my user not being in the sub uid mapping uh so I can get out of that and uh wonder if this will work so U yep and as you can see um I the same file that's on the head node is now consistent with um the file that's on my compute node
Built for scale. Chosen by the world’s best.
2.75M+
Rocky Linux instances
Being used world wide
90%
Of fortune 100 companies
Use CIQ supported technologies
250k
Avg. monthly downloads
Rocky Linux
9
Enterprise products
Spanning the kernel to the orchestrator
Have questions about your infrastructure?
Talk to a CIQ engineer about Rocky Linux, HPC, and AI infrastructure.