How does Slurm and Warewulf work together?

in HPC Administration and architecture and today we're going to be talking about uh Warewulf and slurm and how they work together yeah awesome very exciting and also the the Warewulf community so um just behind the scenes if you would be willing to kind of drop a link to the Warewulf community so that anyone who is interested more in kind of diving into it and has question that are not quite ready maybe to talk to ciq although we're happy to answer your questions as well um and to jump into the the community um you should do that on slack slack is where they really um come

together cool yeah so you want to tell us a little bit about uh what Warewulf is sure sure so uh Warewulf is a open- source OS provisioning Tool uh it boots uh nodes using pixie and uh it uh and how it boots these nodes it boots them with node images which are in the form of containers and you might be wondering oh how is a container within Warewulf different than some a traditional something you might be more familiar with like a Docker container so containers within Warewulf contain a kernel and when you're booting up a uh compute node with Warewulf it basically grabs this node

image sees that there's this kernel inside and it ends up taking that kernel and booting with that Kel uh in order to basically have Theos interact with uh your Hardware uh so the fact that we are using um a container means that we can we can kind of do some cool stuff with uh with it so uh the first thing that you can do is you can build your node image uh using a container file and what does that mean uh that means that everything that's going into your node image is doed in the form of this container file um so once it's all documented

uh you can do you can throw this uh file into Source control uh something like GitHub or bitbucket gitlab whatever you're using in your environment uh and the fact that it's in Source control now uh all these changes uh that you are making to this node image can now be tracked uh approved and all that nice uh process type of stuff uh uh now that it's in Source control you can have some automation built around this uh you could have a CI pipeline uh which builds this container and pushes it to a registry um and then now that it's within a registry you can do

some cool stuff like security scanning on your containers before you actually take them from your registry and consume them in Warewulf in your HPC computer environment uh yeah so so uh that is uh one aspect of it there are also a bunch of uh different types of functions that are within Warewulf the uh notion of overlays uh templating which you can do based on tags within uh tags on some of the nodes uh and basically you can use these tags and templating to generate files within an overlay which at the end of the day get deployed to your compute node yeah well said just curious

is is there any relevance to the type of container that Warewulf uses um Warewulf uh the type so Warewulf uh can use uh I guess Docker containers built with uh Docker and podman it's compatible with uh any oci container so you can build it with Docker podman and also atiner as well so you can consume if you built your node image in the form of a Sith file you can actually consume that into Warewulf as well and deploy that into your cluster awesome these are these are some questions that sometimes we get just because we do provide the support also for Apptainer and then also

Rocky people are like wait do you have to use Rocky to use Warewulf I'm like no it is operating system agnostic for the most part right yes uh yes yes so currently uh I believe uh it works on susd De Debian and Rocky uh but the types of node images you can deploy in your cluster is agnostic so you can deploy anything into your cluster cool thank you for that cool okay so the other thing we're going to be talking about is slurm do you want to give a little overview of what slurm is for sure for sure uh so slurm is uh in HPC

scheduler uh there are two main I guess functions uh within slurm so one aspect of the functionality is uh policy management so let's you're you have the Computing environment uh you have a bunch of compute resources and a bunch of people want to use these compute resources So within slurm uh it can be configured such that um certain groups have uh either a particular access to a set of resources or maybe um a specific priority uh so for example uh if I'm in group a and I try to use group B's resources uh slur might stop me from using them or if I am allowed

to use them uh I might have a lower priority uh on those particular resources so uh the first aspect is uh policy management uh there's also uh the other fun part of this which is uh Resource Management so basically once your job is successfully submitted to slurm slurm basically allocates those compute resources within your cluster and takes your job and runs it on those allocated resources uh and ensures that uh you stay within uh the for example the core limits the memory limits uh that you have specified for your job uh yeah and uh yeah that is a brief overview of slurm uh it can

do a lot of other cool things like uh manage uh I guess license seats uh you can run it you can manage generic resources like gpus and all that type of cool stuff uh for all your HPC Computing needs so just a little question about the difference between Warewulf 3 which is a a a bit older and a little different from what I understand a Warewulf version four um is slurm kind of works across both though right so it doesn't matter if you're on three or four or is there a difference uh slurm will work um across both I think the main difference um between

the two is uh how your node image is built uh so uh on version three I believe your node image is built within a truth versus version four where you could use you could leverage a container and all of the nice things that come with that awesome thank you cool so let's get into it how did these two work together so uh I guess with the with the demo we're going to do today uh so Warewulf has the ability to uh provision compute nodes uh within your cluster so basically uh you can have a compute node in image specified with all your overlays and configurations

uh set up and when you go deploy this uh compute node basically the compute Noe will grab your note image uh boot uh Al and also grab your overlays it boots it up all your overlay files get overlaid on top and basically once it's booting up it'll begin to start Services uh and one service in particular that it will start is the slurm client and uh after that the compute node should be able to start communicating with your uh slurm controller node which runs on your head node uh sometimes also in the same server where Warewulf is running and at which point uh once your

node is up you can begin uh submitting jobs and Computing using those resources cool and uh I guess one more thing to add is uh you can boot more than uh just compute nodes using Warewulf uh if you wanted to manage something like a login node uh which has like an external interface where users can log in and actually do some tests subit jobs that type of stuff those login nodes can also be uh managed using Warewulf and booted through Warewulf as well cool so I guess we can just jump right into the demo then uh yeah let's get into it absolutely I love to

see it see it in action so Warewulf is one of the those things that you can see um all right uh is my font size okay Rose I actually can't oh here we go um I would definitely make that a bit bigger yeah yeah that's good there we go okay okay cool cool all right so uh here I have a uh slurm cluster deployed using Warewulf uh and to just show you what we're running with I can do Warewulf control version running Warewulf 4.5 7 um I believe 4.5.8 is the latest version uh which you can get um I guess either through the Warewulf GitHub

uh or also through ciq through Mountain all right so uh to show you that this is a actual functioning cluster I'm just going to run a quick s info here uh this is a slurm client command that just lists uh some information about my cluster you can see that I have a single partition uh with a single compute node uh this compute node is named ban HPC lab compute zero uh and how this compute node is configured uh and you can before I do that um I can just do a Warewulf control node list to show you that this compute node is uh managed by

Warewulf so as you can see uh my single compute node uh is output here and uh this compute node basically has two profiles on it defaults and compute CPU um so if we take a quick look at compute CPU maybe let's do a profile list CPU d a that this is what I want uh you can see that um have a few things configured on the specific profile I am using a uh node image called uh slurm compute CPU open HPC which is basically a node image which uh contains the slurm client and uh the slurm client is installed through open HPC uh and this

is something I've uh imported uh into Warewulf and as so under that you see um system overlays uh system overlays are basically overlay files that only get pulled once from the Warewulf server at boot time so within Warewulf there's system overlays which get uh pulled during Boot and then there are runtime overlays which uh get pulled uh periodically and by default uh these runtime overlays uh get pulled every minute uh so if you make an update to a runtime overlay uh within a minute uh your compute nodes will see those changes uh cool and I guess some of the to go through some of the

system overlays um I have configured here uh ww nit is an overlay that's default um on Warewulf which helps your node boot basically uh crony is for uh just managing my time server and I guess the most important one here is slurm D which configures um the slurm client and to quickly jump into the slurm D overlay uh we can just do a quick overlay list slur D and we can kind of take a look at the files uh within this overlay um I guess the file that I'd like to focus on here is uh this one here UHC sis config slurm D this is

a uh system configuration file uh for the slurm d uh I guess service and we could actually take a look at this file right here and you'll notice that uh it has the suffix ww here uh what that means is this is a Warewulf template so uh we can actually go take a look at this Warewulf template right nowww and as you can see here uh has the single line on it and basically what this is telling uh the slurm D service is that uh when it boots up uh it is is going to talk to the controller node and the controller node is uh

set through um a Warewulf tag called slurm controller so uh if we were tolist a uh and at the bottom here you can see that um the slurm controller um tag is set to uh my Warewulf uh admin node where the SL controller is actually running on so uh so that's kind of um a high level of uh overview of how this compute node is configured um I can show you that this is actually a functioning cluster by switching to this test user and if I do an S info you can see that my node is Idle I am going to run this command

which will allocate um a single node uh so this compute node has uh four cores on it and what this uh s all command does is it's requesting one node and um and four the four cores that are on it so I can do that now these resources are allocated uh to my test user and if I were to run a simple hello world program uh which I've compiled you can see that I'm able to run this hello world on all four of these processes that I've requested so let's get out of that and let me just back out into my root user and I

guess the last thing I'm going to demo here is I'm going to actually reboot uh this compute node and I will walk you through uh we'll kind of basic we'll basically watch this uh compute node boot up uh by uh over the network uh it will contact the Warewulf server pull the uh node image pull the overlays and up uh so to do that I going to reboot the uh compute node using slurm so uh to do that I'm just going to run this command here HPC lab compute zero and what this will do is basically reboot uh my compute node and before I hit

enter let me switch over to my console on my compute node so give me one second yeah no you're totally good I I usually like to ask a million questions but it looks like you are concent trading so I'll hold hold any questions and if you guys have questions out there who are watching um you can just put them in the chat um even if you're watching not live you can put them in the chat because we get notifications and so we can come in and answer your question or you can always go to ciq.com and ask your question in there and we'll make sure

that we grab Brian and have him help help us answer cool yeah can you make that a little a little bit bigger I think that I can't I can't make this one actually bigger it's uh okay that's all right I'll get my glasses but uh all right so uh let me just hit enter on this uh storm Command right here and cool awesome so uh it's booting over running some DHCP requests to boot over uh Pixie uh so a little there it's found uh the Warewulf server now it's pulling the node image uh so yep grab the kernel Image Grab the container image which

it's pulling right now it should grab your overlay next and uh begin booting so there are the runtime overlays system overlays and now and now it calls Espin in it and then my services start uh booting up and so what is like the ideal environment for a Warewulf slurm cluster like is there like a minimum a maximum like what how does this work uh I guess at the bare minimum you'd probably need uh two nodes uh one to run Warewulf and uh you the slurm controller and then you would need uh one I guess single compute node to uh orchestrate whatever workload uh you you

would want uh but uh I guess in more in different it it varies I guess environment to environment but uh some environments have u a dedicated database node where all your slurm job information gets sent to uh and you know uh and I guess it can manage uh any number of uh compute nodes uh whether it's CPU GPU all that kind of good stuffwolf is something that is near and dear to our hearts the story is kind of fun um so our CEO Greg kurtzer was working at the B Berkeley labs many years ago I think this was like what like 24 years ago something

like that many years ago um and one of the things that he was dealing with there and of course I'm probably going to get like some details of this story wrong with a basic gist is that there was at least hundreds of nodes that he was responsible for and he thought there has got to be a better way to manage all of this right like going into each individual one figure out what needs to happen running jobs like there has to be a better way and so Warewulf this idea of having that um you know that control node that head node and then being able

to manage your compute nodes um in in a simpler way was was really where this idea was born from and then and then it just kind of sat for a while and other people were working on it and several years ago um we jumped back in ciq as a company kind of jumped back in went hey what's going on in Warewulf like maybe we should send some resources there and make some changes and update it a little bit to our you know the current environment and so that's where we are now with Warewulf 4 uh so it's very very exciting lots of people around the

world are using Warewulf so you go in there you'll see you'll see their emails you'll see yeah who's in there running Warewulf clusters so cool any other final thoughts that you'd like to leave everybody with Brian uh I guess uh I guess one thing I will plug uh if you are running a Warewulf cluster today uh and are interested in running uh fuzzball for example you can manage your uh fuzzball compute nodes uh through Warewulf uh so if that is of interest to you uh we'd definitely like to talk to you uh hit us at you know we'll be at SC uh yeah just find

us at our booth and we'd be happy to chat yeah heck yeah so where wolf does um some similar types of of of uh work that slur would do and so we're going to be talking about that of course at SC so we'll be at SC 24 it's going to be awesome we are at Booth 4131 oh I have a little note here yay we are also going to be oh wait ciq Booth wait what is this let's also okay thank you sorry I have to read in order okay you guys ciq is also going to be at another conference a different one this month

October 28th and 29 it's called All Things open and so we are going to have a booth there it is booth number 84 and sus is having a book a booth also 91 very exciting okay so I think that's um that's the jam thank you so much and um at SC we're going to be having a bunch of demos of of fuzzball so that is going to be very exciting and just kind of like food for thought does where does fuzzball use Warewulf at all or can you use it on top of Warewulf is that kind of help not directly uh so Warewulf has the

ability to uh deploy a compute which is used by fuzzball uh so from that aspect uh that's kind of how they work together yeah it's really cool it's wild how all things um you know come come together together so thank you Brian thank you for your time thank you everybody for watching and your interest in Warewulf and all the cool things that ciq is doing and our little presentation about slurm and then of course we want to hear from you more so go over to ciq.com or email us at info@ciq.com

and we will see you later thank you Brian bye see you later