Build Me an HPC Cluster: Rocky Linux, Apptainer, and Warewulf

a little money getting out of the cloud and you know they're they're buying some hardware and they're like okay like now that I just have this incredible investment in Hardware what do I do how do I put it together properly so this is very exciting Brian appreciate it cool cool and I guess in with related to that I guess a performance of your workflow yeah it is really dependent on the type of Hardware you're buying but uh I feel like with werewolf uh it just gives you a very easy way of just provisioning your cluster just to get it up so that you are able

to at least Benchmark your workflows and figure out how they work and really optimize them uh to where you're getting Peak efficiency out of your system yeah I feel like bri I hope you cover this a little bit but having some of those people that have come to us that you've worked with specifically that are bringing older hardware and new hardware together so I have this old thing that doesn't necessarily function the way that I wanted to and building that into something that they can make far more usable so for sure it's awesome looking forward to it cool so I guess uh let's jump into

uh I guess a basic uh just cluster architecture so if I share my screen here can y'all see that uh got it cool so uh this is kind of the architecture that we're working with uh in our HPC lab uh so uh how I've set set up this environment is uh we have a head node that's connected to some external network that I'm going to be sshing into uh this head node is connected to an internal cluster Network which um werewolf will be doing um provisioning on and uh slur the various slurm services are also running on this internal cluster network uh addition uh not

in this environment but this is very typical with on Prem environments as you'd also have an ipmi Network uh what this is meant uh to do is this enables you to control your compute nodes uh from your head node and be able to power them on and off uh without actually going into your data center and uh hitting a button uh which makes it really convenient uh and I I've you know if you're a sist admin you know the feeling of going having to go into the data center and actually hit a button is very annoying and you know if this is a way to

prevent you from having to that uh so uh and then lastly uh as part of this environment we just have a compute node that is running slurm d uh this is basically the slurm client which will talk to the controller to either uh basically accept work and uh communicate with the controller um and there's one more service that I have on here that I didn't mention slurm DVD is basically the slurm database statement uh this database statement is basically responsible for uh WR accounting job accounting data to a MySQL or badb database uh and it also uh is can store job completion data as well

so uh you would want to run this in your environment if you want to uh do some reporting on the usage of your cluster stuff like that and uh by being able to generate these reports you can take these reports show them to your management and show them the value that your equipment is actually delivering uh for your organization so real quick Brian I know we the title of this is Rocky faner werewolf I know we have stuff up here about slurm do you want to give just a real quick I know you talked about what from a reporting perspective what kind of real quick

high level what is slurm uh yeah so slurm is basically a uh workflow um workflow scheduler so basically it offers an interface for users to submit workflows either in batch mode or interactively and uh have these workflows run on your uh compute cluster and uh basically slurm has like a lot of features where you could optimize uh you can optimize slurm configuration for your workflows whether it's from uh setting resource limits uh implementing uh access policies all that type of stuff can be implemented in slurm and also tracked as well thank you cool cool so I guess uh I guess moving on to the next

part so uh yeah that basically covers uh the architecture texure of what we're kind of working with today uh let me just stop my screen share real quick and uh yeah I guess we can talk about maybe why werewolf why slurm or why obtainer and I guess why Rocky uh I guess going into let's start with uh werewolf I guess uh werewolf is a operation OS provisioning uh Tool uh and basically can be used to provision a high performance Computing cluster and uh I guess viewers might be asking why would you why would you even want this right why do I need a OS provisioning

uh tool and why would it benefit me and uh I think the story that I'm reminded of is uh Greg telling the story our CEO Greg CER telling the story of uh debugging a uh I think the what the root cause of an issue with related to a node he was working with was it came down to power on a node but without having your nodes consistent you wouldn't have been able to kind of figure this out and and I guess werewolf is a way of ensuring that what you're deploying to your nodes is consistent and is well docu is documented and uh basically you

you're basically removing all these variables so that you can actually debug these low-level uh problems basically um with werewolf uh specifically werewolf V4 uh some of the cool features that come with this are that um your node images are basically containers so because they're containers you know you can build them in a CI pipeline uh have these written to your uh some container registry uh if you want to do security scanning on your registry you can do that before uh these node images get deployed onto your cluster um and so in addition to that in addition to the uh node image aspect of werewolf there's

also werewolf has the concept of overlays and overlays are basically uh configuration files that get applied to your compute node after um it's been provisioned and uh before your services basically start up um werewolf has the concept of system uh overlays and runtime overlays system overlays get provisioned one at boot time runtime overlays uh periodically get updated so uh if you want to make a change that's part of a runtime overlay uh it should get be deployed to your new node uh or to your compute node within a minute so it makes it really easy to push out these changes to your compute nodes so

have we have a question but I'm GNA ask you a question before we go to that one thanks for the question Smit we get asked quite often whenever we talk about werewolf pushing to a compute node is a container I get people that are asking are you actually running a container on an existing operating system or is it the definition that then gets turned into an actual operating system and run somehow uh it is a so basically uh I believe how this works is um we're basically building a container with a kernel in it and this uh container gets pulled uh to the compute node

as part of the pixie boot process and we're basically booting off the colel that's within the container so when your compute note actually comes up you're basically running on bare metal and not within a container yeah and it's a means of a means of provisioning your operating system but not actually running within a container makes it very clean yes yes cool so we there we go how is ciq different from Bright cluster manager oh it's good one I think mainly werewolf I think mainly with werewolf uh while ciq uh it supports werewolf and uh basically werewolf is I believe I haven't worked with brigh in

a while but I believe brigh is um does stateful provisioning where um uh werewolf is stateless and I believe last time I used bright I believe they're still using you had to build your images uh I guess in chuts versus uh werewolf where we're just using container files you're able to pull these node images in from Registries uh stuff like that um we also speaking of those containers we have uh base uh Linux containers um in a werewolf node images repo so if you want just a werewolf bootable container uh that you want to pull from GitHub you can visit um that repo and uh

be able to get up and running really easily thank you for the question yeah oh so seems to quite come up quite a bit that's why Rose is chuckling we hear how is this different than bright seems like every day uh so I think those are I guess the main points I had on werewolf uh I guess we can move onto kind of Apper and why you would want to use that on your cluster uh I guess from a user perspective uh this Apper has the feature of uh fake root and as a user if you're a scientist you're trying to install software I guess

the old way of or how this would typically be done is you know you need a contact your si HPC CIS admin they have to install the software for you and then there's all this back and forth of hey did you test it oh it doesn't work and then you end up with this back and forth but with obtainer the beauty of it is uh with this fake root feature uh users can just install packages in kind of a Sandbox environment and you know if you mess up your container you you can delete it you can build a new one and uh you can move

a lot faster without having to do all this back and forth with your system administrator and it really puts the power of uh being able to do science in the scientist's hands basically um some of the performance benefits of using uh Apper from like a technical perspective uh especially if you're running uh you know python packages stuff like that if you're doing that over NFS uh typically there are a lot of metadata operations uh when your job is starting up uh which may affect uh your job startup time uh by having these packages all within a container these metadata operations become file seeks resulting which

will result in uh your job uh starting a lot faster basically um and you know job start job starting faster is I think a really good benefit and you know you can get your results faster uh you know get those results into your paper all that all that good stuff yeah I think one of the questions we get a lot to Brian is why not just use podman or Docker um I think with um so the nice thing with Apper is that uh your your container is basically just a a file on the file system which makes uh moving your workflow a lot more portable

uh and yeah yeah I think that that is one of the biggest benefits of it and you know whether you're on your school system you can take your you can take that container move it to the cloud and you can ensure that your software will still work on the cloud as well yeah thank you cool cool so um I guess we can we can jump into a demo if if we let's do it every's ready all right so let's Jonathan has got his coffee so he's not fair he had even started yet you to have a second cup Jonathan coffee coffee is done oh that's

awesome all right so I guess also I mean what does werewolf run on any operating system I mean obviously we want to highlight Rocky but does it really matter as long as it's Linux or can it be anything I believe I believe this Rocky can be installed on any RPM based Linux drro I believe uh at the moment um yeah you mean werewolf so oh yes werewolf my brain is like trying to figure that one out yeah all right so uh with this demo that I've prepared uh I'm basically going to demonstrate um Apper so I'm going to demonstrate getting ater fake grou uh working

on a werewolf manage compute node so uh is is my font size okay no I don't think I mean I have my glasses on and I would love it to be quite a bit bigger all right let's do that a little bit more yeah okay so better on this bottom prompt uh I've basically just set up a test user and uh with in this cluster I have slurm running so if I do an S info here you can see I have one node up HPC lab compute zero uh so uh as a user um on a cluster some something I would try to do is

uh create a uh set up a interactive job which lends me onto a compute node so that I could try you know maybe building my software and testing it out on some of my data uh and this is just for testing purposes before uh I start launching a bunch of workflows into the cluster and I just want to make sure my stuff works and that my software is working so uh the typical workflow that I would do is I would start off with a interactive job uh and this should put me on a c put me on compute node zero uh and right now I

am going to try to do and if we take a look at my home directory I basically have this werewolf Rocky Linux a uh container uh that I'm just going to try to Shell into uh with fake fruit so uh show and let's add a fake route here or first we can we can start with just showing into it normally if we do who am I you can see that I'm the same user so if I exit out of this and I try to use fake rout it gives me um a warning that um I basically don't my user uh I basically don't have a

user mapping for this user in uh SLC sub uid basically and the purpose of this demo is to show you how to get these files um basically onto your compute nodes uh through werewolf uh so uh and because I don't have this user mapping uh I don't have the basically in a the fake root context that I'm in right now is has doesn't have the full capability of um a painter rootless um I guess cap feature basically uh so uh let's back out of this I'm just going to exit and let's get out of this interactive job and let's let's see how I'm actually going

to implement this so taking a look um at into werewolf now um I basically have this overlay set up called obtainer fake group and within this overlay I basically have uh SLC subid sub uid and subg uh with this ww suffix uh on it and you might be wondering what is the suffix uh on this file uh the suffix basically denotes that this file is a uh werewolf template so uh let's actually take a look at what this uh template actually is so if I C this file and let's just take a look at some and basically catting this file uh within this overlay you

can see that um I have this line which includes uh the file on my this F specific file on my controller node and if we take a look at that actual file on my controller node we can see that we have a mapping for um test user um that's already here so by having this file on my compute node um I should be able to use the full featured fake R feature um so to do that let's first take a so I also want to show you uh the node that I am the compute node within werewolf as well so as you can see here

um what I would like to point out um in this line here is the runtime overlays um are coming from this default uh node profile which currently just contains the generic overlay which contains things like uh your users your groups and all that type of stuff um so how do we actually get this overlay onto the node so to do that uh I am going to add this overlay as a runtime overlay on the default node profile so to do that uh we are going to do set and then I'm going to Adder F group so I'm going to run this uh werewolf control profile

set command which will set um the runtime overlays of the default profile to generic and obtainer fit group so do that it'll ask me to confirm yes so uh I also want to show you uh you can also see what um runtime overlays your uh compute nodes are pulling by running the werewolf node status command and as you can see HPC lab compute node zero is pulling a runtime overlay which just contains this generic um overlay um so after I've set um this overlay runtime overlay on the profile I can quickly rebuild that overlay for my compute nodes and we can and because this is

part of a um runtime overlay we can actually watch uh oh okay so as you can see um because this is a runtime overlay we can see that um this has already been pulled and on this compute node now it's pulled the generic and obtainer fake rout overlay uh so if we jump back to our test user now if we do another uh interact we just start off another interactive job and then we can hit it with that fake root command again and now you can see um I no longer have that warning of the UI sub uid not being mapped um and without I'm

not seeing that warning with my user not being in the sub uid mapping uh so I can get out of that and uh wonder if this will work so ID yep and as you can see um I the same file that's on the head node is now consistent with um the file that's on my compute node cool very nice yeah so yeah and yeah uh I guess I could pause there for any questions we might have uh anything coming in from the comments or yeah Smit does have another question we are required to get every software approved by our University do your containers comply with

some security standards [Music] oh I feel like we need a little bit more detail on exactly what that what that is I think yeah I think what we would need to know what type of Standards uh I guess you're trying to meet um and I think we can we can work from there yeah yeah I'm assuming it means the ones that are getting pushed out to the compute nodes so as long as long as you define that in your container to be the image that you need it to be and meet whatever policies you're trying to meet then the answer should be yes yes cool

well that should be pretty easy if you know what you're looking for because containers specifically Apper you can sign them cryptographically right like so you know where they came from is that like a helpful feature for this specifically or are we talking about something different oh I'm wondering if he means the actual ual operating system running on the compute node so it might be a little bit different but to your point if you're talking about the container specific side then as long as your container is running something that is meets whatever security standard you need yes you can encrypt that as well y cool cool

you kind of know where where it comes from I mean that's kind of like one of the the benefits that ciq provides as well right is through one of our our products named Mountain you can have you know your signed images Apper images in there so you know exactly where it comes from and it's verified not important for some very important for others yes totally good point R cool and I guess we can kind of segue into I guess our last topic of just running workflows that work well on the stack uh I mean in general I feel like uh you the stack can B

basically run any workflow but what when it comes down to for like performance I think it really comes down to the type of Hardware that you're running on and also how your uh scheduler is also configured to actually orchestrate these workloads in an efficient manner uh so that you know you're getting the best time to results um I can kind of expand on that uh I think it's different for I feel like every for different like indust IND verticals but uh if there's a specific industry vertical that youd like me to talk about yeah I'm I'm open to uh open to any um yeah any

any suggestions so Brian on the different Stacks doing it if you look at it and think about it that way if I have two different types of science that I'm doing in the same cluster with werewolf I could actually push different images as that executes or as I need them like I'm going to do this type of workload I can make everything on the cluster it's could be know this and then I can quickly move it to something else to execute a different type of job right yeah yeah so I think uh let's see let's try let me try to think of an example here

like for example like let's say you have a cluster you have some machine computer science department that's doing a lot of machine learning so you have you know a bunch of GPU nodes that might run best with Ubuntu for example so you can configure a node image for that with all your drivers and all that stuff that work well for machine learning but for something that's more cfd based you might want to go with something like Rocky uh and these uh cfd workflows uh typically run uh multi noes so you want to probably have them running in MPI and if you're doing MPI you might

want to have a uh interconnect like infin band and you know those drivers would also be installed on your node image and get deployed onto these nodes as well uh yeah thank you yeah and when it comes to I guess cfd in general is like an area that I have more experience in and I think benchmarking like figuring out performance for cfd workflows I think is a little bit easier compared to um other industry verticals uh compared to like for example life science uh I think my thought process on this is with the life science process when you're doing like genomics analysis there's a lot

of um lab prep that goes involved which involves you know sometimes a human dropping you know chemicals into a micro array and then putting that into a sequencer so there's room for error whereas if you're working on kind of just an engineering problem in a simulation the your variables are a lot more controlled and the type of simulation you're running can also be extrapolated a lot easier um and this is because within a simulation you're kind of simulating over a specific time scale so for example let's say I'm doing a 10-second simulation uh and I want to Benchmark how well this would run across these

different um I guess Hardware configurations for example you can scale down that 10sec simulation to 1 second figure run that 1 second simulation across all these different configurations and find which one gives you the best cost performance and and from having that cost performance you can kind of extrapolate what this would look like on a 10-second um simulation and figure out if that's if that meets your business needs and uh and see how that actually affects your bottom line uh and you know this is important when it comes to you know spending your grant money or you know you have a budget allocated to your

cost sender that you want to make sure that the workflows you're running actually meet that budget and you don't want to blow that budget cool cool well said Brian well said cool uh I think we have another question in yeah where is that in the YouTube yes right there most of the National Labs have chosen allma Linux for security requirements I don't know if that's accurate but we'll go with it I wonder what Rocky Linux needs to do in order to comply with Federal compliance I think we just need fed ramp what fed ramp certification and which I believe is well we're currently working on

fits and uh yeah and I believe what is needed there is probably just fedramp uh compliance and once we get that certification I mean I feel like we could we could be deployed in National Labs a lot easier yeah and I will say we are being deployed in a lot of National Labs at very large scale already so love to talk more about that subit if you'd like to yes reach out to us yes www.cq.com you put in you type in anything there you put in your the email address that will come to me and I will respond to you and then we can chat

more about what your actual use case is is where we are at um in terms of compliance and fed ramp and all other types of security things because we're definitely that's on our radar we want to do that right like the whole the whole point here is to be um responsive to our customers and our clients and what people need so um if there's something that you are curious about I love it bring bring bring it to us and put it on our we'll either let you know it's already on the road map or you know it brings it to our attention so we very

much appreciate that okay Brian so you said you actually said something and I know this probably you know we might be wrapping up here but I'm you said something about um God what was the word that you used um like monitoring right where you said that you could have reports back you said something about reporting back to your you know superiors or whatever like of what's going on in your cluster and how it's working and either if it's speed or efficiency or whatever it is is so how how do you how do you get that information um so I I guess from from a slurm

perspective uh because your uh so accounting data um B within slurm basically so slurm basically stores accounting data and job performance data in within this database uh there are a lot of tools that that allow you to kind of visualize this data and also potentially generate report and allows you to kind of really look at your jobs and understand if they're performing efficiently or not uh most of that information can be pulled from slurm uh and basically uh I think from the perspective of like management like CIO for example might be interested in just the jobs that are running and the value that they're delivering

and maybe not so much the low-level performance details of uh the work flow where I feel like that type of data the scientists might be most interested in so that they're able to get their results the fastest and you know things that you would look at are you know maybe uh your memory usage the io on these jobs uh stuff like that and understanding where these bottlenecks are so that you can remove them to get Peak efficiency uh in your workflow so is werewolf able to identify clearly and feed you back the information of what's going on on each specific node uh not necessarily I

feel like werewolf is mainly responsible for provisioning the actual node itself but uh getting that data from the node you could use various tools like various monitoring tools like Prometheus uh some some users use uh xdmod uh yeah so just different types of monitoring tools to actually get uh that type of data but werewolf will basically just provision the node for you with those services on it cool very exciting anything else Mr Zayn you got a question I'm just watching the comments there's another question that just popped up cool are you seeing most customers building MPI inide of their appers or using hybrid mode I

feel I feel like maybe at the moment more so inside the container but uh I feel like the solution that we the solutions Architects at ciq have been pushing are is the hybrid model uh so that is the method we would recommend uh but if you don't know how to do it I believe there are blog posts uh that we have uh that that go through how to do that so definitely recommend checking that out do you want to give a real quick why you would choose to go that way Brian why hybrid makes more sense from our perspective so using the hybrid method I

believe it's you're basically just using uh whatever is available on your system so you as a user do not really need to care about uh installing open MPI within your container and you know when you're using the hybrid model I believe it's as simple as just running uh an srun command uh on the serm cluster versus uh yeah yeah it's basically just running an srun command if everything is configured correctly on your cluster and you know that just makes it a little bit easier on the user uh if you're not familiar with MPI uh you know running a running a srun command is a lot

easier than figuring out MPI in my opinion thank you for that yes okay are we good with questions answering all the question questions and if something more comes up you guys are watching this not live or like 5 minutes later something comes up um feel free to either put it back into the comments right here which is fine we check them periodically and or go to our website ciq um.com and submit a question there we are more than happy to either answer right away or go and research and find the answer for you or work together with you for an answer because sometimes that um

also needs to happen um David Rush has a question thanks for joining Dave it's always good to see your questions Dave Rush what's up man is there a performance difference between hybrid versus dedicated MPI with containers I think we didn't we do a webinar on that like we had like graphs and charts and stuff didn't we like Jonathan did yeah dove into this a little bit but that's been a while May it's time to refresh that one Jonathan instead of making coffee you can go refresh this wait see how long before I get a slack message hold on right do you have a response for

that um Brian uh I'm actually not uh off the top of my head I'm not sure if there is a performance difference but uh I would have to review uh I believe Jonathan did do some work on this which I would have to review before I speak more confidently on this topic yeah we'll find the link for you too Dave we'll find that link for you and P pass it on because I I I remember this happening and it was cool to see the charts and kind of have him explain the testing that he did around that it was it was nice it was cool

and yes Dave we will post that again so you can go right to it yeah cool okay so rather you're like running genomics am I pronouncing that right genomics genomics genomics potato potato tomato potato potato you're at a university whatever is happening you got an OM Prem cluster that you are wanting to make sure is provisioned properly with some help and support I mean that's kind of one of the cool things about ciq is that um you know all all three of those are open source initiatives so the rocky the aper and also werewolf and then we come in as an added value a resource

we're like okay this is your specific environment this is what you need and this is what your Hardware is this is what you're working with this is where your you know security your storage your all the things um and we come and help you figure that out how to set that up properly and get your cluster up and running and then supported and it's nice to have one place to call that really has Engineers working in all of these different initiatives it's um uh you know our our customers that we have you know engaged in these sorts of relationships helping build out the cluster so

thank you Brian and the sa team and all the engineers working on it um it's really been a a a cool thing because I mean you know what it's like if you're out there and you're like what am I doing with my cluster it's like oh my God what am I doing and to have that kind of like help and support and somebody to call who's on a team working on your specific environment just priceless priceless well we actually do we we can put a price on it but you know it's Priceless and the the way of it feels good um so there any final

thoughts that you want to share about clusters Brian uh no not really uh I would just like to call I'm gonna be at fem uh 2024 next week and also SOS connect in uh Brussels Belgium so if anyone if any of our viewers are attending and want to talk more about werewolf ater Rocky uh I'm happy to talk talk more um at that at those conferences so fantastic yeah it's awesome dang Bri live in large yeah no kidding stay tuned on this on this question on on a gooey for werewolf I actually had a conversation about that this morning so stay tuned it's coming Su

it's coming yes and again thank you for pushing that because the rest of the team uh who's working on that sees it where it's like yes this is something that people are asking for and that they want so appreciate that cool okay well I think it's time to wrap up you guys HPC clusters boom boom boom I think Jonathan only had two cups of coffee Brian you are absolutely amazing appreciate your time and your energy you guys we are here to talk about clusters and help you build them out answer all of your questions we love you we want you we need you we'll see

you next week same time same place make sure you like And subscribe and check out our podcast flops and threads have a great day bye thanks everybody thank you Brian [Music] later