Dave Godlove on HPC Containers with Apptainer and Fuzzball at RMACC

time I'm going to review some of the ways in which it does that and I'm going to talk about some new things strategies you can use as well so first off um Apptainer really was one of the first container platforms to support gpus we were kind of a Pioneer in that area and that obviously makes your life a lot easier more recently have Tanner has begun to support unprivileged builds which means that you don't have to be root anymore to build a container you can pretend to be root inside the container and do what you need to do and also now supports unprivileged insulation so

I'll talk about all these in a little bit more detail in the coming slides finally I want to talk a little bit about MPI support this is something that Apptainer has supported for a long time really just because of its architecture but I want to talk about some new kind of tips and tricks that we we've recently identified and just kind of you know talk about how you can make MPI a little bit easier so GPU support if you know if you're using gpus with Apptainer then you know that you can just pass the dash dash NV option if you've got a Nvidia GPU

or you can pass the Rock'em option for AMD and essentially apptainer just goes ahead and locates and mine mounts all the different libraries that it needs into your container at runtime to make the GPU work and this is great because it keeps your container portable so um kind of counter-intuitively what this does it allows you to move your container from one system to another with a different PPU driver or it allows your system administrator to update your GPU and your container will just continue to work it have to like rebuild the software inside of it just because the driver changed so yeah that keeps your

container portable I mentioned before that we um now support unprivileged builds uh leveraging rootless username spaces so uh basically this requires uh an up to speed kernel that supports the username space but it allows unprivileged users to build containers even if you're using things like dnf or yum or you know apt or some some other package manager inside even if you're writing to you know places that are part of the root file system you can still build these containers without any root privilege this is convenient obviously because now you know users can build their containers anywhere but another thing that you might not think off

the top of your head it's a lot more secure and this is um something that we've been kind of trying to impress upon the community for a while is that if you are building your containers as root that's actually an inherently unsafe and inherently kind of risky um thing to do and so in the past we suggested that the best thing to do is spin up disposable VM and build your containers then and then throw them away this makes it so that you don't really have to do that anymore you can build your containers without worrying about the malicious contents of some container that's um

not only is it now possible to build your container without privilege but it's also possible to install apptainer without any Criminal s another pretty new feature um in the past it used to be that you had to have a root owned set uid bit as part of the Apptainer installation which would elevate your privileges for some of the operations carried out by the runtime but now that's no longer the case and in fact if you install appcaner um just with the default configuration these days it's going to be unprivileged by default which is the way it installs now um this also means it's possible for

somebody other than root to install apt paper normally you would use a package manager like dnf or something to install Apptainer but you can actually just install it uh just you know yourself and in fact that we now ship a convenience script in the obtain or GitHub repo that allows you very easily to uh install that painter in a relocatable way in an unprivileged way so this is a pretty new now supports okay now I want to talk a little bit about MPI integration so I've been kind of going through these topics really rapid firing quickly I want to slow down for a little

while and and spend a little bit more time talking about MBI so MPI uh support has been part of Apptainer for a long time it's one of those things that makes your life easier but in the case of MPI you know make sure like a little easier but maybe not that much easier MPI can be kind of tricky to implement with with so the reason for this is because if you try to spawn an MPI job from within a container what it does is it SS and H is out to a bunch of other nodes or whatever and then it tries to execute the

command that it's supposed to execute and it finds that command isn't on the system because it's in a container so it gets out of the container and it doesn't know how to get back in so historically we've suggested that the best way to do this is that you need to have an MPI installation on your host system and then you have to match either the same or a match a compatible version of The Eye Inside the container and what you'll do is you'll spawn your MPI uh using the host system MPI and that'll work in tandem with the container on its API right so there's

a few different models for how to do this the one the hybrid model is probably the most common this is this this is like straight from the Apptainer documentation but in this model you'll actually install MPI directly into your container and you'll match as closely as possible the MPI which is installed on your cluster and then hopefully those two will be compatible with one another will work properly the bind model kind of does the same thing but instead of instead of installing NPI into your container you'll just buy and mail all the bits and pieces from the host system into the Container at runtime so

this kind of you know make sure that the two are identical and then I'm going to talk a little bit about Third Way that uh We've recently identified Jonathan actually did a lot of this work and I'm just kind of just kind of it all for presenting but um this Third Way leverages a tool called PMI or a standard called PMI which allows you to spawn your containers without actually needing an NPI to do so so I'm going to talk about that in a minute first let me talk about the old way to run MPI okay so to do this I don't really have a

demo I've got more like a visual aid um so I I carried out this work in an environment uh you know just a little kind of test cluster environment that we have where we've got access to slurm and you know some other stuff but basically what I wanted to do is I wanted to figure out how hard is it really to to do want to to run NPI within a container and so um on the Apptainer documentation web page there's like a file example definition file for installing an open MPI I use that as a starting point and this is the result of that um

I'm not going to go through this in detail I just the the basic uh the basic thing that I want you to take from this is this was really a difficult exercise this was a hard thing to do to create a definition file which containerized MPI properly to run within this system okay um so I've got this little MPI hello world uh program and that's that's what I want to run just for test purposes um oh wait a minute this is not the right debt file I'm sorry I saw confused faces and I wondered why everybody was confused I'm in the wrong directory all right

sorry about that so this is the definition file I want to show you one that illustrates the hybrid approach of actually installing MPI into the Container okay that looks better okay so I've got this little MPI hello world uh thing that I'm going to be um compiling and that's going to be my test here um most of this just comes from the the definition file which you can find in the Apptainer documentation but if you go down here to this dnf you can see I'm installing a lot of different things with BNF right off the bat um if I go down here a little

bit further let's see I'm actually so in this environment where I'm doing this we actually have an opening an open HPC uh installation and we um we install a lot of libraries a lot of tools using open HPC which is great and convenient and easy but in order to compile against the same libraries I actually have to install open HPC inside the container as well and then use uh dnf down here to install a lot of the same tools that we had on the host in the container so it's getting kind of complicated and then if we get down here to the um the configure

statement you can see that there's like kind of a crazy custom configuration that I had to do and now this took me a long time to to create this and kind of the kicker is I was also the one who installed open H or um uh open MPI on this host so I knew very well how it had been installed and how it was configured and what it was linked against and all that kind of stuff and it still took me a little while to do this now if users don't really know all the ins and outs of how MPI is installed on your host

system this is going to be a really difficult to ask for them now obviously once you do it once it becomes a little easier because you can recycle this a lot of times if you're administrator you can make this a little easier for your users by actually creating a base container for them you know and have them start from that but still this is not ideal and it's it's also not super portable this container is going to break if the open NPI version changes on the host or if you move it to another cluster or something so it's not ideal let me look at the

bind model so you would think you would think intuitively with the Divine model would be come on in no you're good I get that a lot so um you would think that the bind model would be simpler right because it's all you're doing is you're taking all the bits and pieces that are on the host you're just by mounting them into and it is simpler if you just look at the definition file right I will point out that in this case I can't come so in the previous case I copied the MPI hello world uh actually C like code into the container and I compiled

it inside the container I can't do that here because there's no apis inside the container so I have to compile it on the host and then copy the the compile binary into the container right so that's a little bit wonky um this environment it looks complicated it's not really that hard all you've got to do is load the module and that gives you an idea of you know what the environment should look like inside the container but where the real complication comes in with the Mind model is look at this job submission script I've got look at that Apptainer buying path for a second that

is bananas okay there's no there's no getting around that that's a mess to make things easier on myself I just went ahead and I mounted all of opt ohpc into the Container which worked uh but couldn't Break Stuff who knows um and then if you look past that you'll see there's all these system libraries that MBI is compiled against all these user lip 64 system libraries that I had to buy and mount into the Container now that's you don't want to do that that's a bad idea um this worked in this case because my glibc inside my container was the same as the one on

the host because I was running the same uh OS in the on the in the container as I was on the host but there's no guarantee this is going to continue to work containerized operating system is different from the host operating system and I I want to point out one additional thing if I'm compiling my MPI code on the host and then just copying it into a container and then I'm taking all the MPI and the libraries and everything and copying them into a container what am I using the container for right what's the point so okay so um those are the kind of the

old ways to do containerized MPI that are not super great well now okay now I want to talk about a new method that um that once again my colleague Jonathan first identified and it's not new in the sense that these tools are new they've actually been around for a while but it's new in the sense that you know we've recently identified this as a as a good strategy and we we think this is a better way now that some of these tools mature to try to you know use MPI in your container so there's this thing PMI which is a series of standards and the

standards basically say this is how we think that MPI processors shouldn't do wire up this is how we think they should be launched and so on so because this these standards exist and I say this series of Standards because there's PMI there's pmi2 there's PMI X and PMI X's version so it's still not super easy it's still a little complicated but because this series of Standards exists you can do something like have your MPI inside your container have PMI support and then have a tool not necessarily NPI but have a tool on the outside of the container that also implements the same kind of MBI

stand and then you can use that tool to launch your containers and if you do that MBI on the host becomes irrelevant which is really nice in fact you don't even have to have it in the eye on the Island okay so let me show you what that looks like so once again let me go up here now to my different directory so I believe I haven't conferred with with Jonathan about this but I believe that his original intent in doing this work was to look at spec and to use SPAC is this correct that's what I do okay it wasn't necessarily the original intent

but okay so Jonathan was using SPAC to create containerized uh MPI applications and here's a very very simple well so so this is so SPAC so I'm introducing a bunch of stuff at the same time SPAC is a package manager uh for HPC which basically installs everything from um from source so it compiles everything and you you set up a compiler tool chain and then you it's very very simple and SPAC actually supports containerization so in this particular instance what we're doing is we're installing the OSU micro benchmarks um MPI suite and then we're doing so with openmpi and we're giving it a few um

things that we want this open MPI to support in particular PMI is here that's important um and then we're also installing lib Fabric and we're specifying some protocols that we won't let fabric to support uh the the kicker here is psm2 that's the important one for for this cluster because we use omnipath and omnipath is supported by bsn2 all right so uh then here this stanza basically says oh and I want you to containerize this and the way in which I want you to containerize this is to build a singularity container and then there's some details about the container I'm not going to go through

all of them but in any case once you've got that yaml spec file you can run a command within SPAC to spit out a definition file so I've already done that um and I can actually show how that works it's very very simple but this is the definition file I erroneously showed you earlier but once you've got this definition file you can actually just just edit it too that's another thing that you can do so I I liked this definition file but I wanted also to have this MPI hello world test so I just went ahead and used the file section of the definition file

to add that in and then down here at the bottom I compile it um this definition file is actually really cool I'm not going to go through all the details of it but if you're interested in it like you know ask me about it later it's got a bunch of optimizations that are pretty neat that I just don't have time over today um but any case once you've got that so now remember this thing has PMI support now uh I'm just running this on my laptop I don't have access right now to the cluster where I did this work but I did pre-record a demo

for you so I'm just going to go ahead and play that back with a cinema so I'm going to go ahead and play that so on this cluster I went ahead and I just got rid of the uh the MPI so I'm going to demonstrate first I don't even have MPI on the host system right so which MPI exact which MBI run that that's those are not on the path uh they're not on the host Okay so the next thing I'm going to do is I'm going to show you so this uh this definition file that's contained ompi I've created a container from that and

so now because that has PMI support and because I have PMI support in slurm which is also important I can use estron to just go ahead and launch this containerized MPI process and it's going to know how to do wire up through pli I have to specify to the S run command that I want the MPI flavor to be PMI to because that's what this particular um this particular version of slurm supports is PMI 2.

um and then after that it's just you know yeah set up the number 10 tasks set up the uh tasks per node and then I'm going to just go ahead and call apptainer and I'm going to say exec so I wanted to execute something in that contained ompi.cip container and all it's going to be is that MPI hello world and you can tell I type really slow it takes me a long time while you're typing do you need the same flavor of PMI in a container type like you specified PMI 2 here if it's just like PMI and X here it would fail Ure so

I it would have failed but the reason that it would have failed so I think that the container has PMI X support just by virtue of the fact that we added the PMI flag to uh SPAC I'm not sure about that I think it's the case but the reason I would have failed is because this version of slurm doesn't have pmix and the reason I found out at Oak Ridge that um slurm doesn't have this version of slurm doesn't have PMI X's it's got currently a vulnerability ah okay so it's not an accident they left it out yeah we're using the uh the version of

slurm that's in open HBC for this demo and so we just based it on whatever support happened to be there but yeah the MPI inside the container can be compiled with multiple pmix versions supported so um you can see that my little hello world did its thing it started two processes with two different ranks I'm gonna go ahead and run it again and I'm going to provide an environment variable which makes it more verbose and gives me some information about the high-speed interconnect that it's using and then I'm going to take that that spits it out in the standard error I don't really know why

but I'm going to take that standard error and I'm going to grab it for uh the the keyword open and that's going to tell me that not only did this container you know run a fully containerized instance of MDI but it actually used that PSM psm2 that um omnipass adapter to do it so that's pretty cool and this is a lot simpler it's a lot more robust it's a lot more portable it's just a better cleaner way to do and Beyond inside the container now I'm going to caveat this by saying that the reason that that high-speed interconnect worked I don't think that that's a

given okay I think that it worked because we have similar tools installed on this host and within the container um so you still could run into some trouble with your uh with your high speed internet not working that gets into kernel modules and libraries and match kernel modules and stuff like that I'm not going to cover that in detail but uh still I think you're on much firmer footing if you've got this wire up problem solved and that's one big hurdle you've overcome already all right oh yep and I just did the devil okay so and once again uh I covered that in like 10

minutes and I left out a ton of details um Jonathan so we gave a talk jointly like a three hour talk which is crazy uh two um Stanford University uh with their HPC ai ai advisory Council and that's up on YouTube and the second hour is Jonathan going through basically this in a lot greater detail so if you want more details about this I highly recommend that you watch this talk I totally forgot that was on YouTube yeah it's on YouTube okay now um I've got like six minutes left and I've been talking all about containers and in this day and age if you are

talking about containers and you don't cover orchestration then you're doing something wrong like container orchestration is a very very big deal right so no modern discussion of containers complete yada yada all right um so I would pause it though that existing container orchestration Solutions have really not been created with HPC in mind they've been created um for services specifically they've been created with microservices running the cloud native architecture in mind okay and I'm looking at kubernetes in particular when I say this but you know there are other tools as well that have been designed with that mindset because of this they include features which are

not relevant to us users who want to run jobs in HPC environments and you could say okay well who cares you just don't use those features the problem is you still have to learn about those features to use these tools you got to know all about they also exclude scientists who want to use HPC uh with with orchestration so we really need a method to orchestrate not services but to orchestrate jobs now a lot of times in HPC we've we kind of we use slurm or a batch scheduling system like slurm to do our orchestration so we don't really worry about it too much but

you know increasingly we're moving away from on-prem HPC clusters we're starting to Branch out into the cloud and do you know HPC in other places and then it becomes a big glaring issue need to have some sort of tool to perform orchestration and so that's fuzzball that's what we're going to be releasing on Thursday that's the new product that we're going to be unveiling pretty soon um it says I'm going to give you a demo fuzzball in let's see four minutes I'm just going to give you like a little teaser trailer a fuzzball here so buzz ball um has a command line interface and it

also has this really you know shiny pretty web UI I'm going to show you the web UI because it's a little easier to explain and kind of see what's going on but bear in mind that you can do everything that you can do with this web UI you can do all right so um if you so I should also explain that this version of fuzzball is running on AWS so this is actually a real you know instantiation of close ball which is running and what it's doing is it's um a service which is hanging out waiting and listening for somebody to submit uh uh a

workflow and when you submit a workflow you do that with a yaml file and so then it says oh okay well you've told me that you need these resources this many CPUs it's much RAM I'm going to go out and spin up those Cloud resources for you I'm going to grab the containers that you need to run this this job I'm going to kick off your processes inside of it I'm gonna step in independencies that you've got I'm going to take your S3 buckets or whatever storage that you've got map those directly into your jobs you can access your data and you can write it

back out and then I'm going to tear everything back down again so it's not sitting around wasting your wasting your resources that's fuzzball in a nutshell um so I'm just just going to show you really quickly uh so here I'm showing all the workflows that the people in my group have run most of these are run by Forest Burt who's a super smart awesome guy I'm just going to show my workflows for a second and that's so I can quickly and easily find this MPI hello world a demo that I just showed you I actually ran it in fuzzball as well um and so if

you look at the logs from that you can see that uh you know this ran two different ranks now we're looking at IP addresses instead of host names uh because we're running in AWS but yeah it ran two different ranks and let me show you how easy this is so I can open this workflow in this editor this is like a drag and drop kind of thing where you can move multiple different jobs around you can set up dependencies between them you can do all kinds of cool stuff like that uh you can also click into the job and then this is where you configure

everything so the job's name is here the command that I wanted to run is MPI hello world the same as its name the environment specifies what the container is so everything that runs in fuzzball is containerized and this is the container that I want this container is actually up on Google and so I need some secrets to be able to access the container and fuzzball allows you to manage secrets in a really you know 90 secure way but what this all actually boils down to is a yellow file and so I could just pop open them and I could just write all this stuff in

a yaml and then I can just submit it that way too if that's um okay so in addition to um so so the way that the MPI works is hearing resources it's nothing actually here in uh the job type you say that you want this job type to be multi-node you could also have a task array if you want like an embarrassingly parallel job and you can specify that you either want openmpi mpitch or gas net and I also want to show you really quickly let me go back I'm going to show you really quickly um a more complicated job this job actually didn't finish

successfully because the S3 bucket that I had at the end actually didn't exist but that's no big deal it just you know shows that you can map an S3 bucket to to it if you want to and this one I want to show you because this illustrates the idea of a job dependency right set up the these directed acyclic graphs of jobs and in this case I've got you know one container that I'm just going to use to create a bunch of directories then once I've got that container run I've got another couple of containers that are going to download data once I've got the

data downloaded I'm going to go ahead and make a database using blast which is a biomedical application and then I'm just going to go ahead and run and you can see how this you know you could fan out and have hundreds of all of jobs running underneath of the single one and then fan back in and you know do like Map Reviews kinds of things with this orchestration tool okay so that's just a uh and also I should um I should mention that you can easily use as well specifying is it under devices yeah just Yeah by specifying here that you need a GPU and

the number of gpus that you need I thought this job ran with gpus I guess so Okay so that just a really really quick little taste of fuzzball we're going to be releasing that on Thursday and once again and so uh you'll soon be able to use it yourself so just to kind of sum up everything I've told you so apptainer was created to fill a need in HPC for containerization it was purpose built really for contain for HPC as you all know makes your life easier and it's continuing to do so with new features that we continue to develop and fuzzball now just as

Apptainer was originally created to fill a need for containerization in HPC fuzzball is now being developed to fill the now um glaring need for orchestration in HPC okay and that's it thanks very much I'll take any questions I understand I went a little over time um so if you want to just talk to to me or Jonathan later we're going to be at that uh ciq table in the main hall but I'll take questions now too I think Kelly you got a question oh yeah sorry yeah so um services or basic services um we've been talking a lot about this I mean it's it has

been at this point purpose built for HPC sure and so uh because of that so so so buzz ball runs on kubernetes because because that's the best it's a micro service it's it's a the end of architecture it's sitting there up on AWS and waiting for gambles to be submitted to it and doing stuff based on that and so for that for you know that type of service job kubernetes is better but um you know for jobs fuzzball is better and I I don't know I mean I'm sure we would we we've talked about yeah so the reality is like fuzzball is developed by requirements

from people that we're working with who are using it and that is a need that is expressed so it's something that's being worked on it's not the next thing we expect to release but it is pretty close a job can Surface in some ways so there's that and other things that like we've seen uh come up is like there's the whole um the science Gateway use case so there's the idea of having a science Gateway that's maybe running as a service and Buzz ball that spawns off jobs and part of this workload or maybe you have a data provision service like a database or

something like that that you then want to run jobs against so that those are the kinds of use cases that we've been looking at but it we don't have a date on it yet language oh yeah so Chapel runs on gas then so that's that's that was yeah so we we've Run Travel jobs uh we've been Fuzzball and that was actually the reason that we implemented uh gas that as one of the different it's not really NPI but one of the different uh multinational players yeah and as you know I think that that's a pretty pretty unique in this kind of yeah so I don't

really know what else you would you would support but you know as yeah keep asking sure um so uh sometimes Singularity container if you're pretty big which adds to the build time with them a doctor obviously has some kind of layering system and I mean a container with Singularity you kind of build a sift and then build another sieve on top of that sip yeah um but like that also requires a bunch of Def files and much linking this is there some kind of plan or a certain conversations about some kind of building layering process built in so that building is a little faster at

times it's like I'm building machine learning containers and all of a sudden like oh it's taken me forever to just add one package sometime that's not your fault the package is Big I'm just curious like if there's different conversations yeah I mean that's that's something that we've been talking about or people have been talking about have Taylor for you know many many years but I mean the the fundamental fundamental idea is so radically different that it's not really um it's not really a good way to kind of you know so what I suggest um is that when you're first developing your container the best thing

to do is you can box which is uh avatable it's just a bear director yeah and so um and then to make your changes in the sandbox and that way every change you make is just there record all this all those and then only build the dev file kind of like at the end after you're sure that are um but you know I think what you're saying is I've got a container I've been using for a long it's awesome there's another python package that I want in the container I have the global mixes from scratch yeah there's not an easy way around that because you

could just create a definition file to build from that base container right and so but the definition of file is something that could save in the metadata inside the container and if you do that all it's going to say is open the space container and I added this one thing so then if you want to go back and rebuild that container right or somebody else gets it they want to be able to send it it doesn't so I mean you could if you're trying to do like one-offs and kind of you know something quick you can you can totally do that you can build from

your existence and you can do that also I don't know if you're aware of this you don't have to use a register or anything like that like that it's built from yeah I have a recommendation on that if that's all right right so what abdainer is many things to end but the most of its benefit is in the runtime engine the the way it does mounts and sets up your your you know the actual physical container the Sip uh format has a lot of like performance benefits and things like that um but there's no reason you have to use all of that painter's stuff so

what I like to do is Define my container as an oci image and you can build that with anything get all the benefits of the layering that you get in that and then you can just pull it as a sip when you want to run it right and use it as the run time but not as your development specification I so I will um Astros yeah go for it so there are there are some features of Sif files which I think are underutilized um those include things like signing and verifying your containers software supply chain I don't think people realize what a big issue it

is yet I think it's going to take some some malicious actors to own some folks before people realize so um you can do that with a Sif file so you can and you can push it to an oci registry like Docker Hub or something using before ask protocol and then when you pull it down you can make sure nobody's tampered with it by verifying that signature on it um there's really nothing like that right now if you're using oci images with that paper the two of those don't allow you to verify the contents of what you've got so and then there's other things like there's

like um you know you can encrypt containers you can encrypts of files you can run them encrypted and do things like that I mean unless you really need that unless there's a special use case you wouldn't use that but I all I'm trying to point out is that the the sip file format it's got some advantages Gerber oci and you know if you get locked into creating Docker files and building everything within a CI you might you know you might go down a pathway that might you might miss out on some features once they start to be more widely yep um it's like yeah I

got six minutes to get to the next talk which I guess would be fine but if there's any other questions I can I can hang out thanks again before