A New Approach to MPI in Apptainer

will introduce myself for the probably a zillion people who don't know me I'm Dave godlove I used to be a research scientist um I used to work at um at the uh NIH for a while and as a scientist that became interested in high performance Computing and I got on with the container Revolution while I was there and I started working with Greg and some other people helping to develop abdainer and now I'm here to talk about it as a Solutions architect School yeah it is cool man and thank you all for all your good work so if you guys are in the app tuner

um slack community and you ask a question I mean I would not be surprised if Dave God love is the first one to answer you he is on top of it and he's been doing this for a long long time so and he's been all over the place he's seen all kinds of craziness of what it can and cannot do and is happy to help you so we definitely support you in engaging in that community and getting all your questions answered if you're reading the documentation Dave probably wrote it that is accurate so if there's an issue with the documentation now now you know

how to find him in the slack community be like yes save this I I actually have not found that at all I haven't heard anybody come back and be like this is this is inaccurate you're on it man so thank you I will say if you actually literally do want to track me down you made me think of this because I just put this on slack if you're in the Bay Area and you want to track me down um Monday at like 4 P.M on at on the Stanford campus like I dropped a Lincoln slack to the actual like location and stuff almost a two

hour workshop and it's gonna be you know hopefully kind of like audience driven so if you have questions or you have things you want to talk about come on come on down and we'll talk about it and that is open to the public right Dave if I sell it correctly uh yeah that's open to anybody it's excellent yeah it's on the stand for campus but anybody can come yeah I just got news that's hard to hear me hopefully uh just yell I'll just yell okay yeah sorry get all over the heavy equipment behind you yeah you can tell look at him he's so irritated by

it but he won't actually say he's irritated by it phones man so you can link it right up in it and yeah there we go all right you know what Alex Jonathan's got it send him send him something all right cool yeah Jonathan hi welcome would you like to introduce yourself yes thanks Rose um my name is Jonathon Anderson and I'm a Solutions architect as well here at ciq and I've been I haven't been involved in the development of apptainer or Singularity before it as long as Dave has for sure um but I I was using it at a university site in a pretty typical

HPC use case in the past and uh am glad to be on the kind of support and development and benchmarking and all experimental interesting things with Apptainer front uh here within ciq in fact uh just before this I almost missed our queue to jump online and do this webinar because I was busy running a huge Matrix of uh benchmarks that maybe we'll get into uh later as we get to the end of the implications of what we're going to be talking about today Jonathan you know that I I kind of slid in at the ends too but I would have been chasing you down don't

worry I would not let it not have let you go too far yeah exactly exactly awesome okay Zane so where are we going with this we are going to be starting in case we did introductions that is amazing and we're gonna be talking about MPI with apptainer and there's a couple of little Kinks that kind of get in the way and we're containerizing MPI and what does that mean and how does that work with other things um so yeah I just take one step back and just for the people who don't know what MPI is Google or Jonathan okay that would involve me having

to read something and it's much better when Jonathan and Dave explain things to me more of an auditory learner myself totally yeah maybe I'll start with the the MPI introduction so uh MPI is probably the I shouldn't even say probably it's the most common uh practice in in traditional HPC for bringing multiple nodes together and having them work in concert on a single problem so MPI stands for message passing interface and it's a set of standards for writing your application in such a way that it is given a list of contacts for other other nodes other computers that are participating in an application in a

computation or a calculation that kind of thing and dividing that work up so that each node each computer in your job is doing a different part of the work and then MPI gives you the uh the verbs the the actions necessary to pass information between those and there are some simple like point-to-point messages where one node one part of the computation can send or receive messages with another and then on the back of that are collective operations for uh having all of the nodes share information to all of the other nodes or Collective operations that that reduce all of the nodes down to one result

that is received by one node that kind of thing and so there are a number of implementations uh some of the more popular open source implementations are mpitch open MPI and MVA pitch but then there are proprietary implementations usually forks from one of these open source implementations as well so like Intel has one IBM has one that are are kind of built and supported to run in specific Hardware environments but it's pretty typical for at least in the University space to have one or more of these open source MPI implementations on your system so it's really not as much of a they all pretty much

do the same thing if you write your code you have to write it differently for every one of them if obviously if one's a fork of the other it's probably really similar but if it's one of the proprietary ones are they are they similar yeah so they they should adhere to the standard the specification and in general what you'll see is most applications can be compiled against any of the common MPI implementations that's not necessarily the case though and most applications we'll have one that their developers tested against and so often you care about uh which MPI you're going to run your application with because

it might perform differently or better with different uh different implementations of MPI even if um it runs on them at all and it's not unusual for it to be able to run with any of the MPI implementations but it's it's also not unusual for it to have a preferred highest performing one but then that that also is a compile time concern um the the MPI implementation that you're building against has to be built once against that and now you have this compiled binary that only works with that MPI and you can't just mix and match it to all of the others so this is why

lots of HPC sites will not only have uh all of the mpis available but also we'll have the mpis compiled in different ways depending on how you might want to compile your Downstream application what compiler you're going to compile it against or what map libraries that kind of thing I can only imagine in containers that can cause issues well it's it's one of the problems that containers is meant to help solve so when you do this in one big namespace what's off what's typical is you have like a file system that's slash software or slash opt or something like that where you compile all of

this software in one place that's shared to all of your compute nodes but when you start having to worry about the Matrix of possible different combinations of all of these things together you're trying to build an environment that has all of the possible permutations of all of the software that you might care about and it gets very hairy and very big and complex very fast especially when you need to upgrade something now you have this huge dependency chain of things you have to update one of the goals of containers is to be able to say we don't care about all of the possibilities we just

care about what this one application needs and we want to be able to pack the libraries and the MPI implementation that that application needs with that application so it's one thing and you don't have to worry about all of the other possibilities that might be applicable on the system those can be in different containers and not have to be in one big shared namespace yeah I guess that's another topic as well is being able to string containers together so that's very cool so I have to say you said slash opt I feel like last time I said that I was putting stuff in slash shop

somebody laughed at me and told me I was old school is that is that normal now people don't put stuff in office anymore we do old school things in traditional HPC all the time I like thought that's where everything went optional stuff should go and opt who told you I mean thank you I'm a big fan of opt yeah okay I'm just making sure I wasn't alone here I thought it was a little odd that I was being laughed at for using shop still don't get me started man okay spreading out all the configuration files and binaries and stuff all over all the different directories

who thought of that who came up with that not me man slash opt is the ancestor of containers in my mind there you go that's a good point no wonder I feel kinship over I like that I like that so Dave you wrote a Blog it's about apptander and MPI tell us about it cool yeah well um don't read it to us just tell us about it yeah well I wrote them oh yeah if you didn't come Rose and I were gonna take turns reading paragraph by paragraph but it's much better if you just don't have to open why did we show up Dave that

would have been so great no no but it's so awful reading in the round before I before I go into um you know that blog post and talking about that I just want to acknowledge that Zayn and me are basically sitting across from each other in the same room right I mean really our backgrounds it looks like that's kind of what it looks like I agree wow okay um yeah I just want to acknowledge that we're really not we're like you know so yeah so okay so I want to preface all this by saying that um you know it's me and Jonathan tag teaming this

because even though I wrote this blog post uh most of the stuff that's in it was Jonathan's work and he and I had been both kind of you know we feel like we've given this webinar before even though we haven't because we've both gone to a bunch of different meetings but they've all been you know kind of like different fragmented groups and we've talked about this in a lot of different settings at this point in time but we haven't actually done the webinar on it which is kind of an oversight but um I think that uh so Jonathan you will have to kind of like

keep me honest as far as the background on this goes but um I think Jonathan started to investigate like MPI within containers and like there there are there have been for a long time sort of several different like models or you know several different ways in which we suggest that you run MPI inside containers there's like a fully containerized approach which is easy because it's only one node so if you want to use MPI basically to spread your process across multiple CPUs within a single node you can totally do that and you can do that by sticking all of your MPI implementation inside the container

um and then just running it but then there's things become a little harder when you try to reach out and go across nodes and um I want to say it like in the blog post that we keep referencing um hopefully we'll get some links and stuff too um I really lay out like one problem but I actually think and maybe we can talk a little bit more about it today that there's actually two big problems with with NPI inside the container and that blog post really only kind of talks about one of them so let me kind of lay those problems out um and kind

of talk about you know what they are and then we'll kind of go through and talk about you know how people have traditionally solved them and then the new way that we're advocating that people start to look at solving this problem so when you when you first start an MPI application so what you do is you write your code uh to be MPI aware and then you compile it against a certain set of NPI libraries and then you have that set of MPI or that implementation of NPI available to you on your HPC system and what you do in order to like launch it make

everything happen is from one particular node you say Okay NPI exec my process and what that does is MPI says oh okay you want me to go out to all these nodes and find that process that you just told me to start and start it either you know just one on all the nodes or multiple implementations on all the nodes whichever you've kind of set up and then you want me to assign each one of them a rank and basically tell them how to talk to each other and that first initial step is called is like commonly referred to as like the wire up um

of the uh of the job so if you just if you just do that on bare metal like I just suggested it works fine MPI you know lives on the Node where you started the job and it also lives everywhere else on all the other nodes and so it can fan out and work everywhere it needs to work and assign everybody Rank and do all the stuff it needs to do and you know everything's just happy and that process yeah there you go that process lives on all the nodes as well and that's kind of like the big key here is that um the the

process that you've told MPI exec to go out and find it it can go find it it's fine because it lives on shared you know shared storage or something and it just goes and finds it and starts it now the problem when you try to do this from within a container is now you're inside this little container you know on the Node and you say Okay MPI MPI exec this process and MPI says yes yes sir yes ma'am I'm gonna run out and do this and it uses you know SSH or rsh or whatever to go out to all the other nodes and then but

when it does that it pops out of the container and that's kind of the key thing and then it looks around says where's that process that I was supposed to start and the process is stuck inside the container so uh MPI exactly can't see it anymore so the MPI exec just reports back and says I don't know you told me to do something I can't do here's a bunch of errors that you'll never figure out what they mean you know why and then that's so that's the problem that's the first problem okay that's that's the first problem the second problem is uh for MPI to

be performant which you know if we're if we're dealing with high performance Computing we want applications to be performant so for MPI to be performant it has to have really really high speed uh interconnects in between the nodes usually just just plain old ethernet's not going to do it usually you need to have some sort of um infiniband or omnipath or some you know really high speed fabric between your nodes for them to talk to each other um the the problem this problem ends up being kind of similar uh to the way that um the the gpus work and that you've got drivers for for

the the actual hardware for this Fabric and those drivers they have two components they've got like a component which is tied to the kernel which is a kernel module and they've got another component which is the user libraries that are supposed to interact with that kernel module and you got to match them up and the problem is you get the kernel into your container at runtime just like because that's the way container works and so you get that kernel module and you don't control you know what you get but the libraries you have to match those up you do control what those are and those

have to match the kernel module and then that gets complicated because depending on how you've um compiled your application you might have compiled it directly against those libraries so it might be like well I really need to have you know version 10 of these libraries but my kernel is running you know version nine of the module and so basically my whole container now won't work with this and then Things become like not portable and you know so those are the two big problems the wire up problem and then like the high speed interconnect problem those are the two kind of big issues that that um

you know we're trying to address uh you know now and the first of those is the one that this blog post really deals with is the wire-up problem but you know we also have some ideas about the the high-speed Fabric and I think it'd be cool to talk about those a little bit today too so Dave 10 gigabit if people are building kind of the smaller environments they're doing doing their own thing on copper is 10 gigabit okay I mean not ideal but okay yeah so I don't even know what the current state of the art is as far as high speed Fabrics what is

about 100 gigabit I feel like I saw somewhere that there was a 200 gigabyte yeah 100 gigabit is about five years old now um state of the art anymore I'm cutting edge is like what I think 400 and 800 is coming out crazy yeah ten well and then it's very application dependent too right I mean it makes sense yeah it's like how how much essentially how much message passing is your is your application doing you know makes sense wow Dave sorry to cut you off safe yeah no I I feel like you need to take a breath or something those those feel like big problems

and so they're interested in their big and interesting problems but you've solved them no Jonathan's has solved them we have we've been making moves that we think simplifies this and particularly in containerized workloads so we're going to show a bit of that and uh um I don't know at Dave I don't want to skip over like your your flow what you're what what comes next what are we talking about next well I guess next we should kind of um summarize the ways in which people have suggested that this problem should be solved and then just kind of maybe provide some examples of that and like

show you know what that looks like and why it kind of sucks um I think that would be the best thing to do um so okay going back to the wire up problem um you'll recall so the problem is that you fire off MPI exec it runs out to all these other nodes it's outside the container and it says it says I can't I can't do what you asked me to do I can't find them I can't find the process that you want me to start on all these other nodes so what if you called MPI exec outside the container and you said the process

that I want you to run is Apptainer and I want you to app I want you to run Apptainer with these arguments so that you start up a container and then you execute a process from within the container so that should work um and so so what ends up happening then is that so mpis on the host system you give it an Apptainer command to MPI exec it runs out to all the other nodes it finds that painter it finds your container all that stuff is on the host system you know on the host file system and it starts everything up but the problem is

it also needs to be inside the container to handle doing things like assigning Rank and you know setting up the communication and you know just all the stuff that that MPI does and the actual running of the program and everything so you need it if you're going to do that you need it outside the container and inside the container both and um then not only do you need like some MPI outside the container and some MPI inside the container but even though there's like standards and stuff um it turns out that that uh MPI flavors and versions don't like one another at all so if

you if you start MPI on the host system and it's a different version than what's inside the container um it's almost always going to freak out and say I I don't know you know I'm having an identity crisis I can't work with this and it's not going to run or if it does run it's going to be not very performant or whatever um so then you're in this kind of you're in this state of trying to match the exact version of MPI which is on the host with the exact version of MPI that you've got installed inside the container and not just the exact like

flavor and the exact version but even the way in which you've compiled it can matter you know to make sure that the two work together properly so that's that's kind of like you know what you're running up against but um but you know that's the way that people have suggested that you know you should do it that's kind of like the the standard way that that we've been doing it for a while and if you're gonna if you're gonna have MPI on the host and NPI inside the container there's like two different methods that people advocate for doing this um one is um that you

actually install the entire MPI implementation inside the container you just match it to the one on the host and this is commonly referred to as the hybrid model it's kind of hybrid because it's the host MPI and the containerized MPI are working in tandem together and then there's a bind model where you you basically use the bind Mount um architecture of Apptainer to take everything all the MPI libraries and everything that are on the host and you buy and mount them into the Container at runtime to the hybrid model and the buy model are the two kind of like historical ways in which uh this

this problem has been solved so do we do have a comment David I think it I think it applies thank you for the comment yeah somebody's already doing it the right way I also want to point out Justin and David greed in the op so sorry go back to that Tanner so it says we use estron to launch Singularity no MPI on the host uh buying the IB libraries from the host of the containers it works well so yeah this is going to be um kind of like the third model but there's going to be some um there's there's a little bit of a uh

some some variation on that so this is to kind of do you know to kind of um uh look ahead at what we're going to be talking about towards the end after we show the other two two methods there's this new ish technology standard not really that new but it's called PMI and PMI is a standard for basically for doing that wire up that I was talking about and so if you compile a tool like um slurm which is what sron comes from to use PMI and then you've got the same PMI implementation inside your container then all this like matching MPI garbage kind of

goes away and you don't have to worry about that anymore now the uh buying the IB libraries and from the host container I think that we're um you know we're going to be talking about a way in which you you might be able to do even a little bit better than that uh through something called lib fabric that we've been um we've been looking at uh but I'll you know I'll let Jonathan talk more about that um when the time is right I don't know I could kind of uh talk about the older School approaches a little bit more and kind of like go through

some demos and show how those are done is that good now or Jonathan do you have something else that you'd like to add at this point I don't have anything else to add uh and yeah we can do that like uh Dave's been having trouble with screen share so I think that's still going on right so I was gonna well let me try because what I did is I installed the SSH client for Chrome and I think I should be able to share um my browser if I can't share anything else for some reason it seems like it's trying to allow me to share my

browser so I'll try that and you guys can tell me if you can see that or not I got it okay getting an SSH client in your as a browser extension yeah man this is horrifying to me I have a great story about this Jonathan at some point not about this one in particular but I don't know if you were ever dealing with Solaris but Sun as a troubleshooting tool used to have an SSH tool that you could go to a Solar Sun website and actually get a console up you could share the URL so that the sun admin from Sun hoping you could actually

see it too it was a handy and terrifying we could you know we should do a webinar on why this is horrifying I think it's like a totally like the idea so anyhow I don't know so I'm having trouble um previously I had a kind of an epic fail during one of these webinars where uh I really hit my screen share and uh it turned out that I had updated my system and it was running Wayland instead of X11 for graphics and because of that it wouldn't share I'm running X11 now I have no idea why it won't share won't share anything except for browsers

so I think we're gonna go back and put an actual link directly to that part of the webinar where you had to switch to a monitor on camera and turn it around that was fantastic can you see you guys can you seen it yeah that was a day all right so um anyhow uh so the the model that I was talking about where you um you uh actually install the MPI implementation into um into the the container itself and you actually have it on the host too that's called the hybrid model here and so let me go ahead and show you I actually I've got

my own implementation of um MPI that I've installed here in my home directory because we don't have a uh we don't have a version of MPI that is running system-wide on this system so I gotta export some uh environment variables just to make that available and so let me go ahead now that I've got that exported and I'll go into this hybrid directory and I'll kind of just give you a sense of what's going on here so um as I just said I installed MPI uh so I you know I I know exactly how this MPI was installed I'm like the administrator of this particular

MPI I know how it was installed and all that kind of stuff um even so even with all the knowledge that I have of how this NPI was installed I found it very difficult to install a matching version of MPI that was also compiled in exactly the same way into my container and so let me give you a sense of that so first off I'm going to be running this little hello world.c uh program let me just give you a little heads up of what this looks like so this is a this is a pretty short program that was originally written by Wes Kendall and

that Jonathan modified and what it does it just sets up uh you know however many NPI uh ranks on however many nodes you ask it to and then it just gives you some some messages it just prints to you um you know what rank it is what uh what host it's running on what CPU is running on and some other information as well no cows though no no cows today unfortunately man we should we should redo this if it's like an LOL Cal NPI um so that's that's the program that I want to containerize and run so to do that I have to install MPI

inside the container I also have to get that C file inside the container and then I have to compile it against the MPI that I installed inside the container all right so here we go so this is the definition file that I ultimately came up with so I started this definition file um from uh uh like an example that's up on the Apptainer uh on the apptainer documentation page and then I just sort of edit it edited edited the uh definition file until I got what I needed out of it and as I go through down the through this thing you can see it

became pretty complicated uh one of the reasons it became complicated is because um the MPI that I've got on the host system it is compiled against a bunch of libraries which we do have installed centrally which we have installed um here with open HPC and so in order to kind of get everything to work properly inside the container I also had to install some of the same libraries inside and some of the same applications inside of the container so that I could then compile my openmpi against those those same those same libraries and then here at the very bottom I'm just seating to opt directory

and I'm just running NPI CC which is this is the thing that's actually going to compile that NPI Hello World um C program into an actual executable this is pretty complicated it took me a while to get this right um okay and you know so I can't see you guys too while I'm screen sharing so if you're like doing like this or something I won't see it so if you if you uh if you want oh yeah I'll let you Dave yeah just holla and just holler at me and I'll stop all right so whenever you're ready for a question though okay yeah yeah so

I'm I've just showed you the definition file this is the um built container here and the next thing I'm going to show you is how to run the job and I'll pause for a push it's from Mr Dubose appreciate it are there performance impacts using MPI versus bind PMI PMI I'm sorry yeah I know that was confusing to me too I mean you could have named it something else right MPI PMI all the acronyms there's too many letters USM it is complicated uh so you got The Stave are there performance impacts using PMI versus bind um so here's what we're going to do I think

that I'm going to let Jonathan field the NPI specific questions and I'm going to field the container specific questions how's that cool so yes PMI here shouldn't in itself really shouldn't introduce any performance impact or or if it does the performance impact would be on initialization PMI is just used for starting processes on the Node once they're running they're the same processes that were would have been running before anyway so there there are uh more even more modern than the PMI that we're using right now um uh work happening in PMI called pmix this the PMI work for exascale for systems where the scalability of

the system meant that that process startup was taking so long that it just wasn't tenable anymore so that is an issue um that we intend with this work to to further address by moving to pmix but that's that's a work in progress still for us and our ability to use it in this containerized environment um but there are other performance implications that are unrelated to PMI that we're investigating right now but like I mentioned we're running benchmarks and things like that to understand what that is and then once we uh see where there might be performance issues uh dig into that with development just as

another little wrinkle to add to all this too people should be aware that they're so PMI is a standard but like with all good standards there's a lot of different versions of the standard so there's BMI one there's BMI two and there's PMI X and there are multiple multiple versions Within pmix um so it's still you know there's still stuff that you have to match up and be aware of and and but it's it's a little bit easier than all the universe of different uh npis that you could have different versions and flavors and so on does it have to be compiled code Dave can

you just have a compiled library and call it from something like python non-compiled or does even matter does what have to be compiled the the MPS enabled application everything you're doing here is all C code right it's compiled code what if I had just a Python program um they're like I think that there are so so you know I should back up and I should say and like I'm not a super huge expert in MPI um and this is because most of what I did at the NIH was like high throughput Computing so big parallel jobs well parallel groups of jobs but I think that

there are like parallel libraries for python I'm not really sure how they work though yeah there there are MPI implementations for python um and they ultimately wrap a c implementation so you you build that python extension against an MPI implementation MPI itself is ultimately almost always if not always uh implemented as a CEC plus plus and Fortran library that you build your code against so you can compile a python interface for a given MPI implementation and then write python code that that uses MPI it's not super common but it is a thing that people do but most of them sure people wanted to be fast

and efficient I get it but it's not much python as we see out there nowadays I'm just or they're scientists and mathematicians that just do everything in Fortran and I don't know Fortran but that's that's what people do GI GPT I'm gonna show everybody now my job.sh which I wrote myself I did not use chat PPT for this um thank you so this is this is the submission script that I'm going to use to to you know send this off uh through uh to slurm and so you know it's got these press batch directives I'm really what we're interested in today is that we're

going to run just um two different processes and the tasks per node is going to be one so I'm going to use two two different nodes one task per uh per node um since I'm using um since I'm using lib fabric I can go ahead and set this uh environment variable and it's going to give me some more information about um you know what's actually happening and what's the interconnect that's being used and then I'm just going to do an MPI run and um I'm gonna go ahead and do an apptainer exec so once again with this hybrid model you execute the MPI run and

then you give it an Apptainer command and then once you've got that Apptainer command you go ahead and give it so this is going to execute the following command so this command inside this container I don't know why I'm highlighting API hello world everywhere all right so um yeah that's my that's the way that I clear whenever I'm highlighting something okay so um I'm gonna go ahead and submit that with s bat this is going to give us the moment of cruise to see if I actually forgot anything or if this is going to work job number 599 let's go ahead and see it should

run pretty quick it worked um as we can see here so all the other stuff is just information that lib fabric is giving me about uh what it finds what it doesn't find what it opens how it feels uh but we can see here's another here's another three letter abbreviation for you guys PSM so if we can see that it used psm2 which um is not PMI is actually the fabric adapter for omnipath so not only did it run but it ran using the high-speed interconnect the uh the the omnipath fabric adapter and then down here we see you know we ran on

compute node 5 and compute node six so two different nodes two different ranks same CPU two different Mount name spaces because it's two different containers although it could be the same amount name space because it's on two different nodes but it wasn't cool so that's that's the hybrid model now let's talk about why the hybrid model sucks um so I already kind of alluded to the fact that this was a big pain for me to build this container that was a complicated definition file it probably ended up with some garbage in it that it didn't need to have in it just because I was doing

trial and error and trying to figure out how to make it work and once it did work I didn't want to mess with it anymore um but just that wasted time that's not the whole story because if I take this uh this container now that I've built anywhere else and I try to run it again unless I have exactly the same open MPI um compiled in a very very similar way it's not going to work anymore so that's one of the main advantages of having containers right is you know you can take a container and throw it up on Docker Hub somebody else can grab

it and run it you can grab it in the future and run it somewhere else I can't do that with this container it's not portable anymore and that's a huge drawback right um and it's one of the one of the big reasons I think that Jonathan you were talking about this in one of your previous presentations it's one of the big reasons you don't see a bunch of MPI enabled codes that already exist up in places like Docker Hub that you can pull down and just run because they're they're too bespoke they have to be compiled you know um more specifically for the applicant for

the uh the environment where the application is actually going to run so Dave excuse me sorry what what do you mean by it's not portable anymore is it because the MPI had to come out of it to run properly it's because I had to match really carefully um so there's multiple flavors of MPI there's open MPI and Pitch MBA pitch so I had to first off make sure I was using open MPI because that's what I have installed on the host but not only did I have to make sure that I was using openmpi but I had to match the version the actual version of

open NPI that I was using inside the container to what's already running on the host and not only that but I had to make sure that so you can compile it in lots of different ways you can compile it against different sets of libraries you can emit libraries and you basically lose out on feature sets and stuff if you compile it in different ways but I had to make sure that the way in which I compiled it was also very similar to the way that it's compiled on the host and so because of that you know it's it's really matching very tightly what's on the

host and it's you know I'm not going to say it will not run on another in another environment but it would be pretty lucky if it did thank you yeah does that make sense I mean it does make sense but I I mean I guess I don't like totally understand all the details because if you have that specific MPI on another machine or somebody else had it I mean they could still pull that same container and run and run it right I mean yeah like you could the goal for the the container though is that you shouldn't there shouldn't have to be a caveat like

that right there shouldn't be this container will run in your environment if you have all these dependencies or at least you want to reduce the number of those dependencies and certainly reduce how specific they are and so uh Dave if you pull up your um your job file again or catted or something um that MPI run there on the left is installed on the host and then MPI hello world inside the container is also built against an MPI and with those having to match we we say that the the container itself is not portable to systems that have arbitrary mpis or what our goal is

to not not even need MPI like our our previous poster mentioned you shouldn't even need the MPI on the host because it's really just an application concern okay great thanks for clearing that up I think we actually have another question is that is that true just do we want to pop that up now and Dave we're kind of running out of time so we might want to jump ahead just to um to the the final thing rather than yeah really really quick just to show how awful it is and then yeah okay I won't actually run it oh well uh sorry the question I was

gonna read it is it even worth using external pmix versus PMI 2 included in slurm for smaller systems we got bit by some changes with the Rel and pmix and currently fall back to PMI 2 at the moment yeah so we're using PMI too because um the open HPC installation of slurm that we have on this system uh doesn't support uh PMI actor pimx um it just supports PMI too and I guess there's some reasons like maybe some security types of reasons for that but I don't know all the details of um but yeah yeah we're using PMI 2 here and I think that Jonathan's

even finding there's some reasons even to go back and use PMI one or just yeah so what I've what I've noticed is that if you compile an application with pmi2 PMI 2 appears from what I've seen to to have some of that same MPI problem where there's a standard but then the implementations kind of took it in all these spaces so if you build an application or if you build an MPI implementation against pmi2 it ends up pulling in an implementation of pmi2 to build against which looks like by default it typically is building against slurm inside of your container and I didn't like the

feel of that and I also we hadn't tested yet whether you could build an application like that and run it on some other pmi2 supporting resource manager I was concerned that that we were losing some compatibility there so it was pretty popular but you know we wouldn't want to make that a requirement so for most cases right now I've been trying to build against pmi-1 or what mvpitch calls like simple PMI and that's still been working just fine the reason I'm most interested in pmix is to get away from that resource manager dependency uh so that we could be building again against a standard or

an upstream implementation rather than against a specific resource manager like slurm oh did you it's okay Jess you can put that question yeah yeah that's cool like to engage with our people just put oh as a comment um response yeah yeah okay so just really quick um so the bind model you'll remember says don't install MPI inside the container just grab it from the host at runtime and buy mount it in and so if we do this what you have to do is you have to compile the MPI hello world script on the host and then copy it into the container at um at runtime

or uh at build time which is what I'm doing here with this files yeah this is not the NPI hello world.c anymore it's just MPI hello world and then the environment here is just setting the path kind of similar to how you saw me set the path earlier although there's some other libraries and stuff on the path here because they've already exist on my path on the host um okay so that's a little bit simpler because you're not actually installing NPI inside the container but um there's a caveat and here's a caveat um if you do this you have to bind mount all of the

MPI implementation on the host into the Container at runtime that is and then once you do that once you buy Mount that in you actually have to start buying mounting all the libraries that it's linked to and you have to go down this Rabbit Hole where you just start by mounting all this nonsense into the Container uh at runtime and it ends up just being like what you're doing is you're by mounting in stuff that was compiled directly against the glibc libraries on the host and that's kind of what you bring with you into the Container so this this um method is like almost assuredly

going to break um that's that's basically the bottom line here is like not only um is this like you know I mean I guess it is more portable than the other method and your application's still dependent on that specific MPI version though yeah I I compiled it on the host yeah and then I copied it into the container and then I copied the MPI into the container and then I copied all the libraries into the container and so you start to wonder what's the point of the container why am I wasting all my time just copying all this stuff into this empty container just to

say that I ran it inside of a container when I was done there's no you don't get anything out of it so that's I don't and I don't think the buying model is actually you know very widely used and that's kind of a reason but anyway if I ran that it would run you know and you'd see the same the same uh messages that I saw previously I'm not gonna waste time doing that because we are running out of time I'm sorry I'm so long-winded all right um let's go to the fully containerized version this is or PMI rather okay in this example we're making

a few different changes um one of these is uh the the definition file now instead of just like creating this whole definition file from scratch which would still be kind of a complicated Endeavor even though it would be easier because I don't have to match a particular version we're going an even easier route we're just using spec so SPAC is a really cool tool that we have not talked about a whole lot in um in this webinar I don't think but you know it's been around for a while it's um it is a package manager which is kind of uh geared toward HPC which tends

to build everything from Source essentially in a nutshell um and you can make these little spec files that you can use to install stuff in this case I'm installing a specific version of gmape along with OSU micro benchmarks I don't think I actually have to install this for this container to work but I'm also installing open MPI I'm specifying what Fabrics I wanted to use and what options I wanted to be compiled with and then I'm installing a lib fabric so that's something that you know hopefully we'll be able to talk to a little bit if I hurry up and get through this quickly um

but here's a cool thing you can use SPAC to build containers and so one you know when you have all these parts down here which actually specifies some details about the container that you want to build um you can use a command called SPAC containerize and it will spit out for you a definition file now in this case um I've got this definition file here that I created with the spec.yaml and then I edited this definition file myself to add in this MPI hello world which is kind of ugly I probably could have figured out how to just do it all with the spec file

but um you know just to be quick and dirty about it I just went ahead and edited it myself um but so the the point of all this is that once you you know get this together um you can you've got everything completely containerized and I guess maybe what I should have pointed out in the spec Dot yaml were um putting psm2 support here in the slip fabric oh and PMI I'm sorry we have PMI support in openmpi so what this allows us to do I'm going to show you this command I'm about to run and this is very similar to what one of our

viewers were talking about earlier we're just going to use Astron and when we use sron we're going to pass this command that says hey use PMI 2 for your MPI wire up and then we're just going to tell the number of tasks number of nodes and we're going to give it once again that apptainer exec and one of the cool things about this is this works regardless of whether or not you actually have NPI installed so let me just go ahead and run it so I'm just gonna execute the command the kind of ugly way which to be clear is why this environment didn't have

MPI installed and why in order to demonstrate otherwise Dave had to jump through all those Hoops to uh to have a system level MPI we wanted to be able to show that this would run without any MPI system-wide yeah in fact I guess a quick and easy thing that I could do if I open up a new shell I mean you you guys saw me run those export commands I guess if I open up a new shell I'm not going to have that um it's not I'm still going to have those uh environment variables set but what I could do is um I love how

the shell is just dingy like crazy yeah what what is that what does that mean it's hitting tab a bunch yeah it's just it's giving me the finger and saying do it yourself it's awesome there we go all right I'll unset that that should break things right watch it watch it we'll break things it'll be really sad see it still runs so now I I just unset LD Library paths I effectively don't have I don't have NPI I've got the MPI binaries still on my path but if I actually tried to execute them they would break and you know because they don't have the libraries

that they need anymore and it still runs and it's still if we go up and we look let me see if I can find it ah here it is it's still using the high-speed interconnect so it runs and it runs in a performant way cool so okay I left this like six minutes um but I think we should talk so that's that's kind of like the new the more modern way you know this is this is um you know this is portable now you can take this container to your point Rose and as long as you've got slurm which is compiled with PMI which is

going to be coming you know it's going to become more and more common is already pretty common uh you can take this to another place and you can just go ahead and run it without you know having any external uh MPI implementation so that's the first problem that's the wire up problem now let's talk briefly a little bit about the interconnect problem if there's time we have a few minutes for sure cool so I think let me set the problem up again so I think one of the problems um historically has been you've got this kernel module and you've got this set of libraries and

those have to match but then on the other side of the libraries you've got your MPI implementation which was compiled directly against them and those have to match and so now you've got this thing that's got to match from outside the container all the way through or from inside the container all the way through back to the outside of the container and so now it's not portable anymore um enter the fabric yeah are you throwing to me now Dave yeah okay so uh lip fabric there's a couple things going on here one is that it appears and we're still testing this but it appears that

lib fabric is portable between different kernels so Dave and I were before the webinar just uh musing about whether this was true or not um because we've we've seen I think there was someone in I think it was the Apptainer community that was having trouble because they backported the rocky 9 kernel into a rocky 8 System to get additional Hardware support and it broke infiniband or something like that and yeah and that was because there's like they've just said a user space portion that was still there from the rocky 8 side but now the the colonel side was Rocky nine uh so uh so far

it appears like lid fabric is portable across that and my anecdote right now is all this benchmarking that I'm running without thinking about it I was running a rocky 9 container uh on my Rocky 8 hosts and I didn't even know so we want to do the full Matrix of that and do some performance testing of running a rocky 8 container on a rocky 8 host and then both sides of eight on Nine and Nine on eight and then 999 just to test all of that and make sure that there's no weird performance gotchas or that one of those permutations doesn't create a problem but

so far it's looking good the other is that historically you've built an MPI against a given Fabric and so you you would think that you need to uh you would think that you need to um build your MPI on your system to build it against the high performance fabric that you have and so that might make you not want to have to all have it in a container that you downloaded from the Internet or something uh so like historically for example you might build openmpi and when you say Fabrics equals you would say Fabrics equals psm2 or opx the the new omnipath one or maybe

you say verbs or IB or whatever the the thing would be to build native infiniband support well now one of the Fabrics that open MPI supports and in fact all of the major open source implementations now support is lib Fabric in fact I think mpitch and and uh and evapitch just only support lip fabric now um so that's where you have your one-to-one support those those are built against lip Fabric and you must have lip fabric but lip fabric itself is a multiplexing library and if you saw in that spec uh spec definition that Dave had um the Fabrics that it supports is not just

psm2 but a list of fabrics that you're building in support for so you can just do traditional network sockets you can do TCP or UDP you can do verbs which I think is still the preferred kind of interconnect for for infiniband melinox infiniban or psm2 or opx which is what I've been using more recently um oh yeah and and an EFA one um that I we're not building here for some reason and the last time this came up I was trying to do it in the middle of a webinar and I still haven't gone back and done it but um yeah you should be able

to build this one container with this one lib fabric uh build in it that supports all of these Fabrics that you can just take it from one to the next and it would just be portable across them so that's the goal that's what we're working towards and so far it's going well but we're trying to be cautious and uh actually go and and test all of our assumptions about performance so that's what we're doing right now uh if you've done any kind of comparative benchmarking before it quickly turns into a nightmare of permutations and and Matrix uh comparisons so uh my my hope is to

have a new blog post out about that in the coming weeks where we'll at least report what we've observed from a startup standpoint and actually support getting this to run on things it's going well um we just want to make sure that you're getting the best and Optimal Performance that you can uh with the software you're trying to run so that is our future work yeah so just to kind of sum that up I think lib fabric is like an abstraction layer that just goes out and says you know what do you got give me what you got and I'll run it and in that

way it's it's kind of cool because it's kind of like it's kind of like Cuda the way that Cuda works with the Nvidia driver you know it doesn't there's not this tight correspondence anymore between the driver and also the hardware you can kind of go out and say yeah I can run on multiple different things tell me what you got I'll tell you how I ought to run it so that's encouraging it's great and I really appreciate the work that you guys have done Jonathan know you spent a lot of time on this too looking forward to the next installment of where we're going with

this it's exciting thank you very much it just keeps growing without bound so I'm looking forward to actually having a conclusion to it will there ever actually be a finished conclusion I don't know Jonathan it just is going to keep expanding just like the whole universe right yeah we just keep the more answers we find the more questions we have and that's it's both a good and a bad thing yeah it keeps it fun for sure well thank you guys um thank you everyone who showed up thanks for the questions in the comments and your attention uh we are more than happy if you are

not watching this not live go ahead and leave a comment feel free to go to our website ciq.com reach out to us we will all get the emails we will respond to you we're very excited about all the things that we are um creating and working on and ways that we can support you and like as a smaller company which is really cool we have the opportunity to be very responsive to our customers and what it is that you guys are doing and needing and that's part of one of the the joys of like their never-ending questions and things to solve so we're very happy

to do that so make sure that you like and subscribe share this with your friends um leave any suggestions on future things that you want to know more about and we are so glad that you are here thanks guys for being here see you next time and go see Dave at Stanford on Monday Monday that's right yeah it's not four it's two it's at like two or two thirty yeah so the link should two to four go say hi see Dave ask questions cool awesome thanks guys thanks everyone appreciate it