Container Education Series: Apptainer MPI Benchmarking

hey guys welcome yeah thanks Rose happy birthday Rose yes thank you I also joined Zayn in updating my calendar so we have to wait until next year to see if I remember the whole year now you'll remember I'll remind you I'll remember this and I'll come back and remind you right exactly good okay so I did my job I shamed everybody very much so Jonathan Dave it's good to see you again likewise today like I haven't talked to you since yesterday yeah it's that's it's there's a reason that you feel that way accurate statement it happened so it seems like it these topics come up

hey Wolfgang at interesting times because we start having uh customers start asking a lot of questions and we start having conversations around Apptainer again it seems like we go in waves here a lot of stuff's been coming up about MPI and Apptainer lately so I am really excited to hear what you guys are going to talk to us about today and I have like the outline yes but why don't you give us some context Jonathan great so one of the things that we um we've talked a fair bit about here and kind of just in general at conferences and and within team meetings and things

is more application for um for fully containerized MPI workloads where um either because of it's been complex in the past or people are afraid of performance problems um people have gotten generally accepting of and okay with kind of single process single node um workloads in in Apptainer um but we haven't had a lot of data around what that looks like for MPI now that we've kind of unlocked some of the complexity or whatever if we uh undone resolved some of the complexity around deploying it in the first place um so uh you know Dave's been doing some of those presentations here uh recently but meanwhile

I was we were doing a kind of separate project a Warewulf deployment on some equipment that was very graciously provided to us by by Dell and Cornelius networks they're the the makers of omnipath if you're not familiar um where we were demonstrating deploying um just a small little HPC cluster um with with Warewulf and you know I was writing that up for an article you can find that on the ciq blog if you're interested that deployment went pretty well and you know does a I think a good job of showing how simple it is to get up and running but the very end of that

article was supposed to be and look you can run some MPI benchmarks and we'll we'll use this new MPI technique that we have and I say new we we felt like we discovered it we found at least another group that was doing a similar thing with PMI but that's great because it means more people are using Apptainer with MPI but the more I started running benchmarks the more I uh I just kept delving into it and having all these permutations uh I think at one the final tally was something around 64 different Benchmark runs um that clearly was not going to be fit just in

the conclusion of the Warewulf article so it grew into its own thing it is also up on the ciq blog but um I'm excited to talk about some of the results of that can I can I back up yeah please I'm trying to leave a space here because I saw you trying to talk Dave sorry about that sorry yeah I feel like it so so I I feel like I'm always the like the um you know the least informed person in these and so I feel the need to like sort of back things up and make sure that we bridge the gap between everybody else

so uh I think um they've deboned us uh in the chat has indicated that he's unaware of what MPI is I know he's just kidding around but um yeah but maybe we should back up and we should just level set a little bit yeah yeah that's exactly right David thank you for um helping to clarify that for everybody I know so no no no of course um yeah so you know we tend to just kind of like Dive Right In and assume that everybody knows exactly what's going on you know because isn't everyone in in HPC expert Dave yeah yeah I mean maybe maybe not

yeah thanks for keeping us on us both uh devonis and God love here so just in case you're not an HPC expert if you're just coming to the webinar to um you know kind of learn a little bit more about HPC and stuff just for level set MPI of course stands for message passing interface and I often describe it as kind of like bread and butter high performance Computing it is one of the ways in which you can or the way in which you can take um an application and instead of just running it on a single node you can spread it out across a

big pool of nodes and you can make it do things like share memory and you know sort of appear to be running like on one big computer instead of like a collection of small nodes and so the way in which you do that is by this this message passing interface protocol um there's a bunch of different flavors there's a bunch of different versions and when it comes to Containers um historically it's been a little problematic because there's like two stages of it one stage is sort of like wiring everything up and getting everything set and the second stage is actually running everything is set up

after everything is set up and so recently you know in a previous recent webinar we've showed how to make that process a lot easier uh by using something called PMI which is a standard for the wire up part of that that we that I just kind of described and so now the next logical question as Jonathan to kind of bring it back to what he was talking about yeah is um that uh you know so now the question becomes okay well in high performance Computing the P is for performance um if you're containerizing this stuff does that negatively impact performance at all and so that's

kind of yeah a couple levels in a little bit yeah so like they've said there there's um MPI which is kind of a loose standard with you know a Consortium around it that helps Define what different versions of the specification uh need to be able to do and how programs will be built against it and then there's a number of implementations of MPI and um in this testing what we'll be talking about today uh I focused on what I think is kind of the the three primary uh open source variants there's M pitch which comes out of Argonne National Lab uh open MPI which actually

I now that I'm saying it I don't know where openmpi comes from um but it it it's hard to say which one of these is more popular um those are are the big two to me it looks like Dave's going to go Google where open MPI comes from um not if I get to it first and then MVA pitch or mvapitch I'm not 100 certain but that comes out of uh OSU and that is particularly notable to me because they also develop a suite of benchmarks the OSU micro benchmarks Suite that they use in the development of their MPI implementation and uh we then used

as well uh and so we used three of the benchmarks there one is their OSU BW or bandwidth test and one is their OSU by BW by bi-directional bandwidth where it's sending messages in both directions at the same time and then they're all to all which I used as a just a a baseline latency test um and yeah the results were interesting but when you have three different implementations and three different tests and then a bunch of different circumstances that you want to test them in it gets complex real fast uh keeping track of all those results and and making sure they're all running exactly

the same way and you just get a whole bunch of different different benchmarks runs and variants Dave have you found out anything for us yet so the Wikipedia is that open MPI uh represents a merger between three well-known implementations ft MPI which came from University of Tennessee La MPI from Los Alamos and La MPI from Indiana University okay I had heard that before that it was a merger but I didn't know even then uh of what so that's great and then there are a couple um popular you know somewhat proprietary implementations of MPI as well there's there's Intel MPI which I think is really a

fork and variant of mpitch and then IBM has one that they uh that I think they're currently calling platform MPI um and I think that that's a variant of open MPI but I'm not a hundred percent certain of that I'm more certain of of uh Intel MPI being in pitch um but Dave was talking about wire up and uh historically each of these have packed an MPI run or MPI exec application that does things to SSH out on all of the nodes that are in your job and uh and start a process on each one and assigns each uh what is called a rank it's

just an integer so each each process in your job uh has says I am rank one or I am rank two or whatever and then you can have the different ranks send messages to each other or do Collective operations like send this out to all of the ranks or get something from all the ranks and we'll gather it together and sum it or something like that and just build up more and more complex operations that you can do on top of these Primitives and that it's like they've said been been kind of the foundation of what at least in my mind means high performance Computing

there are some other things you can do there's um uh some research around um P gas or partial global address space or some partition global address space I think is what it is one of the those where where MPI might use remote memory to do its work but fundamentally it's sending messages back and forth um pgas tries to actually map the whole thing as one big address space and and give you native access to those semantics and you know you you can do other things but inside of HPC it's largely MPI and some variant of it and then other things that are a bit more

experimental so ignorant question whenever you say wire up you have to go do all of that or the software does it for you you just use the software yeah so historically um you've had that MPI Runner MPI exec command and you've handed it a list of the nodes in your job and maybe your resource manager like slurm will give you that list usually in a file that's one node per uh per line and then maybe it has some metadata to the right of it of which course should it use and things like that and that got increasingly uh proprietary to different implementations so you'd have

to know how your MPI is going to do that wire up and and tailor it to work that way um and then it just got more complex so what we have been advising people to do is is try and abandon all of that and do it with this process management interface that Dave referenced that's a more standardized way of doing that and there are other benefits too around the metadata and reporting of what comes back from that wire up from the instantiation of your your big multi-node process but a big thing for us is that you no longer have a part of your MPI outside

of the container doing the wire up and a part of your MPI inside of the container that needs to be wired up in the past when you've done that the classic way those two have had to be in sync but if you're doing it all through a standardized protocol that's no longer the case you just have a resource manager that supports that protocol and it can do the wire up for you a bit more automatically than you had to do it before and it solves the containerization issue at the same time so that then brings containerized MPI back up to the Forefront whereas maybe people

before thought it was just too complex to look at now that it's simpler people the next question is well how does it perform and so we wanted to it it's easy for us to kind of wave our hands and say it's fine it should be fine uh but I was uncomfortable with how much we were saying yeah it's fine without actually going and doing the work and so uh you know it was an interesting project to dig into so I very good we can start getting into what the questions are that we were trying to answer but um Dave or Zane or Rose anything else

that we should talk about to set a foundation or groundwork for this before we proceed with it what do you think not that I can think of I think you're going to get into kind of the how you went about this in terms of yeah recreating it 64 times and making it look equal every time you know all I mean maybe all two at a higher level this is like this is a a special case of you know containerization um benchmarking really you know it's because this is a question that's come up quite a bit not just within within the context of MPI but also

just within applications in general I remember you know during the earlier days of uh you know when Apptainer was um kind of a newish thing uh people would ask quite a lot you know okay so great you know we're going to start using containers in HPC and that's wonderful but what's the performance impact you know now you're running you're running your application uh you know set aside MPI for a second just think about the application you're running the application through this like this additional you know thing how much uh how much um overhead do you get from that additional thing and the answer is uh

you know it was kind of surprising to a lot of folks I think but if you think about it it's not if you think about what's actually happening it's not surprising at all because the answer is like almost not it's it's negligible um the the older uh version of apptainer the way in which it used to work is so it would find your file system you know is your container Mount that to the host file system create a new Mount namespace which would present uh that new file system to the processes inside as though it were the root file system and then it would exec

whatever you know command that it was that you were going to start whatever process that you would start and then poof that painter was gone and so all and all it was was your process then running inside this new you know this new namespace and that was it um it's a little bit more complicated now because there's a there's an uh like a shim process which starts everything up and is the parent process and sticks around but it's actually not really doing anything other than being like a placeholder and so it's still kind of the same idea is that now we've got this like little

starter process that pops up and spawns off whatever command you're trying to run inside the container as a child process and then it's just kind of sitting there as a placeholder and um you know so every so it's there's a there's a you know however long it takes to mount a file system and then spin up a new Mount namespace which is minuscule you're talking about you know microseconds to milliseconds and then after that uh you know everything runs at bare metal speed so it's kind of cool and you know I think people come to this with expectations from virtualization right and they're used to

thinking that anything that Maps one system inside of another is in some way nested and that there's a layering and an overhead effect of doing that um I forget where I read this but it's more useful to think of containers as you're always in a container you just have like the the main one that your system is in and yeah this is more about you can have more of those they're not so much stacked on top of each other as it is just a way to tag things and and say what resources they have access to while they're running I'm gonna hold on to that

I like that yeah yeah that's true it's just whatever names I mean it's it's all managed by the kernel and it's just it's just namespaces in the kernel yeah and it's increasingly so one one uh I saw I think it was a talk with one of the BSD developers that was because that was where a lot of this work first happened at least in in the open source sphere maybe Solaris was doing it too um but BSD had like the total jail system and when they were namespacing uh different parts of the kernel so that you could start up like a whole network stack and

attach it to a jail and then tear it back down again and they were discovering things like no one had ever had to shut down a Network's namespace before without shutting down the computer and they kept finding things that when you tried to shut down that part of the kernel it wasn't cleaning up after itself properly because it had never had to do so but like that's the work that's been going on inside the kernel it's not actually a layer cake like it feels like it is um yeah so it's cool and and we're here to to give people a little bit more of a

warm and fuzzy around feeling good about running their HPC codes in a container because our results are pretty positive um do you have a question Rose you guys are kind of like went around answering it as I was thinking of you know my questions were just kind of you know pretty simple like um how do you know that it's negligible Dave right like how do you know um who cares and what is like the actual benefit of using a container great question yeah so I think we kind of um we kind of talked a little bit about the you know why do we know it's

negligible because number one because we can understand the architecture number two because we you know understand the architecture is one thing but actually doing the benchmarks and making sure it is another and you know that's that's both been done like I think your second question was like who cares who cares yeah um yeah so that's like uh scientists who are um you know when you get it when you get deep into like really big high performance Computing codes um you know milliseconds matter uh you know if you're if you're burning if you're if you're burning you know some Cycles on something else in addition to

your your code that adds up over time and so people who are like you know doing you know I don't know big molecular Dynamics simulations or maybe big weather modeling simulations you know really big probably you know all kinds of DOD stuff and things like that but really big simulations that take up a lot of the system um it's important to ring every single cycle out of the system and so that's who cares is is um you know uh research scientists who are really trying to ring the most out of uh HPC systems and then your third question what was it so oh yeah just

like uh the the benefit because I mean obviously you can do this without containers yeah um yeah so I would say the benefit of containers in general is just uh make it easy right and you know this is like this is a benefit which has been largely absent from using containers with MPI workloads for a while because it hasn't really simplified your life to a great extent so so one of the benefits of containers is once I get the hang of containers I can install stuff inside containers much much easier than I can install stuff on my HPC system um and that so that's you

know ease of use and then portability once I install it once I should be able to grab that container and pull it to another system or if my admin updates the system I should still be able to use my container that kind of stuff um those benefits until you know uh if you weren't using the the workflow that we've been advocating recently uh we're largely absent for MPI workloads because it wasn't easier you had to go and figure out exactly what was installed on your system and match it and that was very complicated and stuff and it wasn't portable because once you installed it for

a bespoke system you couldn't move it around anymore um but yeah if if you're able to use those PMI kind of trick that we've been advocating recently then yeah it makes um makes your containers uh easy to install portable shareable you know all that kind of stuff and so those are the benefits and uh they come without any cost of performance basically as Jonathan's going to show oh we get to see something oh yeah yeah I've got graphs awesome so the the questions that I you know like I said it this started blowing up and I just was asking a question and question a question

like I actually ran more tests than I ended up graphing because there was just you know I was interested and curious um but the the things I decided to focus on for the purposes of the article and what I think is interesting here um the first was uh does performance differ do we see a performance Delta um and just under Rocky Linux 8 environment if we run on the host uh or a container and because the graphs for these are together I'll talk about the second one as well does the performance change uh if that container is a rocky Linux 9 container so um the

first of these is just the containerization question uh does do we see a performance uh Delta uh for containerization versus bare metal so I had this Rocky 8.6 I think cluster and I tried to match exactly a rocky 8.6 container as well to see just isolating the containerization and then also a rocky 9 9.2 latest container with the same application built the same way inside of it um running that way and let's let's get this I should have shared this before um here and there and here you do sound better by the way Jonathan your voice is coming through clearly okay I just had it

a little bit further away I guess um excellent so yeah uh these were our results with uh bandwidth um so our our blue line here uh is the performance on The Host uh these tests were run with mpitch um so blue is the host red is the container and then yellow is uh the 9.2 container and you know we see a little bit of variance here um but one thing to to admit is just the limitation limitation of our testing methodology here these are pretty light results um I ran each test three times and took the third one for superstitious reasons um but a bit

more formal process uh would be you know to run this many many times and then average um but we can see that you know our curve is basically the same and then as we get up to the top our our limit is not any one of these we get practically identical results for the three of these and then I think the order yeah is by MPI implementation we saw the same with mvpitch we pretty consistently saw worse results with MVP pitch than other MPI implementations which was interesting to me and I shared that with cornelis and and never never found an explanation for it um

but uh back to you know similar results for what we saw with mpitch with openmpi regardless of whether we're running on the host or a matching container or a a different version container we saw basically identical results across the board and so we've talked a bit about you know the concern at least the the the the the fear that maybe when you're running one system inside of another you would have a performance problem um and and you know I'm happy to feel like this is put to rest but we haven't really talked about the difference between a a container that matches the host which is

is one level of test but what if the container is something completely different you know we could take this even further and build a container that's Ubuntu or suse or something like that um but this was you know a first step of this test because when you're dealing with high performance networks there's multiple Parts multiple components of what the implementation for that Network needs to run uh so one is like the the kernel driver support that's in the kernel and and when you're running a container you're running the same kernel as the host regardless your your container doesn't bring a kernel with it so Hardware

support comes from the host but there's also application Level and then intermediary code that supports that high performance fabric as well so it's possible that if we had enough of an OS Divergence um that the libraries on the container side would become incompatible with the kernel support on the host side at some point but getting a major version out and seeing you know no certainly no functional problem we were able to verify with all of these runs and all of these environments that the MPI is detecting the high performance Fabric and utilizing it appropriately um uh but that the performance was not impacted as well

so uh again like Dave said the goal here is to be able to build something once and not have to worry about if the host changes and so the results here at least so far imply that uh you know your your container OS and your host OS can be at least within a major version of the same type of OS Divergent and not have any noticeable performance impact so this is a bandwidth test this is measuring you know the speed at which um one node can send data to another node in megabytes per second um we also ran a set of like I mentioned bi-directional

bandwidth tests we had similar um similarly identical uh results here but the the other side of this kind of a test is latency not just how fast can the traffic go but is there a delay in sending it and so that's what we used the all to all tests for this is another set of benchmarks and here we actually saw so here the the graphs are different lower is better here because this is measuring the delay of traffic from one node to the next and we actually saw improvements um both first from containerizing and then uh from the next one I'm generally suspicious of this

and this is again where I say you know our testing methodology was relatively casual a more formal test might reboot the nodes in between each one of these tests that's where I saw one of the biggest performance Deltas were where the performance was worse than I'd seen before and then I rebooted all the nodes and all of the tests came back down again so I wouldn't make any big claims about oh look containerize your code and your latency will be better but it was at least good to see that back to back it wasn't worse um so yeah that was our first set um Dave

any any thoughts on this like was did you have any concern with with latency that's where I was the most concerned was that we might through names facing Hit latency issues but I don't know what your expectations were for this I mean yeah this is this is basically what my expectations were that would you know we wouldn't see any big differences um but it's it's good to you know it's one thing to sort of assume and it's another thing to actually test and so it's nice to actually have these you know as casual as they are it's nice to have these results in hand and

to have you know another another set of uh tests showing that you know using these methods doesn't negatively impact performance yeah I'm going to take a break from these real quick and pull up this you know this might be getting a little bit in the weeds um but just while we're talking about methodology um I'll show and before we get to sorry before we get too far off of it uh Greg's actually watching cool and um he's got a few comments he's you know he's got some ideas about maybe maybe the those differences in latency might not be uh artificial and um you know there

might be some reasons why they they might be some real differences some real improvements in latency when you containerize your uh your code so I think his first comment if I saw it come up real quick there um was that uh it might be that there's newer versions of the libraries that shouldn't be the case here it's not shown in this set of commands I'll show it in the next one I did all these builds with SPAC and so all of the libraries right down to everything above G lib C I think basically should be identical between these runs from OS to OS and on

the host versus the container so even like the the lib RDMA and all of that should be exactly the same between them um but here's real quick uh some of the the ways that I ran these so I'll go up to the top so here's um like the bare metal tests that we ran for mpitch uh the bandwidth test so we're running this on a storm cluster um I made sure that we were always running on the same nodes so we didn't have node to node variants between uh two sets of tests we're using PMI to do the deployment and then using SPAC to like

part of the job is loading up SPAC this is the the non-containerized version so we're just loading back in there and then uh accessing like this build this is the spec that I used to do the build and then running uh um OSU bandwidth out of it and then buy bandwidth and then all to all and then we have a variant on the Alti all that we'll talk about a little bit later and then the same test for containers there's uh M pitch in a containerized environment so here we have a container that had already built it so I didn't have to specify which um

which spec spec to run because there's only one inside of this container but as we've demonstrated before we're able to just execute this Apptainer Sif as an application and then the way these are built we're able to refer to the Benchmark that we want to run inside of that container that was built in with it so with these again we're running on the same two nodes everything else about it is the same except for instead of loading up spec on the host and executing the Benchmark out of that we're referring to this container and then running The Benchmark out of that and then I mentioned

a spec so I will switch to that we've published all of these on the blog article that has gone up today as well and I expect will be linked uh in the description of this video too if not we'll try and get that to be the case um but here let's see what happens if I do this great um it's just the full spec both what what we asked for and what we got so that anyone who wants to replicate our results can see exactly what we ran so the waste back works you say what you want to build um and you specify it with

as much or little detail as you want and then it what it says concretizes that specification into exactly the versions and exactly the options for each dependency in this tree that it goes down so if you're interested in our results you can try building these with we say in the article what version of spec we built with as well I won't make it up right now but uh you can go pull that exact version of spec generate these exact same packages and and test in exactly the same way that we did one thing that's worth noting as well while we're talking about containers being something

that's portable to different environments um we made sure that these MPI benchmarks had access to opx it's a little small there I'll see if I can there we go um so opx is the lib fabric provider that supports omnipath but we also built in support for verbs which is where you would get more General infiniband support as well so this same container should run with expected performance uh on either an omnipath or an infiniband cluster without modification which is pretty cool that's kind of the dream is that you could just push this up and then anyone could pull it down and run it and it

would execute this brings up an interesting question that I've had too and I don't I don't think that you've I don't think this is one of the sets of benchmarks um that you ran so I apologize to be just like throwing it out of left field but um one question that I've had is is there a performance impact for using uh lib fabric um so if you just remove the lip Fabric and you just um you know you compile your MPI directly against uh opx in this case you know that that's kind of um one of the that's one of the problems that people have

had in the past is that you know a containerized MPI application doesn't want to use the high-speed uh interconnect because it's got the wrong version of it or whatever um and that that's from that's you know that's the reason why Loop fabric help helps get you around that problem and makes your container portable and easier to install but does it come in any performance impact yeah so I a very long ago version of this article when it was still the the conclusion to the Warewulf article uh included that conversation because open MPI had support So before opx there was uh psm2 I think is what

is the right one there's too many acronyms but I think that's what it was there was a protocol for omnipath that predated opx or a lib fabric provider and uh open MPI supported both lib Fabric and also natively supported uh ps02 and so I I wanted to test uh open MPI in both of those cases but uh opx is only a lib fabric provider and in the meantime current versions of mvpitch have also started using lib fabric as well so I I decided that that that concern appears to have been obsoleted or at least become become anachronistic because all of the M provider or just

or all of the MPI implementations are just becoming lib fabric dependencies anyway and it looks like new fabric support isn't getting built into openmpi except for through lib fabric anyway or maybe something like ucx but I'm still uncertain in my understanding of ucx but it seems to be a really good things seems to be a really good thing um and it also suggests to me like if these these Engineers who are developing MPI and obviously are very very uh concerned about ringing every single little uh ounce of performance out of these systems if they are you know switching to default support for lib fabric with

their products that highly suggests that um there's little to no uh performance impact from using lip fabric yeah it at least seems to be what they expect the vast majority of people that aren't themselves developing MPI to be using um so the the next question that I had uh once we uh had pretty good results or great results uh in many cases um with different MPI implementations they all performed the same uh between a host versus container and then a container that didn't match the host um there's another disparity in that uh Apptainer has historically run with this suid the set uid helper that helped

it either do the builds or set up the container in the first place and so I wanted to know if we saw any performance I'll say problem any any reduction in performance if we were running apptainer and it's Now default username spaces mode so Dave talked about this a little bit in the past it used to be that you were mostly just a charroot and you didn't want to let that user be root inside of the container because that's technically also the same route that would have been uh inside of the or sorry outside of the container and there's there's security concerns there so so

recent kernel support that's been added I don't know I say recent um for all I know it's 10 years old um but it lets you have multiple uid namespaces where there are multiple routes and the root inside of your container is not the same route that's outside of the container uh Dave you had something about this I think uh no I'm sorry there's a there's oh there's a question that's come up sorry I got distracted by our thing here no no that's cool so um yeah so you've tried the PMI method I'm glad that you that you like it uh some HPC centers are recommending

using the bind method um and I'll talk about what that is in a second the HPC centers will create a singularity module by binding the host libraries required by MPI into the module file have you compared the performance between the buy method and PMI so the buying method so thanks a lot for that question so the buying method that we're talking about here is um let's we got to backtrack just a little bit to understand you know what all this is so uh we talked at the very beginning about the wire up issue and then like once wire up occurs how um the processes communicate

with with one another historically we you know we've done wire up using whatever the MPI implementation is that we have in the container and when you do that you've got to match the the MPI in the container to the MPI outside the container and that's the problem and that's what we're getting around with the PMI method um so historically there were two ways to do this right there was you either install MPI and inside your container completely and you just you're very careful to match up the MPI which is on the host system or there's this other method that seems like it would be a

lot simpler where you um you bind Mount the MPI implementation along with all the libraries into the Container at runtime and then that just brings MPI into your container you're going to match whatever's on the host because you're actually bringing it from the host into the container and then you know everything will work so I've uh you know I've used I've kind of like uh done some prototyping on that method and used it a little bit recently and I've I've you know included it in some blog posts and some videos and stuff recently and talked about it and there's two things number one you would

think that it would make your container portable if you would do that but unfortunately it really doesn't because what ends up happening is that you have to and this is probably what that module file in the question what we're talking about actually does you end up having to to run down this Rabbit Hole where you find every all the libraries that were that were um used to compile your MPI and you end up very quickly in lib64 and the system libraries and if you start binding mounting those in from the host to your container you know rut row your your glib C inside your container

better match the glibc on the host or else you're gonna you know you're gonna run into problems right off the bat so right off the bat that limit so what kind of container you can use and then secondly so here's this model um I've got a container that basically doesn't have anything in it it's just like an empty container um I have an MPI enabled application I have to compile that application on my host because I don't have them in the MPI I'm inside my container so I can't compile it inside the container so I compile this thing on the host and then I copy

that into my empty container right that just has an operating system and then at runtime I copy all of the MPI into the Container so I've copied the application and the NPI into the container what's the point of the container you know like you're using the container here um for for just to add complexity to your workflow it doesn't it doesn't give you anything and it adds a significant amount of complexity you know for no so yeah the whole bind model I mean I could be wrong in this I don't but I've personally I have not seen really any reason to use the vine model

it just doesn't it's like the worst of all worlds yeah it feels like it should be solving the problem but it's actually picking the exact opposite of the right answer for all of the possibilities and that was the worst of all worlds I love it um so the to answer the question the question was have we benchmarked it no um all of our tests with the bind model have been about understanding the complexity and the the cost benefit analysis in terms of uh of whether it actually simplifies your workflow and how hard it is to get to work um if you have a bind model

workflow that is working I would definitely expect it to behave the same as the host um because it's the same MPI as the host and then we've demonstrated that uh that Apptainer itself isn't introducing any real overhead and in fact there's a lot of the same system libraries but but you've tied yourself to the system you can't diverge the OS from the system in the way that we just demonstrated almost certainly you would not be able to run let's say a rocky 9.2 container on a um on a an 8.8 system uh within the bind model or maybe maybe it would be worse the other

way around but you run into glibc conflicts like that real quick in fact we've been noticing that with the Legacy fake root support inside of apptainer where if you didn't have username spaces it's trying to bring fake root in from the host but if that fake root is built with a different glib C then what's inside your container it doesn't work and so we're trying to improve that isolation uh and and demonstrate that that comes without any real performance costs and and the buying model I'm sure it performs well but doesn't give you any of the real containerization benefits um so we're here I thought

we wouldn't have much to talk about we're getting close to time so I'm going to summarize I always go so fast two questions and then we'll talk about them together so we have the the set uid version of Apptainer that's been kind of the way apptainer and Singularity before it worked for a long time where you have a set uid application that helps set up the container in the first place uh I wanted to see if there were any performance problems with um with using Apptainer's new username spaces functionality so we have the same set of tests with the same MPI implementations to ask

that question and then um specifically focusing in on username spaces uh with we discovered an issue with username spaces where more than one process is running on the same node and there's a lot of technicality about why that was happening let me get the graphs up now and we'll we'll just start talking about it so yeah so timeout too like real quick so we are getting up on time and um so we we hoped to be able to combine I think um you know the the the benchmarking that Jonathan did and also kind of a an issue with a a bug um that we have

found recently and kind of drilling down and figuring out exactly what that bug was uh Yoshi um uh yoshiaka yoshiaki Senda uh who you know as another one of our essays and is just like a complete rock star he also published today a blog post um going through all the details of this and I I'm kind of suggesting maybe we could just kind of preview it and maybe we could save that for an actual like one of these webinars in the future for sure I wasn't planning to get into the detail of it just mentioned that it is an issue that we found so let's

talk quickly you know it's very easy to just say and for this to be anticlimactic the performance was not different between suid and uh end user name spaces um we saw the same for mpitch and MVA pitch and open MPI and then uh um you know for all of the benchmarks that we ran if I think we did see another slight uh Improvement sometimes in the latency um with username spaces but you know sometimes not uh so again I would chalk these up to just run to run variants that would have uh smoothed out with running multiple times and averaging um or maybe being a

little bit more picky about cleaning up the environment between runs um this here actually you know what I'm saying that but now these results here appear to be different um so this was openmpi all to all I'll need to review this because maybe I don't know this is this is not negligible um which does not match my memory so I'll need to check the publication and see uh where this graph came from so yeah uh check in here again maybe we'll have a response and a pinned comment or something with the video um but the other was uh multi-node versus single node or sorry multiple

processes on the same note all these tests so far have been one MPI rank on one node and we had discovered a bug where if you had more than one MPI process on the same node sometimes you had problems with user namespaces and so um Yoshi had discovered uh more instances where this comes up I think especially with Intel's MPI but in my tests I only saw this happen with MVA pitch and so it's it's not as obvious to see but what you can see is if we got real zoomed in we'd see that down here at very small message sizes we have um we

have yellow results where we were running multiple processes on OneNote four tasks on one node uh but then at larger message sizes those results just disappear because the benchmarks hangs at a certain point and so a brief summary of what's going on here is that uh different MPI implementations will use different mechanisms for communicating between two processes on the same node rather than go out over the network and just come back in again it tries to use something even more efficient even more low latency to communicate between two processes on the same node and this MPI is trying to be even more intelligent about it

because it knows that one way of doing that is more performant at smaller message sizes and another way of doing it is more performant at larger message sizes so it switches over and that new message side or that new new mechanism that other mechanism breaks in username spaces so that's a bug that we discovered in apptainer that Yoshi discovered and Apptainer or actually I think someone else discovered it but he's been yeah there were several months start digging into it several members of the open source Community raised um raised GitHub issues about this and I'm sorry I don't have any other uh usernames at hand

so I can't like give them credit right off the bat um but yeah she has been digging into the root cause right that that's really what's been happening and um there are there are some proposed fixes that we have um some of which are workarounds and you can read all about them in uh in the article that he published today um and but then there's more permanent fixes that are being I'll say debated but discussed like what is the best way to solve this permanently Upstream so you don't need to work around so that is something that we're continuing to work on um I see

we have a question that this is the end of our results but we can sorry before we jump into the question I just want to give a shout out it looks like I think Martin Kuma uh was the one who um I think that's who M Kuma is probably Martin Kuma uh was the one who Yoshi is citing as uh giving the first um bug report of this great thanks Martin I think I think it's Martin M Kuma yeah thanks ah what if MPI processes spawn a new process inside the container so that's um you know it's easy for me to wave my hand and

say I can't imagine that being an issue but that was kind of what we were doing before we run tests that's not a specific um test case that I knowingly targeted I'd want to look in to see which uh if any of the micro benchmarks might Target that Dave do you have any thoughts on this yeah that's that's um complete scariness um so yeah so when that happens um that that's uh what ends up happening typically is that that process um so so usually that breaks and you know we're not really addressing that I don't think here um you know with with this at all

the reason that that breaks is when you have a process inside the container which calls MPI exec or you know some other uh thing to to spawn more MBI processes it when it does that it uses SSH and it escapes the container and so when the the SSH command lands on the other nodes and tries to start the process uh typically it can't find um the process that it's supposed to start there's been you know uh in the past um I've played around with stuff like this I've played around with one of the things I've done which I don't really suggest because it feels really

really gross and everything is to actually just wrap the SSH command inside the container with uh a different command that actually calls apptainer so you can go in there and hack around at your container and you can replace you know SSH or rsh or whatever the protocol is that it's using um with another command which will set environment variables on the other uh the other host and also call Apptainer on whatever command you give it um there might be so I'm not really sure I'm kind of like hand waving around here too you know one of the things that we've we've advocated and maybe we

should do a if we haven't already done so maybe we should do one of these webinars on is there's a way in which you can install applications uh so that you don't have to think about or know about or use containers container commands to run the applications on the host system and it could be the case that if you installed your programs in this way within the container and this involves creating a little wrapper script and some Sim links to the wrapper script and stuff like that but if you did that um it might not matter if you're calling your MPI process from within a

container because then um when you know the container when the containerized process the containerized MPI uses SSH or rsh or whatever to go out to the other nodes um that program is going to be exposed from within the container and it still might be able to run it even though it's in a container so I don't know that that's a whole other kind of can of worms that we're not really addressing uh here uh you know in this webinar yeah at the same time though I feel like in like I all of this comes down to I'm a systems guy I'm not an HPC applications

developer so my knowledge of actually writing an MPI application Falls over pretty quickly but it looks to me like MPI Comm spawn has got to do just like fork and exec level stuff from an existing MPI rank so as long as it's not actually needing to do processes initialization the way that it otherwise would have those should start up the way that you would expect them to inside the container and just behave the way that you would expect them to but again that that's hand wavy for me and I would need to I would want a benchmark it and find yeah maybe maybe with the

MPI com spawn you know maybe that's more of an internal what I'm thinking about what I'm thinking more function inside of the API yeah what I'm thinking more is um you know sometimes you actually have just programs that yeah uh you know the user interacts with the program and gives it some input or whatever and then it goes off and says oh okay based on that I'm gonna go spawn one we've worked with recently is called Orca and so Orca uh it it tries to be kind of like you know it's a it's a lot it's a good goal it tries to be a little

easier on the users and it says you want to use MPI even though you don't really know about MPI and so you're just going to tell me you're just going to give me some files that specify like how many ranks you want and stuff like that and then I'm going to take that and I'm going to launch an MPI job on your behalf and that's cool um but if you containerize that and you try to run it multi-node then you're going to have exactly that problem that I just you know talked about you're gonna um you're gonna escape the container uh if you're not careful

about how you install the container and maybe even if you are careful about how you install the container yeah Dave not giving a lot of confidence there um I think we are kind of like up on time is there anything that you wanted to just wrap up real quick the only thing there is one more question from David rush to uh how am I can you recreate the whole thing on on Raspberry Pi's so these uh specs should build on an arm processor so you should be able to do it with specs back or with the spec specs that are publishing um you of course

won't have a high performance network but if they're all underneath their net segment it should be just fine um yeah you would be able to run without any real issue uh you'll have you know there's a lot of dependencies here it's expecting that you have an existing slurm cluster uh which may or may not be easy to do in a Raspberry Pi environment or you can just run uh kind of the traditional way with with MPI run or MPI exec and not do the PMI stuff that we're doing here um but yeah it should be possible give it a shot Dave let us know how

that goes that's exactly what I was thinking okay let us know make a video I did also want to say just real quick because I I've you know I've uh dismissed the quality of our process here as mostly just a fun thing I was doing um but if you look at the article that we published today um there are I think four different links to relatively uh thorough academic Publications so there's one video for a conference presentation and three I think like actual papers um that are included in different conference proceedings and things like that um that found basically the same results that we did

in terms of oh yeah if you do it it's fine um what's interesting is all of them complain about um the need to keep MPI synchronized between uh your container and host environment which is the issue that we think has brought this back up for us anyway um because you don't have to do that anymore and so we think what we're doing here solves the problem that they were complaining about while demonstrating that we haven't introduced any new performance problem problems compared to what they were already seeing containerizing your application does not have a noticeable impact on your MPI performance yeah that's fantastic thanks for

going on the rabbit hole it was great I know it just kept spawning more more questions and testimonies perhaps we appreciate it it's so fun though that's awesome I love it when that happens yeah awesome well thanks guys yeah yeah this has been a lot of fun it's always a lot of fun and if you guys um watching if you have some kind of topics I mean any topics at all related to ciq but specifically around apptainer and containerizing anything just leave us a comment below or you can reach out to us at ciq.com as well and we would be more than happy because we're

doing this kind of stuff every week bringing you guys more knowledge and information just kind of getting down into the what we call the weeds which is like the fun place for you guys so it's probably your garden but it's like our weed land so thank you so much for sharing that with us thank you for being here make sure that you like and that you comment and that you subscribe and we will be back here same time same place next week have a great day great thanks