GPU Computing with Apptainer!

Start with you Ryan. Uh so from what I've seen I I know GPUs are used heavily in AI and ML. Uh from my experience though uh with uh in the automotive and aerospace industries um I have seen GPUs used to accelerate simulations. Uh but this is very dependent on the specific algorithms that are used uh because certain algorithms work well on GPUs while others don't. Uh so yeah that's been my experience. Thank you Brian Dave. Yeah I think especially the whole AI kind of deep learning um you know thing revolution really that we've seen happening since maybe 2015 2016 uh is really what has kind of you know launched GPUs into the forefront of HPC.

um you know I like I was kind of a spectator to this and at the time that this really started I was um I wasn't in computer science I was in neuroscience so I can kind of only speculate but it's it sort of seemed to me that people who were developing GPUs um so I mean so the GPU architecture is like fundamentally different from like the normal CPU architecture right GPU architecture is like massively parallel lots and lots of relatively simple computations all happening at the same time across like tons and tons of processors, right? This is a very naive understanding, you know, based on like just what I think.

But I I I think that what was happening was the GPU um you know people who are developing GPUs were were using them for um graphics for like you know massively computational things like ray tracing and stuff where you need to just calculate a whole bunch of you know relatively simple things all simultaneously and somebody you know had the bright idea and said well I mean there's lots of different problems that don't include graphics that we we could use the these this for and you know I'm oversimplifying and once again presenting my kind of outsers's view but like once once people kind of came up

with that and then also I think at the same time or in a similar time these deep learning models started to come out and these deep learning models really um you know could benefit from lots and lots of uh pretty small computations all happening in parallel because the way in which the the models were put together right it's just it's it's a bunch of linear algebra it's a it's it's you know these matrix um these matrix uh calculations which are happening and so you know everything kind of worked together and just sort of uh launched GPU processing um really into the HPC mainstream. So from your research perspective Dave before you got into computer science were you using GPUs?

Is that something that came before after during? Um yeah that's a good question. So I I remember like way back like 2012 2013 um you know I was actually not using high performance computing at that point in time but I was in a lab in which a bunch of my labmates were um we were doing a lot of simulations and at that point in time I I remember the the that that was at Vanderbilt University. I remember um Vanderbilt started to to invest in GPU technology at their HPC center and some of my lab mates were like, "Yeah, it's kind of weird you can do these operations on graphics processing units instead of CPUs, but um you know, they're they're really investing heavily in this.

It's like supposed to be kind of a you know, something that's happening in the future." Um and so then uh around uh guess around 200 I guess around 2013 actually. So I I might have got that timeline missed mixed up. That was probably 2010 2011. Around 2013 is I think when I went to the NIH uh as a scientist and shortly thereafter is when you know uh deep learning really started to uh make a big impact and you know some of some of the models that people were putting together began to solve problems that have been huge problems in computer sciences for many decades.

Um and so at that point in time uh the perspective that I had on it so I was um investigating uh the neural basis of vision and you know which wow like a lot of a lot of AI research is is really you know focused on vision and so at that point in time from my perspective one of the big um one of the big things that a lot of my colleagues were wondering about and you know talking about and writing about and arguing about is to what extent do these deep learning models do you know do they approximate in any actual like so so

you call them neural networks right and there's there's a there's a um there's a metaphor of them you know being neurons but are they neurons and you know I mean do they operate in the same way that real biological neurons do and you know to what extent do they approximate the actual real workings of the brain and like there were people there's people probably still on, you know, the side like all the way at one end of the spectrum just like these things are completely different from the way that the the brain behaves. There's no way we can, you know, we can use these as research tools to try to understand the brain.

And there's people all the way at the other end of the spectrum that are like, you know, basically we should build these models of vision and we should study them and we should you, you know, extrapolate from them how the brain works. And then there's people all along the spectrum right in the middle. So, uh, it's a pretty interesting, you know, topic from that point of view, too, which is not really, you know, something you hear too much of from computer science, I don't think. That's great. Thanks, Dave. Now, from a background perspective, I guess the next question is kind of how does the GPU driver work?

I guess from from my perspective uh my knowledge of how this works is this the GPU driver is just the interface for the OS to actually uh communicate with the GPU hardware. But uh under the hood though I'm not too familiar with how that works. But uh if Dave would like to shed some more light on that, I'm I'm all ears. So I can give you Yeah. So once again, I'm going to give you just my perspective, which I, you know, I feel like there might be like actual engineers who uh, you know, really work on this stuff who who are going to watch this later and they're going to be like, "Wow, what is this guy talking about?

He's got no idea." But I'm going to give you, you know, what I know. And um, what I know is that uh, the the G the Nvidia GPU driver, which you know, I should specify, we're going to be kind of sticking to Nvidia today. Um, and that's, you know, in in no way an endorsement of NVIDIA or anything like that, but it's just like, you know, those are the w most widespread GPUs that are currently installed, uh, and most of the clusters that I've worked with and most of the most of the nodes I've worked with and stuff. So, I don't have a lot of experience out outside of um, Nvidia GPUs.

Uh, so that's what I'm going to be kind of sticking to and talking about today. But um from my perspective, the Nvidia uh driver for GPUs when you install it, it installs basically two major components. And the first of those two major components is a kernel module which it you know compiles and then adds to your kernel. Um and then the second major component is a collection of libraries that get installed kind of in the user space that are used to interact with that kernel module. And and the reason that that's important is because that that that makes it kind of difficult as from a um from a container perspective to pass the GPU with its driver through into the container and to use it.

So that's that's basically you know um at a at a high level. So from Dave, neuromorphic computing is definitely still a thing, yet abstract models of the mind still beats the uh beats to them to date. Thanks Dave. Thanks for joining. Yeah, good to see you. All right, so Mr. Glove, give us a little bit of a history of GPU support and I'm we're specifically talking about Appainer Singularity is kind of what we're focused on, right? Yeah. So, um, so yeah, this is this is kind of a cool cool topic. I, you know, I think and it's it's personal to me. Um, so if we if we take, you know, if we go if we go all the way back and we take a trip down memory lane, um, around 2016 or so, I believe it was 200, actually 2017, I believe.

Uh, no, no, it's 2016, that's when, um, Greg, you know, first kind of committed the first V1.0 O of singularity to GitHub and um there was a community kind of coalescing around singularity and there was a lot of excitement within the community. It was you know really new technology and it was a it was really kind of nice inclusive thing and one of the things that um we really wanted to do and uh you know was really important to us was to be able to utilize GPUs from within the containers. Um but I I kind of I I I told you a little bit about how the GPU container the GPU um drivers installed to kind of set up this problem.

So one of the or you know basically one of the key differences or maybe the key difference between containers and virtual machines is and one of the things that makes containers so so fast is that when you use a container you're sharing the kernel with the host system underneath. And that's kind of like a key thing to understand understand why um why uh the driver becomes such a you know kind of prickly you know thing to to use inside the container. So I already said that the GPU as far as like Nvidia's driver goes um that driver installs um a kernel module and then you also have to install a set of libraries to interact with that kernel module.

So what ends up happening is that when you run a container in which you want to use the GPU, well when you run that container, you get the kernel along with its modules uh in your container. And so you've got this specific version of the the driver, right? And so then but then you know you don't you don't like get the libraries uh installed. You just you just get the kernel module. And so then it's up to you to try to match exactly the same version of all the libraries with the kernel module. And if you don't match them exactly, you're just going to get errors where it says that it just can't locate the GPU or whatever.

So it becomes kind of an issue. Um so you know we had a lot of users who were interested in using uh the GPUs at the NIH um on the on the cluster you know and were also interested in in singularity containers at the time which you know ultimately became appainer containers. Uh so we were trying to you know figure out this problem. How do we uh how do we like instruct the users and and and you know figure out a way for them to be able to install uh the right versions of the libraries and their containers to use our GPUs. And the very first thing that we did and I was I was helped with a few other members of the community in doing this but the very first thing I did was really a a pretty stupid thing.

We created this script and the script just it was just a helper script and it and and basically it's just a it's just a shell script and we said um we're going to provide this for you uh you know up on you know our storage space at the NIH. You can download this script as you're building your container. Um and it's still it's still actually up on GitHub. It's called GPU for singularity. And um what it does is you just you basically just give it as input a uh version of the of the uh GPU that you're interested in. And to try to make sure that we really match these up, I actually grabbed so the um GPU drivers from Nvidia are redistributable uh according to their license, at least the Linux versions are.

So, I actually just grabbed like the one that we were currently using and I put that on the FTP server uh at the NIH or one of the FTP servers. And so then um you could use either that one or in the script you could download it directly from Nvidia depending on the options that you passed so that people outside the NIH could also get benefit from this. So this helped a little bit. I mean, it's a little bit better than having to go figure out what the current version of the driver is and then go manually download it from Nvidia or download it in your container and then install it.

But the problem is um okay, that container now works, you know, at the NIH with the current version of the driver that we're running. Sometimes we had to go and update the drivers, right? Then everybody's containers break or sometimes these people might want to take these containers and run them somewhere else. So I tried to make those scripts so that you could update your containers too. So that you could you know without rebuilding your containers you could just like um you know pretty easily just like uh rerun that in the same container and then basically rebuild that into another at the time I think they were ext3 uh images instead of the the uh file format but it's still super clunky and it you know it just didn't work very well at all.

And so that was the beginning. And then um so I started to say okay well I mean I can see what gets installed inside the container when you do this. I can see what we need in order to make the container work. So um why don't I start going through and trying to find you know manually all these libraries where they exist on the host and maybe I can just start to bind mount these libraries you know because I have a list here. So maybe I can just go through and start to bind mount manually all these libraries into the host. Um and so that took a while and there was, you know, some trial and error involved in doing that.

Uh but I finally found a pretty reliable method to be able to bind all the libraries from the host system into the container. And so this is better, right? Because now you don't have if you can just find them and bind them in. you don't have to rebuild your container every time the, you know, the administrator comes by and updates the uh the um driver. And you also don't have to rebuild your container if you want to move your container from um you know, from from one system to another. You just you just plop the container on that system and then you bind mount all the libraries in.

But but you know, manually going through and finding all those libraries and figuring out how to buy mount them in, it's it's a little tricky. It's not super easy and it's it's very it's still pretty clunky. So, I got really excited about it though. I I went and I you know I pinged Greg so we're talking in Slack and I'm like hey you know I mean I've been working on this problem and um you know we can we can bind mount all these libraries in from the underlying host into the container. And um this kind of shows you how you know I'm I'm not a computer scientist by background.

I've just kind of like picked stuff up along the way. And he was kind of like, that is crazy and it's probably never going to work. You know, if you got it to work, then you're lucky. Maybe your um maybe your container was the same, you know, operating system as the underlying host or something. But like I mean libraries are com they're going to be compiled for you know specific operating systems and spec like this is not going to work um in a in a very you know general flexible kind of way and I was like well I think it does I mean it seems

like it does and so we started talking to some folks at Nvidia and at the time so this is a big problem in containers generally right this was not just like an issue that uh we were having with Appainer um people couldn't really do this with Docker either which was the other prevalent solution at the time. Um the problem was so bad that Nvidia had created a a different version of Docker called Nvidia Docker in which they had you know figured out how to how to make container how how to make GPUs work within containers and that was their advice at the time is that you were supposed to grab a different version of Docker.

You were supposed to grab Nvidia Docker and use that if you wanted to use GPUs inside your containers. So we started talking to some developers in Nvidia and it turned out that that's basically what Nvidia Docker was doing. It was finding the libraries on the host system and it just it it wasn't buy mounting them. It was kind of writing them into the container but it was basically doing the same thing. And it turns out that this is it's kind of lucky because it's the way in which um Nvidia compiles and then distributes their their libraries.

they basically just compile them in a very very general way that they're they're expected to work on a wide wide range of different systems and so because of that you can you can just grab them from one you know operating system and put them into another one and they'll go ahead and work. Um, and so ultimately we ended up, you know, even though it's it's kind of gross and it's weird that it works and it's surprising and all that, we ended up, you know, creating the experimental, which has been experimental for like the past, you know, six years, the experimental-v option. Um, which that's how it works.

it uh it goes and it finds the libraries on the host system, binds mount them, bind mounts them into your container uh in a in a specific library and then it just sets up your LD library path with all those libraries, you know, on the path and away you go. So Dave, it whenever you say it finds the libraries, I'm assuming it's different per DRO. So if you're on a Debian based system like Iuntu, it's probably putting or it could put libraries in a little bit of a different path than what it would on Enterprise Linux. So from a developer standpoint, are you having to tell it all of those versions of where to go look?

Because I mean Auntu 18 may be different than 20 different than 22 or does it just go look? So this is this is really it's dead bang simple. Um, so when you install, uh, you know what? Maybe I should go ahead and and show you. Maybe the best thing to do is to show you. All right, let me, um, let me see if I can figure out how to present. Uh, I've never had any trouble with presenting in the past, right? So, no, I mean, we've never actually had to watch your physical monitor anything. Not today, right? Oh, look at that. Wow, look at that. Okay, cool.

Magic. All right. Yeah. So, I I came I came prepared to kind of show how this works. I've got this example here. I've got this example here. Um I've got a little uh TensorFlow container that I could show. Um but it it doesn't really matter what the container is. I mean, this one has CUDA and you know, it's got a deep learning framework and stuff installed in it. So, I could actually do real GPU work with it. But, um the same you know, the same basic principle applies uh anywhere. So you asked Zayn um how how does Appainer figure out you know depending on where you know like regardless of what your distribution is how does it figure out uh you know how what what libraries to get and where to get them and stuff.

Um, and it turns out that when you install, and I'm I've got this in, this is really gross, too, but I've got this installed in opt. Uh, when you install Appcainer, there's a configuration file within Etsy underneath of the it's it's not, you know, etsy. Well, it depends on how you install it, but there's this configuration file called envylist.com. Okay. And the reason that we did it this way is because so when you install Appainer, it installs for you this configuration file. And this configuration file has a bunch bunch of libraries um that are a pretty good first guess at what you're going to need in order to you know run CUDA or run a graphical um you know some sort of a 3D uh you know graphical representation or something like that from within your container.

Um, and we did it like this specifically. And so, so it also I should highlight that there are some libraries that it's going to find and then it's going to try to find also some some binaries on your host. Uh, you know, based on the fact that this is the way Nvidia drivers are usually installed and it's going to try to bmount those into your container also. Now um the reason we did it this way is because we didn't want to to provide like an authoritative uh list in the code of this is all the libraries that you absolutely have to find. We wanted this to be user configurable because you know what you might have some weird system you might have some you know weird requirements.

You might be doing something that we haven't anticipated and tested. So we want you to be able to figure out what this list should be on your own. And so this list is editable by the uh administrator. Um I think there's even ways, it's been a while since I played with it, but there there's probably even ways for uh a user to edit the list themselves and change change the list. Um but in any case, that that list usually provides a pretty good starting point for, you know, being able to utilize the GPU inside the container, it turns out.

So um and then so what happens is uh this is it's actually pattern matching and so what it does is uh obtainer looks in the LD cache which is where you know all the libraries uh for the entire system are are you know there there's a cache basically saying here are all the libraries um to look for when you need to link stuff. um and it looks in that cache and it finds not only like libcuda.so but libcuda.so.1 for instance or libcuda.so you know whatever it is it pattern matches and it finds all those libraries on the system and then it bind mounts them into one particular place within the container.

So if I do an app tainer shell d-nv and then I do that tf. then it basically just says that it had to create a bunch of uh a bunch of different bind mounts to get that Nvidia.smi in there. But now if I did uh Nvidia SMI, oops, you can see the GPU from within the container. Sorry, it's really big and gross there. Um but the way that it actually works so if you go to the top level directory um of appainer there's a do there's a hidden directory calleds singularity.d and that's kind of like the guts of the container that's where all the metadata and stuff go.

There's a subdirectory there called libs and if you look in libs oh you use a command that works to look in libs shortcuts man no shortcuts. [Laughter] Wow, look at this. This is all the same um libraries or many of the same libraries that I just showed you out of that configuration file. Um it's gone and it's found these libraries and it just jammed them all in flatly into this one directory.

And then within the container, if I do like uh echo LD library path on the end of that, I can see singularity.libs lips which this is set by default with appainer that LD library path I believe is set no matter um whether you use the d-nv option or not uh but in this case there's some other stuff so this container I should say too was grabbed directly from Nvidia's um container registry the NV uh or the NGC um and so they've prepended some other stuff that you know is going to be important for this container to work properly but you know this is this in a nutshell is how works.

And you know, to your point, Zayn, you could go to your host could be an Inbuntu host. It could be um you know, Debian, uh CentOS, it doesn't really matter, you know. Um this is pretty much the way that Nvidia installs the container on every different, you know, distribution that it that it supports. And so, uh you know, this pretty much works no matter where you go. So you said that that you could add things to this file or there's the capability of adding things to that file. So does it read every time you execute a container or something does it install? No, it reads every time you execute this container.

So, so uh yeah, so you you could and it sometimes is the case that you find okay so so this list is not perfect and it's sometimes the case that um you know an administrator or something finds wait a minute there's something in here which uh conflicts with my user's ability to do X Y and Z and it's not working properly and so you can just comment stuff out or you can add different things um it for a This was like a undocumented and kind of unintended feature for a while. Um, this actually I believe uh with earlier versions of Appainer or actually earlier versions of Singularity, it would create an environment variable in on the host to set the bind path and then it would use that when it ran.

And so you could actually create that environment variable yourself and you could use that to um copy libraries into that that slash.sularity.dlibs directory. And people we found out were using that because that was like an undocumented it was kind of a side effect and uh it wasn't really expected that users were going to use that environment variable and then they were and I think we tried to change it and then users were kind of like wait a minute stuff doesn't work anymore. We were using that. Nice. And then just to kind of show I mean I could show really quick. Uh so if I did like uh I show you that this does actually work I've got a Jupyter notebook uh running in this little or that I you know installed in this um container.

So I can do something like this and start up a Jupyter notebook. And if I go over here, this is just a basic um GPU demo. So I'm going to watch the GPU over here. Notice I have XOR running. Um this is why I can share right now, by the way. So last time I tried to share, I my computer updated and it swapped me over to Wayland and Wayland's unhappy with uh uh with trying to share through Streamyard. So, you fixed it. Yep. All right. So, I'm going to go ahead and connect off screen to this. All right. So, I've connected to this Jupiter uh this Jupyter notebook I've got running that I started with this command.

Um I've got this uh mnest example. So, mnest is a bunch of handwritten d um handwritten digits. It's the modified uh National Institute of Science and Tech Tech Technology handwritten digit data set and it's it's used really widely uh for stuff like this just silly little examples. You can see I've run it once before and it gives me a bunch of warnings. Uh but anyhow, I'm going to go ahead and run it again. It's going to give me hopefully if it works some warnings. Uh, are you not running? Oops. Oh, it started to run and then I stopped it. All right, but in any case, um, you know, so I see Python installed here or Python, there's a Python process running on my GPU here and, uh, it's training.

So, basically all this is doing, this is a very, very simple example and I hit it twice and that's why it's not giving me output here. Um yeah, it's giving me a bunch of warnings back there. But in any case, you can see it's running here. What Oh, here we go. So, what happened was um this is importing TensorFlow. It's downloading this uh this Mnest data set or you know loading it. Um it's creating a very very simple model. Um it's compiling the model a certain way and then it's fitting the model uh to the data. And so I did 10 iterations here. that's more than enough to get this model to, you know, recognize these handwritten data sets.

And so now my accuracy is like 98%. So I'm only wrong 2% of the time. So that's a really really simple little example. Thank you for sharing that, Dave. I we've talked about it. I know Mr. Deon has put in here a couple different times talking about compiling or building CUDA kernels. I think that was something else we were going to talk about is CUDA. Yeah. I just wanted to kind of like um Yeah. Okay. Leverage GP off to make some more available. Yeah. So I wanted to talk about uh CUDA a little bit because this is something too that I find that there's a lot of confusion on and it sometimes um sometimes causes errors and issues and problems.

Uh and so I wanted to just kind of you know talk talk all this all the way through. So CUDA um sometimes people refer to CUDA as as like the the CUDA driver or the you know the the they they conflate um the CUDA libraries with the NVIDIA driver and I think that this is this is it's not this is something which um could be made a little bit more explicit by Nvidia truthfully. So um so CUDA is different from the driver right CUDA is a collection of libraries that you can use in order to do parallel computing on GPUs. Um cool glad glad this is helping Josh.

Um, so, uh, so you can you can install the library or I'm I'm sorry, you can install the driver and and you don't have to install CUDA. And you could in theory kind of install CUDA as well, but you're not going to get very far with it without the driver. The two are separate. The the um the problem is that it's it's kind of confusing that the two of those are like uh conflated together because Nvidia, for instance, packages. So, one of the ways that you can get CUDA is by downloading a self- extracting binary uh a run file that um that Nvidia provides for you.

And that's a pretty standard way to like get and install CUDA. And when you do that, if you actually extract the binary and you look at it, you'll see that it extracts a bunch of libraries and stuff and extracts, you know, like an installer and everything, but it also in extracts another self-extracting binary, another run file which is packaged with it, which is the drivers. And so all everything's packaged up together. And if you just go ahead and just run that run file and install it, it's going to try to install a new driver on your host system as well.

This makes sense um if you're installing CUDA on bare metal on a on a you know on a node because probably you're installing like the latest version of CUDA and if you're doing that whatever driver you've got is probably not going to be up to speed to be able to run the latest version of CUDA. So you probably have to update your driver too. So that makes sense. But when you grab that run file and you try to install CUDA inside your container, you don't want all those driver libraries inside your container. they're probably not going to match whatever host system you're trying to run. And if if if they actually get seen, if they, you know, if if they actually don't get masked by the LD library path, they're basically going to break your container and make sure that it's it's not able to run on the GPU.

So, this is where it becomes really um the waters are muddied and it becomes problematic for people, right? you. It's really important to recognize that there's a difference between the driver and CUDA. And it's really important to recognize that the driver is on your host and that's where it comes from and that's what it's tied to. And that CUDA is in your container and that's what it's tied to. And that's because you know if you're running you know TensorFlow or uh you know PyTorch or you whatever you're running some CUDA enabled um program it's been compiled with a specific version of the CUDA libraries and so you need to make you can't like you know do the same trick that we did with the driver and buy mount CUDA from the host system in the container at runtime.

That's probably not going to work. You need to have CUDA in your container and you need to have the driver on your host and the two need to be be separate. Are they is one dependent on the other from a versioning standpoint? So if you had CUDA in your container, would it could you possibly break it on a host? You could um but it's it's a little bit more forgiving. So when it comes to the the drivers, remember you've got a um kernel module and you got a bunch of libraries which interact with that kernel module and those have to be exact and that's why we get them from the host and put them into the container at runtime.

With CUDA, it's a little bit more so you basically just need a minimum version of some driver to run CUDA. The drivers are backward compatible to run older versions of CUDA. So you could install like the the very very latest CUDA into your container and then try to use that with a driver which is older and it would break. But as long as you've got a relatively up-to-date driver on your host system and as long as you know your CUDA is not like bleeding edge within your container, you're you're going to be okay. You should be able to run it. Thank you.

There's also so so um before we get off that topic there's one other thing uh I showed you actually in that configuration file a minute ago like the very first library that I showed you that I was bm mounting from the host into the container was called lib cuda.so So, and that's not CUDA. That is not CUDA. That is that is the library which is part of the Nvidia driver which allows the CUDA libraries to talk to the NVIDIA driver. It's actually part of the driver. It's not the CUDA libraries. This is this is why I'm saying that um like I I think that uh Nvidia could could make this a little clearer.

Um and it's kind of complicated, you know, keeping the two separate. Um, but it it's it's confusing. All right, Mr. Deon had another statement. Uh, shared objects quite Yeah, here we go. So, shared object and a large HPC system with a distributed file system have historically caused IO storms, bottlenecks. Is there a way to cache these SOS libraries to avoid dynamic loading to the node? It's an interesting question. Well, um, so it it kind of depends. So, um, so in this context, um, you're you're probably not going to have your uh your drivers. In fact, I I I think that you are, you know, it would be it would be really weird, I think, if you had the uh the Nvidia driver libraries installed on shared storage.

Um it's it's it's very you know that'd be a very atypical although I don't know it might work I guess but it would be a very atypical kind of uh installation normally when you're when you're installing drivers I mean you go through node by node and you install the drivers on the nodes and that's stuff that's like on the nodes themselves. Um yeah I don't know. Um so I'm I'm you know what do you think Brian? You've got a little bit more HPC. I actually agree with you. I I've never seen these uh typically installed on on a shared file system, but if it if it were me, I think I would this should just go on the node itself just for just to avoid that specific bottleneck.

Uh yeah, I mean you could get into a situation if you installed um so once again the you know part of the driver is a kernel module and that you know that's not going to be installed on a shared file system. So, you could get into a situation while you're in the middle of an update. And in fact, you know, I' I've been in this situation at the uh NIH before where you're in the middle of an uh an update and you're kind of doing a rolling update. And so, some of your compute nodes have one version of the driver and others of your compute nodes have another version of the driver as you're going through and trying to update them without, you know, taking them all down at the same time and messing your users up.

um if you had the driver libraries installed on the shared files system um it would break on whatever nodes those drivers those libraries didn't match the the uh the kernel. So you you'd want to avoid that. Now, as far as like the CUDA libraries go, those are going to be installed within your CI within your uh within your container. And so, even if that's sitting on shared storage, the way that in which the image is mounted um onto your your file system for your node is going to avoid like all the little metadata lookups which can cause those those IO storms that you're talking about. So, so um just the fact that you're using containers and that specifically that you're using uh appainer containers in which the the container is a single image uh that's going to actually help with that problem.

that's going to actually, you know, improve that problem and make sure that you avoid because you could you you could see a situation, in fact, it's a normal situation to have CUDA installed on a um on a shared file system, right? So like you could um you could have CUDA installed just like any other application on a shared file system and have that controlled with your module so that you're you're loading you you have your users load whatever um version of CUDA they need for their particular application to run and so that yeah that you could you could totally have um you know IO bottlenecks but yeah the the using the containerized approach is going to help with that.

Thank you for the question, Dave. I think Greg was agreeing that he liked the question as well. I was kind of hoping it was going to be one of those you ask him if he can do it, he tells you not a good idea, and then you go do it and prove that it was there. Well, I I mean, I wasn't I was I wasn't proving anything. I was just sort of stumbling around in the dark. And it turned out that it was a a good good direction to go in. I like it. Very good. All right, guys. If we have any questions, now is the time because I think we're kind of the end of what we wanted to talk about.

Um, there's some other comments in here about libraries and um I see there's anything else. Give it a minute. Well, um, so there are some other, you know, there are some other things that we could talk about. Um so as I said uh this you know the the way in which the D-NV option works and by mounts these libraries in is kind of dead bang simple and even a little stupid and there are ways and there are times in which it'll break um you know when you've got uh you know kind of an unusual workflow or something. So I' I've seen that and that like issues are raised, you know, once every couple months or something.

We've got an issue with somebody who's doing something unusual um and they'll raise an issue in GitHub and say, you know, this is a problem and and a lot of times the solution to that is to edit that file, edit the nv um lib list.com file. So there is another way. So Nvidia um actually I probably uh around the time or shortly after we began working on the D-NV option they began to develop um a a product uh it's called lib NVIDIA-container on GitHub but it's also alternately kind of referred to as the NVIDIA container CLI or NVC NVC CLI.

Um, but what it does is it's a it's a different program that you can install in a standalone way on your system and you can query it to do things like figure out what you know libraries are on the uh on the host system and not only figure out what libraries are there but figure out what libraries for a specific application like I want to use visualization for instance.

So what libraries exist on my host system that are important for me to be able to use you know 3D visualization and so because that exists um you can actually if you have that installed there's another command line argument that you can pass to obtainer which is called d-nvc cli um it's kind of a mouthful but if you pass that instead of just you know going and kind of looking at this dumb configuration file which has a list of libraries ies which are probably right for most applications. It'll go and query um Nvidia's third party tool and say tell me what do I need to have uh passed into this container.

Now it works a little bit differently. Um it doesn't want to just bind mount um these libraries into your container. Instead it wants to like actually write them into your container. So typically uh if you don't specify anything, what that'll do is it'll set up a writable temp file system in memory and it'll overlay that memory onto your container as though it were like a you know like an like an overlay in Appainer and then it'll write to that. But that's um ephemeral. So it'll go away when the container gets done you know is is um done running.

So there's that consideration and the other consideration is the places that it's going to write to are places where a normal user doesn't have access to write and so you need to have fake fake root access enabled and you have you you need to run it as fake root typically for this to work properly. So there's a little there's a few little gotchas and little cav caveats, but there are circumstances in which uh you know that option is going to be a better fit for what you're trying to do and it's going to be a little bit more robust than just relying on the D-NV option.

Thank you, Dave. So I do have one more question from from Greg. Greg wants Dave to remind him who won the basketball shootout at the arcade. You're key. I'm sorry, Kenneth. Yeah. Um, that would be Kenneth every time. Like, he is absolutely awesome at all things, I think, video game or carnival related. Uh, he's a he's a pretty amazing guy. So, we're taking this trip down memory lane. Thanks for the question, Greg. We appreciate it. You guys don't have any more questions? Go ahead, Dave. Go ahead. Yeah. So I I can just you know me man I can just keep talking forever. Um absolutely keep talking.

So the reason uh actually so the reason that we got back into this recently and I started going through this and looking at all this um this was prompted because uh we started having some trouble with some cloud uh some cloud instances recently which prompted me to to you know um kind of go through my memories and dig up all the information that I had and revisit everything um everything GPU driver related. So, and I'm writing a blog post about this now, which hopefully will be, you know, published pretty soon because uh, you know, I think it's kind of interesting and it might help some people actually.

So, we use uh a cloud provider here at CIQ pretty extensively called Vulture uh just for like single spot instances and testing things and and you know, doing stuff like that. And so Vulture uh has some GPU instances like many other cloud providers um that you can spin up. And so I was we you know me and several of the other um the other uh solutions architects I think Brian included have have been spinning up some GPU enabled um Vulture nodes and then trying to run uh GPU enabled workflows on them. And we we were having this weird kind of like problem where we would try to run the workflow.

We if you just did like Nvidia SMI and you just looked at the graphics card from within the container, you could see the graphics card and you could see, you know, what was what was or was not running on it or whatever. So that was working, but then when you tried to run the workflow, it would just complain that it couldn't find any CUDA device. And it was like, well, the CUDA device is right there. You can see it with Nvidia SMI. Why why are you having this trouble? And so um I struggled with this for a little while.

I think everybody kind of you know struggled with it and then you know just kind of did some workarounds and and you know kind of went and did other work for a while and I struggled with it for a little while because I was trying to put a demo together that use the GPU on a cloud instance and um I finally said I don't know what's going on here. I'm going to just use my local machine. I'm just going to use my local laptop to do this demo which has um a GPU in it although at the time I didn't have a driver installed for it.

So I went and I installed the driver on my laptop. I re rebooted my laptop and then I went and tried to run the workflow and I had the same error. It couldn't find you know so this was not because of the cloud. And I started to think and I started to say wow five or six years ago I think I had a similar error at the NIH. And it turns out, this is this is complicated, but it's it's kind of neat. So, it turns out that when you reboot a node, there are some device files in /dev that are needed in order to run the GPU, and those are not created by default.

So, when you reboot the node, they're not all guaranteed to be there. What happens is when you run a program which is compiled using CUDA or when you start the Xserver or you do anything else which is going to you know ping the graphics card and start using it. It calls um a program called Nvidia mod probe which is installed as part of the driver which goes and looks in /dev and says uh do I have all the device files I need and if it doesn't it creates them. Here's the kicker. A normal user can't create files in SLDEV, right? So this is an SUID bit program.

It escalates your privileges behind the scenes and it it allows you to run a specific operation as root. That one operation being let me go create the right device files that I need to talk to the GPU card. Okay. Well, that works on bare metal just fine, but it doesn't work in an appainer container. You can't escalate your privileges like that and use an SUID bit program within an appainer container. So the upshot is if you restart a node and then the first thing you the first uh like CUDA enabled workflow you try to run is containerized, it's just going to say I can't find the CUDA device.

If you restart a node and you run um a CUDA enabled workflow on bare metal, it'll create the device for you and then subsequently you can run it within an appainer container. So this is a really weird hard to diagnose kind of strange problem. Um, and I remembered, man, I I we had this problem a long time ago, uh, at at the NIH. And the solution ultimately when we figured it out was to just create a little startup script that goes through and creates the proper devices and, you know, make sure that that runs on the GPU nodes at startup.

And so once I figured that out, I Googled around a little bit and I found that Nvidia actually provides for you a startup script in for this particular weird situation uh in which you know you don't have Nvidia mod probe installed as suid or for whatever reason you can't use it like we can't use it within an appainer container and so I just grabbed that start script threw that onto vulture so that it runs when the nodes you know boot up and then it all the device files are created and everything works for you.

So that was why so I you know I started looking at this and started talking to the other solutions architects and then it came up well why don't we do a why don't we just do a um a webinar where we just kind of talk about uh GPU stuff with Aptainer it's fantastic and I'm also really impressed that you remembered an error you got five years ago but it took me days it took me days to remember that hey you got there impressive Impressive. So very good. Well, now we are actually up on time. So you don't have to we no more time to tell stories, Dave.

I'm sorry. I appreciate it. I appreciate the topic. Very very interesting and certainly something that a lot of people are interested in obviously. So we really appreciate it. Uh Brian, thank you for joining us as well. Guys, go like and subscribe and join us next week for another uh round table I believe. So, thank you and see you next week. Maybe it's like we have to have more stories. So, talk let Brian tell a story this time till we get cut off. I don't have any good stories.