David Godlove Demos Apptainer at Stanford High Performance Computing Center

about Apptainer and you might not have heard of Apptainer or if you have you might have also heard of Singularity and you might know you might not know like what Singularity what's Apptainer basically Singularity is what the project used to be called a few years ago we asked the Linux Foundation to sponsor The Singularity project and he said we would love to do that but we want you to change the name and so we changed the name to Apptainer and so now uh what was Singularity is now Apptainer it's pretty much that simple okay so let's talk about downloading and interacting with containers um so

I'm going to try to follow pretty closely with this material so that you guys don't get lost and we don't get into different things but you know we can kind of jump off and start talking about uh kernel builds a little bit Okay so the very first thing you need to know is that you can always if you have Apptainer installed just run apptainer help and that'll give you a list of all the commands that you can use it'll also give you some options and arguments that you can provide and you know some information about apptainer um but really the meat is these different commands

so I want to pull a container I don't have to pull a container to use it but I want to show you what pulling a container looks like that's the first thing I want to do so I'm going to go ahead and do an apptainer pull I actually want to just do help because if you're just learning containers this is a great way to learn about these sub commands too get an idea of what they do and you know how to use Apptainer okay so if you look at the help it basically says that pull is going to download or build a container and you

you give it a URI of one of these forms and you can also name the container if you want to if you want to give it two inputs so I'm pretty fond of using kind of a silly container which doesn't really do much other than try to entertain you a little bit and the reason I use that silly container is because I don't want you to focus too much on what's in the container I want you to focus more on the container process itself so let's go ahead and pull that container um we're gonna pull a container called LOL cow we're going to call it

lolcal dot Sif that stands for a singularity image format file and we're going to pull that from docker from my username God love DC in this case and the container is going to be l-o-l cow all right if you have the GitHub repo up that has all the the stuff in it and you've got that kind of like beside your terminal or in another workspace or whatever most of these commands I'm going to try to follow that like as closely as possible because then you can just copy and paste the commands Okay so um so is everybody got this typed in pretty much we're gonna

go ahead and pull this container and it's going to give us a bunch of warnings because we did this without any root privileges and so it's saying I have all these root owned files in here that I can't make root owned anymore because you are a lowly unprivileged unit user and that's fine that's okay uh and then it's going to create a Sif file so what's actually happening here if you're used to using something like Docker or podman or something you might have seen a pull command before and what Docker podman does is it goes up and it finds all the different layers that an

oci container comes in and it pulls them down and then it creates at runtime an overlay file system out of them and then it kind of hides that somewhere where you can go and find it using Docker commands Apptainer is different Apptainer goes and finds all that layer all those layers that make up a container and it squashes them all down into one file system and it creates a file out of that and then you can interact with your containers using that file if you want to and so that's a kind of a fundamental difference between Apptainer and other container platforms it's one of the

reasons it's called Singularity because it's a single file so um so after you do this you've got this new file in your directory called lolcal.sif and that is your container it has inside of it um a little operating system image with a new root file system and it also has some metadata inside of it okay how are we doing on the pacing so far is everybody pretty much following along up to speed am I going too slow cool okay so now that we've got this um this this container we can do stuff with it in fact we could have done stuff with it even when

it was still up on Docker Hub but I'm going to go ahead and just do stuff with it here I'm gonna try to stick pretty closely to this oh you press enter well I I typed LL um yeah you don't have to but it's uh it's an alias of the command LS and LS lists everything that's in your current working directory and dash L um gives it to you in a long format so it's an alias of ls-l okay so now um I'm gonna go ahead and let's see I'm going to show you that you can also pull so what we just did is we

just pulled um a container that was an oci format it was in the format that's recognized by Docker you can also pull a container which is in straight Singularity format from Docker Hub assuming it's up there so I Docker is probably the most widely used Linux container platform it it's kind of waning now people use podman instead of Docker um but it was the one that kind of like popularized containers and made containers in the cloud native space really popular um yeah so this is a great question and this is something we can talk about too I'm going to take a little detour and we're

going to talk about Docker for a couple minutes okay so um okay as I said I have there was a previous lecture that some of the people who were here attended and I said that the dirty little secret about containers is that they don't really innovate and create anything new instead what they do is they take a bunch of features of Linux and kind of wrap them up in a single package and present them to the user in a way in which the user can use them okay Docker did that first before there was Docker there were charoots to root environments there were BSD jails

there were a bunch of other like little virtualization things which were kind of containers but they're really hard to use and Docker said okay I'm gonna we're gonna take all these tools and kind of put them together in something which is easier to use and um that kind of really it became extremely popular partially for because it's ease of use and partially because it was like a right time right place kind of thing web developers really needed a way to create microservices out of these big architectures we talked about this in the last course a little bit basically it ushered in new way for web

developers to create applications yeah yeah web applications and stuff like that yeah so Docker was the most popular container format for a long time and for several years and then people in the HPC Community started to be like oh my gosh Docker is awesome can we start to use that to run jobs in HPC and uh administrators like myself and others looked at Docker and said this uh assumes that you are going to have root privileges it assumes that you're going to be a root user or that it's going to be okay for you to have elevated privileges so we can't install it and let

users use it on the system um Greg Kurtzer at Lawrence Berkeley National Labs looked at this and he said we can't install Docker but could we create a container platform that would work for HPC and that was where Singularity started now you're going to hear me talk about oci a lot so um a few years after the the Apptainer platform was founded and started as Singularity at the time there began to be a desire mostly from the people in Docker to standardize containers and so they said we're going to start a Consortium called the open container initiative or oci and we're going to say this

is the new cons this is the new standard for how containers should be stored and how they should be run and by the way the standard is going to be based on docker so so and then they got um a bunch of companies involved and they got um the the Linux uh some Linux groups involved as well and so the open container initiative has become a standard it has since kind of grown Beyond Docker but it's a standard for how containers should be stored as layers within tarballs uh commonly referred to as a bundle for how containers should be run so how the run time

should work and how containers should be stored in a registry so now there's an oci standard for Registries as well okay that's a bunch of History um and then just so you know Apptainer sort of predates the whole oci thing and it's not oci compliant at all the way the containers are stored the way the containers are run and there is no like register well no there was a registry for apptainer for a while but none of that stuff's oci compliant okay cool so when we're talking about this Singularity image format file the Sif file this is kind of like one of the ways in

which Apptainer is not oci compliant because the way in which you store containers according to oci is you have a bunch of layers that are all in tarballs and you have a manifest and you take that you take them and Stitch them together into a container at runtime all right so let's go back now you can actually download from an oci registry Sif files because you can push sub files up to oci Registries so if one of those exists you can download one so let's let's look and see what that looks like I'm going to issue this pull command again but now I'm going

to change and I not and so instead of using Docker I'm going to use something called oras oras stands for oci registry as something it's it's basically a way in which you can you can kind of use an oci registry and just push arbitrary data to it so now I have to give it the entire URI because the oras command is not quite so forgiving as the doc or the or as transport is not as forgiving as the docker transport it doesn't assume docker okay so I have to say docker.io and then further I have to specify a tag so a single image can have

multiple tags and this is a way in which you can have like multiple different versions or you can have multiple different you know formats in this case of the single image so let's do that it's going to say I I don't want to do that because I'm going to overwrite your work I'm going to say Dash f oops or dash dash Force yes I do want you to do that and then it's downloaded the oras image from your perspective nothing has really changed here except now um you've gotten a native oras image and you can do things with it that you wouldn't be able to

do with uh with a Docker image you can do things like verify that the image has been signed oops which it hasn't in this particular case but you can do stuff like apptainer inspect def file and then you can actually get the definition file out that was used to create the image so I'm kind of jumping off the script here a little bit but this is why you would want to use an oras image instead of just a straight up oci image okay so this is just and sorry I I am really getting off the script I apologize um but this is just um you

know this is just how you get an image now that we've got it let's do things with it okay so I'm going to give you the most simple um container demo that I can think of I'm sorry Etsy OS release so I'm gonna look at the contents of a file on the uh on the operating system called seos release and that file is going to tell me what is the operating system that I'm running I'm running Rocky okay now I'm going to go ahead and do an apptainer shell lolcal.sif I'm going to Shell into that container image when I do that my prompt is going

to change cool that lets me know that I'm in a new image now and I can do a cat let's see OS release again and ta-da I just swapped out my operating system from Rocky to Debian and that's containers in a nutshell containers allow you to take an application and package all its dependencies and I mean all its dependencies down to the entire operating system and then run that that application anywhere because it's got all its dependencies with it and this is how it does it all right um I'm gonna go ahead and exit that container and then I'm going to show you really quickly

some other commands that you can do with apptainer so I just shelled into the container and executed a command um manually let me just make sure this is what I'm supposed to be doing next oh actually I was supposed to show you hold on I'm going to Shell back in and I'm going to say who am I and I'm going to say host name and I'm going to say ID okay now if you guys have used Docker or podman or any of the other container Solutions run C or whatever all the other ones that are out there this should be surprising you um you are

the same user inside the container as you are outside the container and the host name is the same inside the container and in fact you have the same uid and GID inside the container as you have outside the container and that's a core kind of like architectural decision that Apptainer makes on your behalf it basically writes entries to Etsy password and Etsy groups or Etsy group when you go into the Container to make sure that you're the same inside and outside and that makes reading and writing files from the host system trivial okay it also underlies the entire security posture by the way it's

like what makes Apptainer container secure so it's kind of a kind of a big deal that that's the way that apptainer works okay I'm going to go back into the Container I keep popping out and I'm going to execute a command called Cal say and I'm going to say cow say moo and when I do that you're going to see a cow and the cow is going to say moo um I'm also going to run pay no attention to Rose Don I'm going to run Fortune and Fortune is going to tell us some random Fortune okay it's gonna it's gonna select from a library of

Fortunes that it's got and it's going to tell us you know some Fortune then I'm going to run Fortune I'm going to pipe that into cow say and then I'm going to pipe that into another program called LOL cat okay and now we arrive at the ultimate purpose of this container okay it is to create colorful cow fortunes yes so class is dismissed we're done no but the point is when you downloaded this container you got access to these programs if you exit the container and you say cow say moo it's going to say bash command not found I don't know what you're talking about

so when you downloaded this container you got access to these programs and that in HPC is what containers are all about right I I um this other lab that I really admire has created this program that I want to run to analyze my data but I go to their GitHub repo and their readme says the first thing I need to do is sudo aptkit install these 10 programs as dependencies and then pip install whatever you know for them and I'm sunk because I can't use sudo I don't have app get install on my you know Rocky Linux cluster obviously so what can I do I

can go to my admin and I can pray that maybe they've got a solution and I can hope that they'll install it on my behalf or I can say that's easy I'll spin up a container with Ubuntu in it or a Debian in it I'll be root inside the container and I'll run apt-get you know and I'll install those dependencies and pip install the application and I'll just run it and next time I need it I'll just move it to a new cluster and run it or better yet I'll just go up to Docker Hub and I'll see where somebody from their lab is already

containerized it and I'll just download that container and run it easy as that yeah so that's that's the entire purpose behind containers here all right so so far we've just shelled into the container and done stuff manually which is really lame we don't want to do stuff manually in HPC so let's let's um try some other stuff let's exec a command so exec says give me the name of a container and give me the name of a command and I will go into that container and execute that command and then pop back out and then another way to interact with containers is to run them

so I'll just run this container now this is a little tricky what happens when it just runs a container well it turns out that there is metadata inside the container that says this is what should happen if you just run me okay and because Apptainer containers are single files which is pretty cool they're executable so if I just do an LL and I look at this this file holy cow it's it's executable so I can just execute it so I can just do this and there you go it runs all right so that is basic Apptainer look and feel and Action commands and how

to download them and stuff um there is yeah container uh running containers there's some stuff here on pipes and redirection I'm going to skip that the punch line from all that is that they work the way that you expect them to you can pipe stuff into the Apptainer command and it'll actually the input will actually go into the container and go through the program and you can pipe it back out which is not actually quite as interesting it just works um and then I showed you a little prequel here I did this inspect command I showed you that you can container inspect or not contain

what am I doing apptainer inspect def file lolcal.sif and it'll give you the definition file which we're going to talk about in a few minutes that was used to build this container what if we do something else what if we do apptainer inspect run script what the heck is a run script well a run script is that metadata that I was talking about earlier that actually tells the container what it should do when it runs and in this case it's exactly that command that we just executed earlier it's just Fortune piped into calcite piped into LOL cat so that's how the container knows what it's

supposed to do when it runs okay um XYZ Lord Gap um but it should give me like XYZ colorful texture um I don't think well lolcat works like that um you'd have to you could say Echo XYZ pipe to LOL cat yeah you'd have to you'd have to yeah because if you just said x y z what so bash is going to say there's a command X Y and Z and LOL cat that you want me to run it's gonna say I don't know what x is so I can't do the rest of it yeah let me just show that instead of telling you so

if I shell back into the yellowcat.sef so if I did X yeah I'll just do it all as one XYZ LOL cat well I don't know but if I do echo which is a command x y z and then I pipe that into LOL cat that's proper um that that's that's proper uh bash syntax all right um I'm trying to think um I don't know if you name is yeah it is so you name is in this container so now we're going to talk about kernels my friends so okay so you name Dash I think it's what Dash a like that no I was right

the first time I should just trusted my fingers okay um this is the kernel that we're running it's got this little Cloud stuff in it because we're running a kernel this is this is a gcp instance if you don't if you didn't you know already hack into it and figure out what it was and take over our account or whatever but that's what this is is a gcp instance um and so it's running this version of the kernel which has been specifically built for cloud and that's in the container right if we exit the container and we run the same command again it's the exact

same kernel and this is the major difference between containers and virtual machines okay a virtual machine um it starts off by virtualizing the hardware itself and then has a kernel that runs on top of that and controls the virtual hardware and then on top of that you've got your file system and your applications and all that stuff containers dispense with all that they just say I assume that this host has got some hardware and a kernel and I'm just going to grab that and I'm just going to slap a file system on top of that and that's going to be my container and that's why

you can start a container like that while it runs like immediately and why it's very performant why containerized applications run at the same speed as bare metal applications because they're just using the same kernel and the same Hardware so you had the question about kernel so let me ask you is that is there anything else that you want to talk about about kernels and containers like python that execute faster when you use containerized applications is that because of multi-threading no that specifically has to do with i o uh um so input and output operations and it well it kind of is because it's specifically related

to IO as it pertains to a network file system so you only see those kinds of i o bottlenecks when you're running on a network attached you know like a big parallel file system in HPC and you see that because there's a server that sits in front of the file system that handles all the data operations and when it has to handle a zillion of them all at once it starts to bog down OS running on the gcp that's a VM it's a virtual machine running on top of a hypervisor on top of gcp so this this yeah this software stack is actually there's some

Hardware somewhere and probably maybe Ashburn Virginia huh yeah yeah and actually you'd have trouble um I don't know if you could run a virtual machine here because this is not bare metal this is a VM running you know that's how like Cloud instances almost always are managed is you've got a bunch of Hardware but you don't actually run the OS on top of the hardware you run hypervisors on the hardware and you give your users VMS and um that's you know you might they might have some shim which allows you to run VMS inside of VMS I'm not sure um but it it's it will

probably be problematic but containers they can run VMS just fine that's easy in fact um containers they're only a Linux thing and some people like they tear their hair out when I say that because they're like I've got Docker on my Windows machine I run Docker on my Mac and I'm like ah yeah but little do you know that Docker installs a little lightweight virtual machine on your Windows machine or on your Mac and then it spins up your container in that virtual machine because now it's got a kernel it doesn't have a kernel otherwise and containers rely on kernel features Linux kernel features in

particular they're not native to any other operating system material you know Windows versus Mac oh well I was thinking more like x86 versus Arm yeah so that's a problem you've got to have containers which match that architecture yeah there's no there's no like virtualization layer that can say oh I'll translate one instruction set to another and it'll be really slow but at least it'll work no that doesn't that doesn't work at all internal containers on M1 and then let's export them to no um and uh another issue um so you can um so we're getting into the Weeds now but I like this this is

like I like that you guys are driving the the uh discussion here so the the kernel um kind of controls the hardware and stuff and then on top of that you've got a set of C libraries which are like Primitives that let you um let you interact with the kernel and these are called in linuxland the gnu C libraries or g-lib c for short and there is those don't have to match exactly there are you can run a different set of versions of g-lib C with a different set of kernels but that is that set is not infinite so um whenever a new kernel is

released there is pretty much guaranteed backward compatibility with whatever other glib C's have already been created that way you can update your kernel without breaking your system but the opposite is not guaranteed you can't necessarily take a really old kernel and run a really new glib C on top of it and you'll run into this in HPC sometimes you'll be running on a system that has an ancient operating system on it because they don't want to update because things work and you grab the latest greatest Ubuntu um and you slap that on your HPC system and all sudden you get these g-libs it's very nice

it actually gives you globsy errors it actually says your glimp C is is too new I don't know what to do with it um but yeah so you can run into that kind of uh mismatch between um your your container and your host operating system um there's lspci which will give you um Hardware which is attached to the the motherboard through the PCI bus um there's yeah there's a lot of there's lscpu which will give you information about the CPUs that are running I'm sure there's commands too to query you know what kind of memory you've got and stuff like that I don't know them

off the top of my head but those those will probably those are low level enough that they will probably be installed inside the container that you run uh we could test it but they should they should return the same where they were in the container or we're out of the container uh so just lspci real quick no that's not even oh LS CPU there we go lspci might be an s-bin sudo lspca yeah really uh sudo dnf I'm not going to pursue this for too long but I'll look use your S Pen bass OS okay it's probably not going to be inside the container

so it's a it's a moot point um wait a minute user has been LS yeah I don't I don't know I'd have to look around a little bit and figure out how to install that foreign CPUs here if we shell into the container LS CPU that's going to give us the same same information okay um now we could start talking about building containers where's my clock at there we go how much time do I have do we just have 15 more minutes or do we have oh we got an hour okay um so we could start talking about building containers so I've got this information

here about oci Registries uh uh there are a lot of different oci Registries not just Docker Hub there's also like um uh Harbor is another one there's um nvidia's got one uh there are every cloud has got their own registry you know Google artifact registry or AWS has a you know and some of them you need to authenticate to there's ways to authenticate I would basically just point you to the documentation um and say you know to get information about how to authenticate how to pull and also how to push to Registries and um you should also know that containers have tags and that there

is a special tag called latest and if you don't give your container a tag it typically just gets the latest tag and if you don't specify what container you pull you get the latest tag and that can that you can get into trouble doing that because it's not reproducible latest always gets overwritten whenever somebody puts a new container up so you can pull the same container twice within the span of a few minutes and get two different containers if you're not careful and if you want to be really specific you can get into Shaw songs and stuff and pull your containers by that but um

I would suggest that you check out the documentation on that topic um let's talk a little bit about building containers and then maybe if we can get through this quick maybe we can get into some other fun well let me ask you guys do you want to talk about building containers or do you want to talk about another fun thing I could talk about is how to um how to put your containers together as though they are applications and how to run containers as though their applications and another fun thing I could talk about is like how to manage multiple different containerized applications and present

them to the user as though they're just there uh but that would we would get into building a little bit when we talked about that stuff so I'll let you guys Choose Your Own Adventure what do you want second and third is there does people agree or do you want to talk about building containers I think we're going to cover you want to talk about building containers I don't know if we'll have enough time um okay I'll give you a crash course on building containers how's that you ever use a Docker file Docker file to build a Docker container it's the same but different you

use or use a definition file to build a singularity container the syntax is a little different instead of having like all those capitalized yelling at you words in front of everything you just have different sections I mean other than that that's that's it in a nutshell you can there's more complicated things that you can do once again there's there's good online documentation um yeah but let's do something more fun Okay so today oh oh oh I don't I don't know the answer to that but that's that's a good um question because when Apptainer containers were new and I was at the NIH as I didn't

give I didn't introduce myself to you guys at all um I I did in the previous thing I'm sorry I'm Dave godlove I used to be a neuroscientist I used to work at the National Institutes of Health then I became interested in high performance Computing I did that for a while and then I got interested in containers and I'm doing that now um so that's really quick so when I was at the NIH and container technology first came out we started to document it for the users because we're like building this and stuff and putting it on the system and I assumed when we're documenting

this the first thing every user is going to have to do is build their own container and so we had a big section at the beginning of the documentation about building your containers and how to do it and yada yada and before we actually got to the like running container stuff that I just went over and after a couple of years it became apparent that most users don't build their own containers they just grab them already pre-built from Docker Hub and run them and so we ended up taking that whole documentation and just turning it upside down and then just having like almost like a

footnote at the bottom like by the way if you have to build your containers here's how you have to you know here's the way to do it but yeah it's it's it's really um and part of that is because um here at Stanford University we had a collaborator who worked in medical school who was awesome by the name of Vanessa socket and she said oh my gosh there's all these containers up on Docker Hub already wouldn't it be awesome if Apptainer could leverage those and so she wrote a bunch of python code in a very short period of time to make that happen and

all sudden Apptainer was able to leverage all these hundreds of thousands of containers that already existed so that was like a huge contribution so yeah right here at Stanford cool awesome question um let's talk about a few different concepts and how we can use them for data analysis okay I'm going to gloss over building containers for the time being I'm going to introduce really quickly a concept called bind mounting okay um you don't have to follow along with this just watch what I do okay so I touch it I touch a file called Foo I shell into the container I look for what files I

have I still have a file called Foo it's still owned by me this is extremely confusing if you're used to Docker used to docker I touch another file called bar I exit the container I do ls-l and I still have Foo and bar and they're still both owned by me and all that and so the reason that this works is because apptainer does not try to fully containerize your environment it tries to do something that we call intelligent integration right what it says is um I know you're an HPC person and you're going to want to read data off the host and write data back

to the host so I'm going to take some a few special directories and bind Mount them into the Container from the host system at runtime so that they still appear to you within the container one of those special directories is your home directory another one is temp and then there are a few like process Dev that are important for just the operation of the container itself so that's the magic it's called My mounting you can control that you don't have to have that happen if you don't want to you can turn it off with this flag call it contain or if you want to bind

other directories you can do it with the bind flag okay so those are and you can do like another and another and you can control that with an environment variable okay so that's a crash course into bind mounting not in route not in Roots home directory and that yeah and that's there's that's a little bit special as far as root is concerned so um if you if you ran the container as root then you could see the files there uh in your in your home directory as root but you don't want to run containers as root um if you are not root when you run

the container you probably don't have the permissions to see what's in route yeah so your home directory yes that's what I just did this worked because I'm in my home directory so that's how I was able to see Foo and then create bar from within the container and still see all those things but if I did uh shell and then contain and then I did an ls-l I don't see those files anymore what happens if I say contain is that Apptainer just I think what actually it does is it creates a new home directory that's just like a blank directory and it by mounts that

in and pretends it's your home directory so I don't think so if you installed this is a problem with oci containers if you installed a bunch of stuff in the in like let's say you created some user and you installed a bunch of stuff in the home directory then that wouldn't work you wouldn't be able to see that from within the container even if you had the same uid on the host system and you had the same home directory um you wouldn't be able to see it because your home directory gets bi-mounted on top of what's there already so yeah you run into issues with

that kind of stuff that's created uh uh when you use the user ad or add user command to create a user on the host so so backing up I created a Google image I created a like a like a a golden image pretty much a Google VM and then I issued a command to start 150 instances of that in a loop on Google and so that's what you're running in and so when I created that image I made a root user I made a password I you know made sure that sshd would allow you to ssh in with a password and all that kind of

stuff so that's that's the environment we're running in right now okay all right so now you know about buying mounting and you know about building containers so we can do this other cool thing um do we still have the LOL cow yeah all right let's create let's remove Foo and Bar let's create another file okay I'm going to actually let me follow the script I think that we're going to do something in data yeah yeah okay um so let's pseudo make deer data okay we're making a directory at the root of the file system which we don't have permission to do without sudo so we

did it with sudo and now let's um sudo uh chmod and we'll do 777 super secure okay so what am I doing here um no I'm making yes well it has it is setting it executable because directories have to be executable for you to see what's inside of them but I made a new directory at the root file system called Data and I made it readable and writable by everybody on the system that's what that chmod 777 did okay so now anybody can read to or write from this slash data so you could think of this if you're in your HPC environment as maybe scratch

maybe you've got some scratch space where you tell your users you can use it during your jobs but don't be upset if somebody else overwrites it don't store anything there that you don't want anybody else to see because it's open to everybody okay now let's make a let's make some data in our in our slash data directory I'm you can do whatever you want here I'm going to Echo a word called that is happens to be routini and I'm going to say I'm going to Echo that to slash data java.txt because if you're a Star Wars fan you know that Javas say routini a lot

okay so now if I cat data and you can you can put whatever text you want in there but make a file call it something and Echo some some text to it and put in slash data okay so now this is like you're a researcher and you've got data in a shared directory and let's say you want to analyze this data in this case all we want is we want our Cal State program to run on it so that our cow ends up saying woutini that's going to be our analysis but you can imagine that you've got a file that you need to run an

analysis on and then you need to produce an output okay so one way you could do this is you could say apptainer actually I'm sorry you could say cat data java.txt pipe that to apptainer exec LOL Cal say and then we want to save that in a file so we could redirect that output to data text I'll let you digest that for a second so what we're going to do is we're going to take data java.txt we're going to feed that into our container do analysis on it and then we're going to save the output into Data wcal.txt and in fact that's not going to

work if you've if you've hit enter already it's going to say I don't know what you're talking about I don't see no data directory so you got to do a bind data and now from inside your container you can actually see that data this in essence so now I've got a new a new file called Data or data.wcal and if I cat data wcal calcutini I analyzed data through a container so this in essence is how you use containers to read in input perform some analysis and then save some output however that's ugliest sin right that like who can remember all that that's just terrible

so let's rebuild our container and let's make it analyze the data for us let's let's uh actually use some of the features of Apptainer a little bit Okay so I'm gonna see if I can do this um and see how well I can do this just from memory or just from figuring stuff out I'm going to do an apptainer inspect def file LOL cal.sif and I'm going to redirect that into lolcal dot Def cool now I've got my def file there is the good question if it's a single carrot it overwrites anything that's already there if it's two carrots it um it appends output to

it so I'm going to go into lolcow.def and look at it okay and this is the part where understanding definition files would have come in but for this for our purposes we're just going to look at this run script all right and I am going to rewrite this run script oh I should have told you um I typed to Vim and then the name of the Run script you might want to do Nano instead um if you already did vim and you need help exiting out of them in a few seconds let me know and I'll help you that was my fault because I did

that but you probably should have used Nano that's my bad if you're not if you don't if you're not a Vim user all right so now I'm going to rewrite this run script and I'm going to say cat um dollar sign one that's the first input that this run script gets feed that into calcay and output whatever I get to dollar sign two now in the manual that I've got that I've showed you it's a little bit more complicated than that I actually do a little bit of really Bare Bones error checking in my run script but it could be as simple as this so

let me just I'm going to double check because I'm I want to make sure that I actually get this right if I can find it yeah so that if where is it am I even in the right is this the right file yeah but that if is just doing error checking and we're not going to make any errors so I think that's okay uh yeah here it is okay yeah so I'm gonna cat dollar sign one into cow say and then I'll put that to dollar sign too so all this is doing is it's saying if the number of arguments is not equal to two

then Echo hey give me the right input and output and then exit with an error that's what that's saying um you know that's nice but we're not gonna We're not gonna type all that out right now um Escape colon X if you were in vim and that'll save and end exit is anybody stuck in them okay just uh hit enter don't touch anything else hit Escape yeah so yeah use Nano instead so hit Escape then make a colon then hit Q it's going to say no right so you can hit escape again colon Q force with Bang there you go and then hit type Nano

and then that's it'll give you instructions at the bottom it'll say here's how you use Nano oh man Rocky what's wrong with you Rocky all right sudo dnf install Nano yes okay all right okay so I'm gonna give you guys what time is this class over so uh we still have oh cool okay my computer is still on the wrong time zone still on East Coast so all right so I'll give you guys a second to edit that file oh I should show you the file yeah so just make it like this I'll circulate and check your work I don't know all right yeah if

you need any help just let me know and I guess let me know too when you're ready to go just all right how we feeling with thumbs up okay all right if we're not thumbs up it's okay you can watch what I do and be assured that it would work for you too all right so now if I've done this properly I can just run lolcal.sef I can give it the location of some data and then I can give it the location where I want it to save the data say data2.txt and it'll say there you go it will not work it'll be upset and

the reason is because um well there's two reasons sorry number one got to rebuild the file so I haven't actually rebuilt the container we're still running with the old container sorry about that so let's do an apptainer build Force because we want to overwrite our existing container lolcal.sif lolcal.def it's what oh it shouldn't be compiling anything no it's actually so what's actually happening here is it's just downloading pre-compiled binaries so the first thing it does is it downloads in this case I think an Ubuntu based container from docker and then it steps into that in Boone 2 base and uses apt-get to download pre-compiled

binaries from the mirror and just puts those in the right places it's not compiling anything oh yeah yeah uh not not right now I can't but yeah I can scroll up it's apptainer build here we go Force because I'm overwriting a container which is already there do you have an LOL cow.def file okay um um you could just watch for now um yeah and then you can pick back up if we yeah when we go to the next section you'll be able to pick back up again when the old version which was already built in this it was referring to the previous definition yes yeah

so when you're building it again why is it like fetching all the binaries again because it has to rebuild the entire thing from scratch so it has to execute the entire definition file this is also different from Docker in which you've got like um Docker has different layers and so it's intelligent enough to say oh if I ran these commands these layers would be unchanged so I'm not going to do anything to those I'm just going to pick up at this command where stuff is going to start to change it's going to change that Apptainer doesn't do that apptainer builds everything from scratch because it's

all in a squashed file system where it's it's actually um it's a compressed file system where you can't even like look at the files in it without actually mounting it up and doing all that okay so now if I've done my job right I should be able to do this command and it's going to say I don't know what you're talking about because it doesn't know what data is so I can do apptainer bind path equals data so this is the environment variable that I was telling you about I just mentioned existed earlier that you can use to set your your bind path you can

access these extra directories we did that with a bind command before we a bind equals so this is the same as saying bind equals but now I'm saying any container that I run after this environment variable is set just pretend I said bind equal slash data and this is we're going to have to do it this way now because we're actually executing the container with this syntax so we don't have the opportunity to pass command line arguments and so this should be silent because it just created a file called slash data slash data2.text not very imaginative I apologize and so let's stop for a second

get out of the weeds and get up higher level and look down at what this what this is what we're doing we have a containerized data analysis program within our lolcal.sef file and now we can run it as though it's a command and we can give it input which does not exist inside the container it only exists on the host system and we can give it output a place to save the analysis and it works so if you containerize your applications in this way you can kind of forget their containerized and you can just use them to run your analysis the way you would any

other command so that's kind of like the high level summary of what we've accomplished by changing the Run script and using this this apptainer bind path environment variable so this is kind of we're now trending toward the way people actually use containers in HPC yep and I expose an IP address I essentially create an API if yes if um the hmm yeah you could do that but you would have to be able to curl the IP address and get it to run an Apptainer command you could probably do that so if you have a web server or some sort of a you know web application

or something running inside the container um then yeah that will work and that's actually people do this with Jupiter notebooks right they install jupyter notebook inside of a container run it containerized and then just expose the IP address and the port and which happens automatically and then just connect to it um yeah you can run like nginx inside your Container for instance yeah yeah okay I'll show you so let's let's unset Apptainer minepath I'm gonna do an apptainer shell lolcal.sif I'll do an ls-l of data there's no such thing I'm going to get out of here reset the Apptainer buying path shell under the container

again uh skipped it LS Dash L data and it's there and it it is the same directory which is on the host it's just when it created the container it took that directory which was on the host and it bind mounted it into the container uh it's more it's not really vert it's it's the same storage it's not virtual yeah under the hood what's happening is complicated um Mount commands so I'm sorry every time yeah I'm sorry uh every time um every time you start a container the first thing that it does is it mounts a new file system it's actually much more complicated than

that it does a whole series of mounts because it mounts the file system and then it mounts it goes through some contortions to mount your data directory onto the file system that it just created and then it goes through some like unmount nonsense and it keeps doing that until it builds up this entire new file system which actually doesn't exist anywhere but it does out of all these mounts that it just created and so when you pass the bind environment variable or the CLI option you're controlling that process and you're saying here are some more mounts that I want you to make from one place

on the file system to another to make it appear that it's part of this new root file system the same files yes and they're not there's no copy there's no virtualization there's no um Sim links or anything like that it's just the same files and in fact you can do stuff too like this um you can bind that to something different with a colon syntax so I can say I'm going to bind that to something called new data and then if I enter my container there is no data anymore but there's a directory called new data and it's got the same you know it's got

the same files in it it's yeah it's not it's not a Sim link it's not a hard link it's nothing like that it's actually these are the files they're just they're just bought so you can bind a directory or I'm sorry you can mount a directory just like a file system from one place to another and that's all it's doing just remounting it yep all right it won't be there yeah yeah good question but yes that's it's still data so if I try new data yeah that was we we um as as root users in the root file system we created the slash data directory

find attention bypass data to new data so when in the activator we access database or you can or you can just or you can just use the syntax slash data yeah you can just say slash data without the colon new data you can just say slash data and then it'll appear there that's what we did the first time right unless it's in your home directory is special an Apptainer buy mounts that by default in for you so this afternoon part is stored in the um outside of actina it's stored in the home directory for students you mean the environment variable yeah yeah uh yeah that's

just part of the environment so if we type EnV you've got all these environment variables which are set up for you uh some of them just based on bash some of them based on some different configuration files that you have some of them might be in a special file called your dot bash RC in your home directory and so these are in memory these are not saved in any file anywhere they just configure your environment basically so I could do so I've got an environment variable called PS1 for instance so I could do Echo PS1 oh that's not right specify that it's an environment variable

and Echo its value that's what my environment variable PS1 does what does that do well let's save it so I'm going to say export old PS equals PS1 now I've got an old PS environment variable that says that okay so if I echoed that you'll see that so I just saved the value of my PS1 and now I'm going to say export PS1 equals that PS1 controls my prompt and so now my prompt is just a dollar sign it doesn't have all that other stuff about me being student at Apptainer class yada yada this is shorthand for telling it to say user at hostname and

then the directory that I'm in the working directory and then give it a dollar sign then if I say export PS1 equals the value with the dollar sign of old PS I got my prompt back and so environment variables do all kinds of stuff internally they can control programs like we just we just saw with Apptainer or they can control bash um that's all that's a whole that's a whole thing no um environment variables are not shared uh environment variables actually from your from your current environment do go into your container unless you specifically tell them not to but you can use the contain all

option to say for them not to go into your container if you so I did an export of the environment variable right there if you don't export it then child processes don't inherit those variables so if I start a new process from this process it doesn't get that same environment but it's really I think is a environment very irana very good right in your class yes charoot is an environment variable churroot is more than that though um charout is kind of what I was telling you about in the class so we talked a little bit in the class about the architecture of apptainer and what

it does and I went through the very first thing maybe not the very first thing but the first thing that we care about that Apptainer does is that it takes a new file system and mounts it uh into the the host file system and so that's all well and good but now you've just got a bunch of files sitting there on your host file system so how does it actually present that file system as a new root file system to the processes which are inside and to do that it creates a new um Mount name space and it pivots into that that's basically a true

root a charroot has been around for a really long time it's kind of a precursor to Modern containers so they're actually I think there's even a command true right yeah so you can just use actually the command to root to do that process that I just said to create a new root file system for your process um in your case you've got Dollar Sign capital to root and that points to a path and that's saying this is where my true root is yeah so it's the terms kind of overloaded a little bit there's like you know but but that's what it's saying hmm okay how

are we feeling we could take a step back and chill out and talk about building containers and fill in some gaps or we could be adventurous and we could take a step forward and we could go further down this Rabbit Hole of making applications appear as though they're installed on your system so which way do we but that's I'm going to tell you right now that one is a little bit more strenuous if you're tired and you're already kind of overloaded as far as information you might not pick all that up they're both yeah they're both there okay all right oh okay you have elected

a leader no okay what does everybody think um the we do we want the tougher option two three four Yeah okay or do we want to go back and fill in some gaps on you want both two I I can't take Soul responsibility for that the document the documentation is a um Community effort um let me go to Apptainer docs okay if you want to learn more about building a container go online here's the Apptainer documentation and I would suggest there's a section here called build a container and there's a section here called the definition file these are your pointers these were this is where

you need to go so build a container is going to give you all the commands and all the stuff that you can do with your environment and probably more stuff than you really need and the definition file is going to go through every single section of the definition file that you could possibly have and what they all do it's very exhaustive you contact Rose at ciq yeah yeah yeah so um if you go back to the the top of this where it's at actually I could have just clicked home I spent all this time putting all this navigation in and I never use it so

if you just click to home here there's a link there's a list of links here that can be helpful for you um so you can go to the Apptainer home which is where I just went to get documentation you can go to GitHub but there's also a slack workspace and there's a bunch of people there including me who sort of lurk and you know people ask questions and we try to answer them and then there's also there's a mail there's a Google Groups but it's kind of I don't think anybody really uses it anymore slack is kind of eating its lunch yes and then

you can also contact ciq UM I should have said that this is just like this is a little two-hour training um in which I'm going over a couple sections um I can I have given up to three six hour days of intensive Apptainer training and I I can do that you know uh pretty much at the drop of a hat if so if if anybody actually needs a more intensive not just like user but also admin Level Training of Apptainer or if you just want like a couple days you want to go into like deep dive or just a single six hour session or whatever

contact ciq and we can set something up for you okay all right so we're gonna press on we're gonna go to faking a native installation within an apptainer container um all right so this is the approach so when I was at the NIH um I basically convinced the other staff scientists who were there who were helping to administer applications on behalf of users that this was the best way to install applications moving forward at the NIH there's over a thousand different applications which are currently administered by the staff so and they've moved pretty much exclusively to this as the method that they install new applications

so this is like something that's used extensively in the real world um and uh you can use it too so what I'm going to do here is I'm going to create normally when you install an application on bare metal it goes into a directory hierarchy you'll have like you know especially well under Linux it's usually a little bit different but if you're like in a a high performance Computing environment you'll usually put it under a place like opt or maybe user local apps or something you'll have a directory where all your installed software goes and then you'll name the the software and then you'll give

it a version and then underneath of that there's usually like some special directories like bin where the binaries live maybe source for where you got the source code from maybe like lib or lib exec if there's libraries and stuff like that it kind of there's a kind of a specific directory hierarchy so we're going to go ahead and create that but instead of creating it with like a binary in bin we're going to create it in such a way that it'll call binaries inside the container and it'll be very very tricky and when we're done the user who ultimately uses this won't even have to

know that Linux containers exist they won't know that they're a thing and they will use all these containerized programs happily blissfully unaware that everything is being virtualized on their behalf so that's the beauty of this so you as an administrator if somebody comes to you and gives you something really hard to install that you're like oh my God I don't know how to install this no big deal just install it inside of a container and then just cram it all in this in this directory structure that allows your users to use it don't tell them it's containerized they'll never know all right that's that's what

we're going to try to do here so to do this I'm going to start off by creating this directory hierarchy and I'm just going to follow along with exactly what this does I'm going to copy and paste so I don't deviate it all so that none of us get lost and we can just do exactly the same thing I'm going to CD first to home just to make sure that everybody's in the same place you just type CD and then enter and then I'm just going to copy that command make directory PV says do it recursively so if the Parent Directory doesn't already exist make

sure it does exist and then V says do it verbosely so I can see it happening okay now we're going to go back over and we're going to CD bang money this is my favorite little bashism because it says bang money says what's the last argument that you gave me just take that argument and it just says CD to the directory you just made I don't know um it's I mean it's not so it's not going to work it'll work in bash so I think you're probably running bash on Unix so yeah it should work on your Mac yeah the Mac is not Unix by

the way the Mac is yeah that's a that's a whole thing it's it's Darwin but um it's unix-like okay so then we're gonna make a couple more directories called lib exec and Bin and those are just going to be right here in our current working directory so we've got now Ben and lib exec okay and now you don't have to do this but I'm just going to give you like a view of what's going on here uh uh yeah too so if I do an ls-l right here I can see I've got this this directory called LOL Cal install oops tree is not found yeah

yeah don't do this just watch me oops no no I don't know what that is I don't want to install that it's in the Rocky uh mirror so I'm sure it's fine anyway I'm going to go ahead and do a tree this is what my directory structure should look like I've got LOL Cal the version and then underneath that I've got Ben and lib exec okay all right so let me um CD into live exec you can just do lib exec if you didn't follow me I got to do this mess here to get to it but either way let's end up in lib exec

okay live exec is a directory where you would usually put like helper stuff helper binaries that are not really the program that you want your user to run but they're going to run on your pro on your user's behalf that's what libexec is for I'm going to go ahead and pull um my container to this lib exec and I'm going to call it app.sif and that's to underscore that you can do this with any application you want it doesn't have to be LOL Cal this is generic so I'm going to call this container app.sef okay okay now I'm going to create a wrapper script um

in this don't do this don't follow this in this I usually do this with more time and I go through step by step every bit of what this wrapper script is going to be doing um and that's why I do it stepwise like this what this should end up looking like is this and if we have time I'll go back and I'll talk about what this is actually doing but for right now just copy this text use Nano probably but your favorite text editor and call it uh uh wrapper.sh okay so we're going to call this wrapper.sh and then once we're there we're just going

to copy those contents there now what is this wrapper.sh script doing I'm going to go through it really quick while you're trying to make it exact to make sure that everything's good it's basically saying what's my name okay I'm surprised that Rose is not laughing right now like what's my name that's what the that's what the wrapper script is doing it's saying command equals base name zero that's who I am that's what my name is okay so it's like trying to figure out what its name is well you know already its name is rapper.sh so that's that's no big deal but the magic of this

script is if you Sim link another command to it it's name changes and so when you create some links to this wrapper script that CMD is going to change to something different if you don't understand that right off the bat don't worry about it I'm going to illustrate in a few minutes and then it just says what's my name where am I what arguments has the user provided and what's the name of the image I'm supposed to run and then it says okay you want me to execute an Apptainer command oh man this is only gonna here let's get rid of these quotes here it's

actually just execute it it's going to say okay you want me to execute an Apptainer command and that that command that the image is going to be in the same directory that I am it's going to be that image you just told me about app.sef it's going to be whatever my name is that's the thing I should run and then it's going to run with whatever arguments the user gives me okay so now what we're going to do actually um we're going to edit that script a little bit so go into that script again and we're going to add one more thing here we're going

to bind data now the user who's running this doesn't know about containers so they sure don't know about buying paths so they don't know that they have to you know tell this application that it has to be able to see Slash scratch or slash data or slash whatever they just want it to work so this is how you can control that and in fact there's other you can put whatever options you want in here if this is a GPU enabled Container for instance you could put an environment variable here that says run this container with the dash dash NV command that makes the GPU work

on the user's behalf without them knowing they have to pass a special command to make it work so you can really you know tailor this to the needs of the user okay now we're going to go down here we're going to create some Sim links we're going to CD to our bin directory because now these are the things that so Ben is the directory we actually want to put on our users path so you could install this underneath of lmod and when the module gets loaded you could add this bin directory to your user's path and so now all of a sudden whatever's in here

the user has access to and I'm going to make some Sim links and the Sim links are going to be of this syntax they're all going to be Sim linked to that wrapper.sh and I'm going to sim link our good old programs that we've been running this entire time Fortune Cal say an LOL cat I am linking three different commands to the same script and remember that this script if you make a Sim link to it that CMD environment variable in the script is going to change so it says what's my name oh my name is kausay you must want me to run cow say

inside the container what's my name my name is Fortune you want must want me to run Fortune inside the container all right so let me I'm going to CD back up One Directory um actually let me CD up a few more directories just one I don't know where I'm at anymore CD up up okay this is what this installation should look like we've got a a version number so we can install multiple different versions of this if we had multiple different versions we've got a bin and a lib exec directory and we have all these Sim links here which point to the wrapper script which

points to the the container and so now oh go ahead yeah you don't have to do the tree I just I installed the tree myself but you could do if you wanted to pseudo dnf install tree uh one question so in our wrapper.sh we put the obtain of Point part as data right but you don't have any data here is it outside uh I do have data right here remember I created that slash data directory oh because it is at slash it's somewhere else entirely it's at the root so it's at uh it's up here but that that slash data is going to appear inside

the container so we can access that data uh by doing this all right so now here is the like the big thing here's the big reveal if it works um I should be able to do LOL Cal install the version bin and then Cal say and that's because rapper.sh is not executable so let me change that jamad 750 uh because I've done this before and gotten the same error before you'd spend the next 15 to 30 minutes banging your head on your computer and then say wow what a dummy I am all right so LOL Cal not install uh live exec nope version live

exec rapper okay so what I'm doing here is I'm setting my permissions so that I have read write and execute anybody in my group just has read and execute and anybody on the system has read and execute that's what 755 means that's that's part of Bash yeah um it modifies permissions change permissions yes and that's um octal so yeah it gets it gets into yeah yeah it's bits yeah yeah that that's I'm not going to cover that in this class yeah all right in any case so now if I do a tree that should be executable let's try my magic command again huh oh uh

the color change just means that I haven't catted that to but let me not do that right now because I can make this even easier export path equals whom I'm going to add this binary path this bin path to the front of my path and now if I say which Cal say oh it's right there which Fortune it's right there and now I can do our good old and it all just works Army that's a pipe and that says take the standard input from the previous command and just use it as input to the next command yeah so let's back before we ask any more

questions let's once again try to get the high level uh the high level thing of what we just did here so I could pass this off if I had another user on the system and I could say hey just add this path to the beginning of your path and all sudden now you you can run Fortune Cal say an LOL cat or you know whatever like uh whatever I install you could run all that stuff so now another user and they don't have to know what Linux containers are like I can do these commands and I don't have to use apptain or anything they just

run you know you can install stuff on your system that you don't actually want to like dnf install or you don't want to sudo dnf install you can just install it locally on your system like this and when you're done with it you can just delete this entire directory hierarchy and not worry about it anymore so this is a nice clean way to use containers without having to run any container commands at all how can we delete this director why would it work immediately why would it work if we delete the directory so if you don't it won't work anymore if you delete the directory

so like I'm saying if you're so a lot of times I don't want to pollute my bare metal laptop by installing a bunch of stuff because it's going to go in you know I don't know where user local bin uh user Etsy all these different places where I don't want it to go and I don't know if I'm going to be able to cleanly get rid of it again afterward so instead you can just install it like this and then when you're done with it you can just delete the entire directory hierarchy and remove every single trace of it from your system like it was

never there this is kind of so if you guys know Ubuntu in Ubuntu um has gone to a kind of a newish package manager called snaps this is it's not exactly the same thing but it's kind of similar in spirit to the way that snaps work so Snaps are actually a little squash FS file systems which come down and run as containers pretty much and they have pass-throughs like that Apptainer um bind path environment variable which allow them to read and write data off the host system and all that kind of stuff so it's it's similar those are a lot more advanced than this but

it's the basic idea um one question and knowledge that all of them to rapper.sh right yeah yeah so what happens is yeah I should go back through it again now that we've seen it I should go back through um so let's go through line by line what the script does when I call it with let's say cow say this CMD equals so this this little thing here is a bashism and it says whatever the standard output of this command is allow me to use that and in this case I'm setting an environment variable to that so it says CMD is going to equal this command

base name with this is a special environment variable dollar sign zero which always refers to itself so if you just do dollar sign zero you're going to get the full path of whatever it is and base name strips the directory off and just gets the command so that's everything that that line is doing and so when I run this with the SIM link cow say that points to it CMD ends up being cow say no it ends up being the text cow say CMD ends up being a variable and its contents are the text cow say the directory uses a different approach to get the

current directory where the wrapper script is this is just a pretty robust way of saying where am I so deer ends up being the location uh home student lolcal dot install v 0.0.1 um lib exec that's what that ends up being ARG this syntax dollar at wrapped in quotes takes whatever input I gave to that command and just puts it into Arc it just transposes it directly and that's a kind of a special syntax in that it doesn't mess up quotes or do anything like that it just takes it as it is and just sends it directly into that environment variable that's how I can

do like calcium Moon it still works um and then image this is just hard-coded to say app.sif so if I change the name of that image this wouldn't work anymore I'd have to change it here too and then you know what this does and then down here it's just taking it's saying execute in the same it's making the assumption that that container is going to be in the same directory with the script so it's saying execute in that directory that image CMD or sorry app.sif give it whatever name I'm named in this case it would be calcet and pass all the same arguments to calci

or fortune or LOL cat or whatever else we're going to have to wrap up we'll get another class coming I will refrain yeah you're quite welcome