Container Education Series

to start off a little bit very philosophically I think what does it mean to actually build a container uh this could mean a lot of different things but I think today for our purposes I want to say that building a container is anything that ends up giving you a Sif file a singularity image format file that's the file uh format that we use for containers Navigator so by that definition see you could really uh do something like let's do something like this is this building a container so when I pull a container from like a container registry like this library that I've got configured I

end up um with a sip file so you could say that that's building a container right well you know really we typically kind of think of that as um downloading you just download a zip file and that's really all you're doing um yeah you're breaking my heart here that's how I build containers well let's look at another example Zane I think that there's there still is hope for you as far as I'm using the pull command and now instead of pointing uh at uh this Library um image I'm pointing at Docker instead right and so let's think about what's really happening when I pull this

container from Docker what really happens is that I download a bunch of different layers uh tarballs um is what they really are and all those tar balls are then untarred and then I use overlay FS to take all those tar balls and smoosh them together and make a single coherent file system out of them and then I take that file system and I um compress it into a squash FS file system and then after that I put that into a Sif file so even though we're just doing like a little simple build command really or I'm sorry a little simple pull command really this is

building a container okay this is taking a bunch of different pieces and putting putting them together and making it a container out of them um and in fact the reason that I'm going through all this is you know a lot of times people won't do this you won't really think about doing something like this but you could actually just build your container like this from a URI and this is syntax that I think a lot of people are probably not so familiar with but it works just fine that painter is quite happy to build your container like this and if you actually looked under the

hood to see what was going on you would find that app painter was traversing pretty much exactly the same code path when it did this as when you were doing the poll um so this is you know I introduced this to talk about like you know um you know the differences between pulling and building and the similarities actually between them and also to talk a little bit about like building from different uh from different sources right so right now we have pulled or built or whatever you want to call from the library we've pulled and built um from uh uh from uh Docker hub um

you can actually so so these are these Uris like and when we start talking about building containers these are actually um we're gonna start talking about these as bootstrap agents so these are agents that you use to start building your container from you can um also build your container from pre-existing containers which are right there on your file system but since I just built this container I now have this python zip container here and you not only can you build uh sip files but you can also build sandboxes right you can build directories which are just uh they're just bare directories and they contain a

file system in them so since you can do both those things you can build from a pre-existing container which is sitting on your disk and you can also um build to a Sandbox you can use the build command to just kind of switch between different formats so let me show you how that works so I'm going to go ahead and just build a directory called python from this python.sif very simply like that so now I've got this directory called python and if I look inside of it it's a root file system right um so that's that's a kind of a nice uh trick that you

can use to debug your containers uh if you if you're not sure where something is in your container um it can actually be difficult to find things if you shell into the container and I'll give you a specific example sometimes people install things in the container in root and they do so because it's not you know uh an unusual thing to to think that somebody might run the container as root especially if they're using Docker so they'll have access to anything that's installed in the root but if you're running a container that was built using Docker by apptainer then you're not going to have access

to the root directory because you just don't have permissions to do that when you enter the container you could enter the container as root but if you do that by default your um your home directory as root on the file system gets overwritten into the Container at runtime and so it can be very difficult sometimes to sort of figure out where things are installed because of that so you can just convert your container to a Sandbox and you can just look in here and say okay is everything in root and in this particular one there's not uh this is also a cool trick that you

can use if you're just trying to figure out how to uh build your container for the first time so you can do something like now that I've got this as a sandbox I can shell into it it's writable and then I can do things like you can update the container I can install things into it and so say I'm trying to install something for the first time it's got a bunch of dependencies I don't really know what all it needs I can Noodle around in the sandbox and then I can record whatever I do in the sandbox um in a definition file and then go

back and use this as a substrate to figure out how to build my container right so I'm going to go ahead and install them now because I told you that you can swap back and forth between different formats uh on disk by using the build command you might think okay great so now I can go ahead and build uh uh a new Civ file from that directory that I just created and that's you could do that so now I've installed them inside there and I did an update and so now the container is a little bit different but it's not considered a best practice because

uh if you do this you're not going to have any record in the container in terms of a definition file which we'll be talking about later of what you actually did so you won't have any record that this container was updated the Vim was installed anything like that so in general if you're going to Noodle around and try to use this as a development substrate to build a new container you probably want to record all your changes in a new definition file and then use that ultimately to build your new container so Dave would you say this is a good way to just test so

going through you're building your definition file you build it from that file system you test it that way you know that it works and you're not having to rebuild every time or change your definition file every time you're kind of playing yeah so one thing that you might do is shell into the Container um you might have like two windows open this is a way that I developed quite a bit so you'll have two windows open one of them will have your definition file and the other one will be this one in which you're interacting with your container and you'll be trying things here you'll

be like try this try that and you'll continue to do things and record the things that work in your definition file until you break your container right and then you'll build your new container from your definition file shell into it again and continue to Noodle around with it um recording as notes in your definition file what works and continue to do that until you ultimately have everything that you want to have installed you've got your definition file you build it one more time and there you go foreign all right so let's let's talk about definition files um okay so I've got a definition file I'm

going to go ahead and just remove all my python stuff I've got a definition file here called um example uh dot Def let's go ahead and look in this I've been noodling around with it so I've got things uncommented right now let me go ahead and just comment everything all right so a definition file uh this is a this is an example definition file which basically has pretty much all the commonly used um sections in it that you would use to build a container a definition file begins with a header and then after the header uh it has a lot of sections right um and

I've got all the sections commented out right now so that in this particular instance uh none of them are going to run when we begin so here in the header this is where you uh you you tell aptainer what is the base container that you want to start off with and here I mentioned earlier the concept of a bootstrap agent this is where you specify what is the bootstrap agent that I want to use and then depending on what bootstrap agent you have here you'll have different keywords that you can add to your header right so here I'm saying I want to bootstrap a container

from Docker Hub that's what the docker bootstrap agent means and then I'm saying I want to start from Ubuntu now this syntax might be a little unusual to most people that I'm using here um and you know most people might be more familiar with something like this or you know some tag here I don't know if this is actually a tag I'd have to look at look on Docker Hub to see but it might say something like this the reason I'm trying to highlight this particular syntax is because if you just bootstrap from the latest of a particular image then obviously it's you know that

latest one might change and so if you run the same definition file to build a new container tomorrow it might not be the same as the one that you built today um that's true pretty much for all tags so while it's not a best practice developers can swap tags with new containers anytime they want to so if you can actually use a shot sum to specify that I want this particular image and that's the only one that I want and if you do that even if a developer removes this image from Docker Hub you'll get an error that says hey there's I can't find this

image anymore and that will alert you that things have changed um you know you won't get just like silent errors you'll get more specific error telling you hey that this is not going to work anymore you need to have a plan to move forward foreign that's the header and we're going to talk more about different bootstrap agents hopefully later on if there's time um but I'd like to say one other thing about uh the benefit of using a Shaw there you know you're pulling these down yeah from the docker like the the normal docker.io registry but with the shaw there you could point it at

a different registry and know that you're getting the same container no matter where it comes from which is also a pretty cool thing that's right yep um okay so I'm gonna uh start going through the sections too here so we'll talk about different bootstrap agents and different headers later on um right now I just want to cover all the sections so the most common section that you'll see in almost every definition of follows this post section and post is called post because it runs after okay so it's a post something and that something is that it's going to download um your container it's gonna jump

into the container that and so after the container downloads and after the container is created it's going to jump into the container and do this and so this is just like this is just some bash gripping it's just a little script lip so anything you can write in bash pretty much you can write in post um here I'm setting the environment variable Debbie and front end to non-interactive this is going to ensure that apt doesn't prompt me for any input when I use it um and then this is kind of silly I'm just going to set this uh package variable to wget and then I'm

going to use apps to install the package we got to start off pretty simply when I build it I'm going to use this fake root um directive and I'm going to use Force which means that this container already exists just go ahead and overwrite it it doesn't exist right now but I'm going to build the same container over and over again so might as well just start using Force okay so you don't use the Force Base is it just override it or does it air out uh it doesn't either it actually prompts you and says um you've you've it gives you a little yes no

prompt and says are you sure you want to do this okay and so you can just hit Y I can share that to you later if you want Dave are the is the post section always like explicitly and definitively Bash or or born shell or can you tell it to use different shells do you know um that's a good question uh so it used to be so there have been iterations of abtainer and I don't know if the current iteration is like this but there have been iterations of Singularity slap slash obtainer that allowed you to specify um I don't know about specifically post but

the Run script for instance you could specify a different shebang and you could use that to do things like uh create a run script that was written in Python um I don't know if the current version of apptainer still does that or not I'd have to do some tests to figure that out so I'm not prepared to say definitively right now sure all right um cool okay so so it went through and everywhere you get a little plus sign here that tells you what you actually ran during your post section so I set the environment variable I installed wget and then it created the Sip

file and so now if I did something like exactly um just to make sure everybody gets actually there yep then I can use it okay so that's pretty straightforward so this is analogous that post section is analogous to anything that's prefaced with her with the the Run directive and a Docker file right okay so now I'm going to go through the environment section that's another very common section that you'll see um I've explicitly created the same environment variable in environment as I've created in post and I've done that to highlight that these two are fundamentally different so if you create an environment variable in post

that environment variable will be available to you at build time but not at run time and vice versa if you create you know this Foo equals bar that's not going to be available during build time um I just wanted to highlight that and then also you'll see LC all equals c quite a bit uh in the environment and that's basically just setting your Locale there are different things like Pearl and some python packages which care about how the Locale is set and this is just a simple way to sort of set it generically and make sure that you don't get warnings and errors and things

of that nature um let's see in the entrance of time let me go ahead and and uncomment a few a few more of these before I go ahead and build this um I'll go ahead and uncomment the next two explain what they're doing okay so here we have a run script section and this is once again um Justin's oh and I should also highlight that here in the environment section uh this stuff actually gets um gets run it gets this is a scriptlet also which runs so typically you would guess set environment variables here but you can actually um you can do other things here

uh as well although it's usually it's not really considered a best practice to do it here the environment section if you want your container to do something dynamically you should probably put that in the Run script sorry I got a question on the environment variables can you overwrite those on the command prompt can you overwrite what those variables are as you execute a container yes you can you can do that um and in fact you can do that uh let's see there's a special environment variable within aptainer um that will uh you use it to inject environment variables into the Container at runtime um and

then there are special environment variables that you can use to pre-pend and upend to the path at runtime if you want to change those as well okay so yeah there are ways to change these at runtime if that's you know what you're looking to do okay yeah I run across containers before where that was nice to be able to do yeah yeah um so the Run script is just like a little scriptlet so you're just going to write bash here or um you know if you change the shebang uh to something else changing the shebang in the Run script is not as easy as just

writing a new shebang here I should say uh you have to I I actually forget off the top of my head how to do it I should have uh brushed up on that before doing the demo but um you can I'm pretty sure with the current version of obtainer change that um and then I understand we've had it confirmed that you can but not yeah I haven't messed around with that since I paid her with like in the 2x series I think but that's something that uh actually I fought really hard to like have include um the file section um you you basically just put

a path to a file on your host system and then a path to your file within your container and it copies files from your host system into your container at build time so that's basically the syntax there let me go ahead and run this again and then we'll have a look at you know what's going on inside the container all right while that's building Dave you mentioned that the environment section is a scriptlet uh similar to post the the one place where I've used that to I think good effect uh is if I install HPC applications inside of a container and it comes with lmod

um uh module definitions inside of it I used um I used environment to load environment modules and then load those modules into the environment so they would be ready for the Run script to run and to do that you Source you know an lmod.sh or something like that and I was pleased to discover that that worked just exactly as I wanted it yeah that's cool that's a that's a very cool and creative use of uh of the environment section that's one of the things I like about apptainer is that yeah there's like the best practices of how you can use it but then it's flexible

there's other ways you know there's other things that you can do with it um so I used the uh the file directive to copy a file called too many Daves and that's just a little Dr Seuss um uh little Dr Seuss poem I decided to include it because there are lots and lots of Daves in the apptainers uh Singularity community in particular but everywhere so um I thought that would be funny I also like how you left it okay I like how you also less file less too many days okay so I'm going to jump in this example dot zip file and so right off

the bat so that Dave's poem that's when I copied it to with the file section so now that's there um yes they do yes they do um so I I went ahead and uh what else did I do well I I set some environment variables let's have a look at those really quick so ifu actually got set appropriately um I trust that the others did as well um I also created a run script and um I want to show you where that ends up and where the environment variables end up so if I see to my my root in my directory and then there's actually

a hidden uh directory here called about singularity.d so now we're going to start getting a little bit more advanced stuff so this.cingularity.d directory is actually where like the configuration for the container itself lives within the container so if I look at that I can see this thing called Singularity um I can see a directory called EnV I can see something called run script let's look at that real quick if that's the Run script that I put uh into uh into the definition file notice it's got this shebang here so by default that shebang is added for you that's why if you add your own shebang

at the top of the Run scripted actually it won't do anything it'll just add the shebang underneath so you actually have to change I think it's a directive that you add to the end of your um your run script section uh but so anyhow um that's where that ends up let's look at the ENB subdirectory here this is a whole lot of just Little scripts and what these little scripts do is they set up initially the base environment so what the path ought to look like if you've imported this from Docker it takes the environment variables from Docker and it actually you know puts those

in um here in 90 that's actually where the environment variables that you've put in yourself end up okay and so what happens is when you run the container all these scripts get run in order and it layers up the environment variables and the one you were talking about Zane um at runtime you might actually inject other environment variables well those get written into this script I'm pretty sure and then get Source last so that they overwrite anything which was sourced here in the environment and there's some other stuff here too that has to do with like um like apps you can install multiple different apps

and there's a whole kind of way to do that within a container I'm not going to go into that but that's a pretty cool feature that we've had in that painter for quite some time um but yeah so that's how that's how the environment works and then there was this kind of mystery file called Singularity here um back when single back when aptainer was still called Singularity like everything in its brother was called Singularity um and this happens to be the definition file that was used to create this container so this is how you always have uh a record of what was used to create

the container um when you build it and it's worth noting too that even when you do one of those pull or build commands uh with obtainer under the hood what happens is that aptainer translates that command into a little short definition file and uses that to create your container and it will actually Jam that definition file into this Singularity uh file so you can still see that you know you you use the docker bootstrap agent and you got it from you know the following Container for instance okay all right let me go ahead and just go through some of the rest of the sections here

okay I'm gonna I'm gonna just go ahead and do the next three sections so labels is a is a section that you can use to put just arbitrary metadata into the Container um so this this stuff doesn't actually end up in the container itself but it ends up in the Sip file and so you can use this to do things like declare your authorship or what version of the container it is or whatever um The Help directive the help section is not actually a scriptlet it is just um text just straight up text and whatever you put in there if you use the Run help

command with the container afterwards it will display this text to you so you can use this to you know give your container some help for uh users down the road then test is an interesting section test allows you to so at the when the container builds at the conclusion whatever is in this test section is just going to go ahead and run um unless you specifically tell it not to when you build the container and then after the container is built at any time you can go ahead and run this text Test Section again so maybe that'd be right just do it and that'll uh

make note sense so I have a couple questions about that Dave the first is do you happen to know if the authorship and other metadata is included in this uh the signed payload for if you if you sign a container yes it is yeah it is okay yeah it would make sense to me but okay my second question is if if the test scriptlet changes like makes a change to the file system if it writes something out will that change get captured in the final container or is that discarded that's a good question I don't think it can change the file system you think it's

read only by that point yeah because I think that at that point um the container has already been created and I think the test gets run probably without well I don't know it it probably can't change anything that's rude owned because it probably just runs with normal permissions that'd be an interesting thing to play with I don't know okay okay so at the end here we now have extra info telling us that we've added labels um we've added a test script we are running the test script it tells us that yes wget has been installed if I want to run that again I can do

something like test wget has still been installed um yeah so so the test script is added and is run before the Sif file is created um so at that point we're not the container is not immutable this is why I'm I'm kind of thinking about your question I'm not totally sure whether you could alter the file system with it or not um that's a good question so it's something to play with okay I wanted to show you what the labels look like really quick so if you wanted to inspect the container for labels um so your container ends up with a lot of metadata just

whether you put it there or not uh and this is it's all called org.label schema this is oci um formatting this is why it looks like this but you can see that I added some arbitrary metadata as well the version the container and uh you know username or my my email address and then finally you can run help on the container and all that text that you put in the help section will just go ahead and get outputted so you can you know for future you or for somebody else uh you can go ahead and um you know put all that in there so that

so that the container is well documented okay um do you have any questions about that stuff now you're Dave all right so now I'm going to get into some dicey stuff Daisy stuff but you probably don't want to mess around with um these are not it occurred to me after I was putting this together there are a few other things you can do with definition files which I'm not really covering today uh one of those is there are keywords that you can use to set up different apps you can use this is I don't know how widely this is used but you can actually it's

a pretty cool feature you can install multiple different apps in your container and you can do so in such a way that you can actually use app team the app tainer itself is made aware of the different applications which are installed in the container and you can call the different apps through app tainer itself okay that's it's whole other that you know that would take a long time to go through so I encourage you to look at the app Tanner documentation it's pretty well documented and it's it's a Nifty little feature uh and then another thing that I'm not going to be covering today that

you can do is multi-stage builds and the idea there is that let's say you've got some complicated um you know build tool chain that you have to install to compile some code in the container but you don't want that in the ultimate container because it's just going to be bloated and you know whatever you can uh create one container that has all the uh the build Tool uh the build tooling installed in it uh compile your code and then in the same definition file create another container which is has all the libraries and stuff that you compiled against or whatever it is that you need

and just dump that compiled executable into the new container and you know basically just throw away the first container this is using go quite a bit uh this is a pretty um you know Common and straightforward thing that most people who use Docker are already familiar with so go uh creates uh statically linked um executables usually by default and and so a lot of times you'll just compile some go code and then you'll just take the resulting executable and dump it into a bear or very spelled container and then you've got a container that just basically runs that go executable and that's it once again

check out the obtainer documentation um it's really well documented how to do those things now I want to talk about a couple of sections which are not really very widely used um and you know when you would and wouldn't use these so setup is pretty much Legacy setup runs on the host system not inside the container and it runs after the container has been downloaded it used to be that if you wanted files to go from your host system into the Container you would use setup and that was pretty much what it was for since then the files section has been added to the definition

file so setup is pretty much not really used anymore and it's pretty it's kind of dangerous and I'm going to show you why it's dangerous uh in a little while so you probably shouldn't use this unless you've got some weird Corner case that you absolutely have to do this free can also be a little bit dangerous because it also runs on the host not in the container and it runs actually before the uh the container gets downloaded so um so just like post means after pretty means before and it's before the containers actually downloaded and once again this is something obviously since it runs on

your host system it's a little dangerous to use but uh there may be some you know places in which you really do want to use it and I'll show you why later on okay let me go ahead and build this again um all right so now we are doing that pre-scriptlet and it's echoing well don't use that because that's what I put in Pre unless you know what you're doing and then sleeping for three seconds so we've got time to read that then it tried to do that setup and it said um it failed because you got permission denied now okay why the heck do

I have permission denied I ought to be able to create a file called made by called slash made by setup right all right well I ran this as fake root maybe I should just run on his real root work darn it okay so now um hopefully uh it's gonna let me create this made by setup file in my container you know wouldn't that be wonderful um I'm just gonna go through and do all this stuff [Music] I think I'm getting messages to the effect that I've been fired there were too many Daves we had to get rid of some of them it makes sense he

makes the official name change forms people don't like laws in this it's true it's intimidating all right so this time hopefully my container worked my internet's being slower now so whatever internet um okay so now I should see inside my container that it made that file that I was supposed to be able to make uh like made by setup or something I called it and unfortunately I don't see it here I wonder why not until you're dating oh wait a minute let me look at my root file system of my host oh crap crap so I've got this file called made by made by setup

now at the root um it's owned by root that I made with a definition file and that's why you know set up and pre are a little bit dangerous because they run on your host system and sometimes you build containers as root so if I had decided to delete a bunch of stuff for instance I'd be in big trouble because you know especially if this was a host system that you know I really used a lot and cared about and everything um so anyhow you probably shouldn't use those let me go ahead and just remove this okay any questions oops before um I get into

talking about different uh different bootstrap agents I think you mentioned that more than set up there were instances where you might want to use pre did I miss that or did you talk about some reasons to use it now that's what that's foreshadowing right so that's upcoming okay and yeah so as I start talking about different bootstrap agents pay attention because there is one in which um pre becomes kind of important I think it's pretty interesting it's probably one that people don't know about very much okay I have here in my directory this um directory called aptator and this is the app payer source code

that I just grabbed directly from uh from GitHub so you can grab this too and if you go into it you'll see that there is a subdirectory called examples and examples has all these different short little definition files mostly uh what they illustrate is different um different bootstrap agents so rather than just like write a whole bunch of these myself I decided to just use this directory I've written a couple ones that aren't highlighted here very well um and that I you know I should probably just make some pull requests and try to highlight those a little bit better but for the most part you

know this is a pretty pretty nice and complete library of different bootstrap agents that you might use all right I've already talked uh in depth or shown you in depth uh the docker bootstrap agents just grabs you know things from Docker Hub right um and so that that makes sense in this case we're grabbing the official busy box uh latest one let me show you there's also Library so you can configure um you can configure obtainer to point to a library and a library is a container registry which contains native sip files which is pretty cool so instead of uh grabbing you know something that's

a bunch of tarballs like an oci format you can actually grab things that are native Sif files and so that's what this looks like on this particular one points to the scilab library um and they're grabbing the Alpine the the official Alpine 311.5 container here um there's another cool one that allows you to interact specifically with Sif and that bootstrap agent is called oras oras is a is a oci method uh for um storing arbitrary data in an oci or Docker style registry which is cool because it may means that you can store Sif files in an oci registry um so in this case we're

using the oraz bootstrap agent to download a container from Google from um Google from um from GitHub the GitHub container registry right here so you can grab their stuff up on the GitHub container registry um you could you know it you can use that yourself to store your containers and then you can grab them and and use them as base containers to build your containers from pretty cool uh okay um then we also so there's another cool one that I want to talk about another one um I think this one's a little bit less known we don't actually have I think this is the one

of the ones that I wrote specifically for this demo real quick but there's a bootstrap agent called local image let me just CD into that local image directory I actually have a Sif file that I've created here and if I look at the local image definition file the bootstrap agent is local image and you give that either a relative or a full path to a sip file which is on your Local Host foreign and then um after you do that uh you can you can use it as a base in your in your definition file that's pretty cool a little less uh reproducible than some

of the other methods that we're showing here but um could be handy if you're doing development okay now all the different bootstrap agents that I've talked about so far you start from an existing container right and there are some drawbacks to doing that um what if you want to create a container and you don't want to start from an existing container you want to start from you know something else maybe you want to start from a mirror URL uh from which you would get the components for a file system for like you know Rocky Linux for instance or maybe in Ubuntu or Debian or something

how would you go about doing that okay well let's take the example of Rocky [Music] um the way that you would bootstrap a new Rocky Linux image not one that's based on a container is that you would use yum to create your file system now this is going to run outside of your container obviously because it's got to put this thing together and make a container for it to jump into so for this to work you have to have yum installed on your host uh once you've got the Yum bootstrap bootstrap agent you get some um some different keywords that you can add to your

header uh so here we've got the mirror URL keyword for instance and this includes keyword actually takes the configuration of yum on your host system and puts that into the Container at runtime also so that you'll have a functioning usable yum in your container and that's that's something that's a little bit specific to RPM based images that you start from uh from like a mirror like this so this is kind of cool because if you think about it um almost all of the container ecosystem depends on existing containers and depends on starting from existing containers and when you start from an existing container you have

to trust a bunch of people right you have to trust you know the The Entity or the organization or whoever that built the container that they didn't put anything malicious in it and you also have to trust whoever is administrating wherever the container is owned or wherever it's stored so when I download a container from Docker Hub I am like sort of implicitly trusting that the developers of Docker Hub itself um are you know putting some security best practices in place and are themselves not malicious which I'm sure they're not but you know there's a little bit of trust that goes into that right and

says I've grabbed this container from this third party registry and I trust that everybody protected it did what they were supposed to do to make sure it doesn't get messed with and I'm trusting that whoever originally created this container didn't mess with it and so if you do this instead really the only entity that you have to trust since you're ever maintaining these mirrors right which you ultimately have to trust one way or another anyway so this is you know if you're paranoid which everybody ought to be right this is a much kind of cleaner and better way to start off building your containers and

you lose though some some key features by doing this it becomes clunkier and harder to write your definition files um you can't use things like signatures and things you can't have your containers encrypted so I'm going to show you there's like a um like a compromise that you can do where you can basically build yourself your own library of containers to start off with from mirrors you can sign them all to make sure they don't get pampered with and then you can use those as your base containers that you start off with and you can always you know be pretty well assured that you're starting

off with a good nice clean container at the beginning of your build all right let's look at um so if you wanted to build a Debian type of container like Ubuntu for instance um that looks a little bit different so in that case instead of using yum or instead of using App there's actually a tool called ve bootstrap or Dev bootstrap which you would use to create your file system and this is specifically a tool created uh or you know programmed to create um Debian style file systems once again the tabs to be installed on your system um which is kind of weird so I'm

running you know a rocky host system here but I actually have uh the bootstrap installed on it which is kind of strange but you can do and it works just fine and then this is kind of like the final thing I want to show and I I see very very peop very few people using this but it's pretty cool um one of the community members contributed this code a long time ago and had this great idea and uh so I want to show you a scratch build this is like this hardcore you got to really know what you're doing to do this uh let's say

there's two here there's Alpine and BusyBox I'm just going to look at Alpine really quick so there's this uh bootstrap agent called scratch and what it means is give me an empty container that doesn't have anything in it give me nothing and then I will do everything and this going back to Jonathan's question here's where the foreshadowing pays off is where you would actually use something like setup because remember that setup runs before your container gets downloaded and it runs on the host system in this case what we're doing is we're defining an environment variable which is a URL and then we're curling that URL

to grab this this Alpine uh this Alpine file system it's a it's a tarball we're untarring it uh and we're sending it to this this special this um this special environment variable called aptainer root FS which is only available at build time during the setup um during the setup section and what this does it basically extracts the root file system into our container and then we're just excluding the slash Dev and Etsy hosts files and directories because you know we don't need those for our container to work properly so with this scratch um bootstrap uh agent you can basically install anything you want inside of

a container um you could go through and use de bootstrap yum uh any of the other you could use Pac-Man you could use basically anything that you can create a file system with or I mean it doesn't even really have to be a root file system but your container is not going to work very well under apptainer if it doesn't have a root file system in it probably although it might but in any case this this allows you ultimate flexibility this is like you know however you want to build your container go ahead and do it all right so let's take any questions and I

just want to talk about one more thing um so is there any questions on bootstrap agents I haven't seen any yet Dave all right cool very cool I haven't seen a lot of that before so that's very cool thank you yeah so I lulled you and you know I started off really really slow and then I got into some of this crazy stuff um you mentioned that you lose out on some functionality like signatures when you use say a yum bootstrap agent um why is that that surprises me that you wouldn't be able to sign a container like that oh no so um and that's

that's a great segue that's exactly what I'm gonna get into so it's not that you you're unable to sign a container like that you can you can easily sign a container like that in fact that's that's a great thing to do but you can't like it's start you can't the point of origin doesn't have a signature you're I see and I hesitate to say that because it does right yeah the repository will have signatures and things like that but yeah okay right but it takes a long time to um it takes you know if you could just download a Sif file that already has the

root file system already in it versus building from scratch the root file system every time you want to build a container you know what I'm saying it's just a lot easier to just download the existing especially if you've got like a slow internet connection or whatever it's a lot easier to just start with an existing root file system so to do that what you can do is you can just create so so and this is like this should be like practical real world kind of stuff right so when I was at the NIH one of the things I was kind of tasked with is let's

come up with a library of Base containers that we can vouch for that we know you know right off the get-go don't have malicious code in them and we'll use those as um you know as the base containers that we that we build our containers off of right and so those still exist they're up on the library so if I do something like obtainer whole all right so I've got this container up there I've got it in a uh I've got it in a collection that I call secure okay and then if I let's see let's name I did this into the repo I didn't

mean to do this in the repo but any case here's what it's named centos8 latest.sev so I can um look and see apptainer inspect file okay so this is the definition file that I used like two years ago to create this container I built this specifically right from the mirrors so I didn't use any intermediate like uh container that somebody else had built um and I didn't even use like a container registry I just used the mirrors well how do I know this thing's been sitting on a third party registry though for for the past two years how do I know that this container hasn't

been tampered with well that's where your verify command comes in so I can verify I can go you know look up my key and I can verify that that's the signature that I signed this thing with two years ago which it is and so because it is um I know that this container is a bit for bit reproduction of the one that I made a few years ago and that it was made you know from these mirrors and so I don't really have to trust anybody uh when I use this as a base container I can be you know pretty confident that this is clean

and it has in it what I know is supposed to be in it to use this as a base container and I've got like a a couple of Centos containers of a couple different versions a few Ubuntu containers a few Alpine containers so all those can be used as nice clean base images um to start off with foreign cool so uh any questions about that workflow oh yeah one more I didn't let me close the loop though because how do you how do you check the signature when you do a build right do you download the container and then do a container or didn't do

an app trainer verify and then build from the local container no you don't have to do that so write down the definition file you can put that fingerprints uh keyword and this is really nice because apptainer is going to bomb out immediately it's not going to try to run anything if it sees that this container is not what it thinks it's supposed to be um so it'll just run the verify command on your behalf at build time and you know as long as the container checks out it's good it'll go ahead and build your container and then you'll have um you'll have this record right

in the container in the definition file of you know what you built where you built it from what the fingerprint was and everything you need all right cool that that hopefully that closes the loop I think so that's fantastic thank you so we don't have any questions and I haven't seen any come in I'll have anybody have any things they want to say before we leave or right up on time closing thoughts Jonathan no this is great uh I like you said same a lot of the specifics of some of the the outside um uh bootstrap agents were something that I hadn't seen so thanks

for going over it with us Dave absolutely yeah absolutely I agree there's a lot of info there that I didn't know so far um about like those bootstrap agents and yeah just a lot of great little tidbits in general Dave so thanks for bringing that yeah appreciate it my classic LOL awesome thanks for comments Ryan well on that note we will go ahead and wrap up Dave thank you very much uh join us next time in two weeks for the next of this series of Education uh without Tanner if I believe that is the way it's going to go so like And subscribe and we

will see you soon thanks guys thanks everyone bye everyone