Apptainer: Deep Dive, Use Cases, and Examples

to begin with you know um when i started giving talks like this several years ago i would i would usually have to start by talking a little bit about or maybe a lot bit about what linux containers are but these days i feel like most people have a pretty good handle on what a linux container is um if you don't i think i can summarize it really quickly by just saying you can conceptualize a linux container as a lightweight virtual machine it's a little bit different from a virtual machine in that it it shares um the kernel with the underlying host system and so that

does make it a little bit different makes it a little bit more lightweight and a lot more high performance and um linux containers utilize uh some features of the linux kernel and the some other linux features of the um of the file system for instance and it just they just basically take these features and wrap them up together and present them to the user in a nice package that makes containerization easy um and you really see them everywhere these days they're ubiquitous within computing so what is apptaner now if you're a regular um watcher of uh you know of this webinar then i don't have

to tell you what apptaner is but maybe you're not um so it's a container solution which was originally built for high performance computing by greg kurtzer when he was working at lawrence berkeley national lab several years ago uh it's really got a focus on security and ease of use for hpc users um this information is a little old it's got a really large and active community lots of installations um but really it's the linux uh foundation successor to the singularity container platform so if you if you're familiar with singularity i mean you know that's that's pretty much um sorry that's pretty much uh it's it's

the it's the linux foundation successor to that and you can learn more here obviously to do you seem kind of here's a relevant community today sorry go ahead zane no i was gonna say have you seen a growth in the community since it went into the linux foundation have you had more people join in or um you know i think it's actually kind of hard to say because the the community had already gotten so big um but you know i mean singularity became uh like like a ubiquitous tool which is used everywhere before it became you know pulled into the linux foundation umbrella and so

i mean at this point if the community grew it's already so huge it's a little bit difficult to track excellent thank you it's like saying you know have you seen more people use the copy command in linux you know it's like well it's a pretty widely used it makes sense so so what makes apptaner different so there's lots of different container runtimes right what what what differentiates Apptainer in this space i'm going to focus on three main things that i think kind of sum it up the first is it's got a simple um straightforward security model which is based on the idea that you're the

same uid and gid inside the container as you are outside and that whatever privilege privileges you have when you enter the container those are the same privileges you have inside the container you can't escalate your privileges um the second thing that really differentiates apptaner is it has a focus on integration over isolation so it intelligently integrates with the host operating system this is a little this is kind of unique within the container space i think and i'm going to go over some um some examples of that you know as i as i do the demo and um finally uh so the singularity image format or

sif this is a huge differentiator for apptaner um so essentially within and i'm going to show demos of this too but within uh the apptaner singularity ecosystem a container is just um is it's just a single file and that allows you to do all kinds of things from you know managing your containers easily to cryptographically signing your containers to encrypting them um and so on i'm going to go through some demos of that as well okay and uh you know once again i'm here representing ciq it's not just apptaner that ciq develops uh it's also you know rocky linux Warewulf um we have some things

cooking with uh something called fuzzball that we're gonna be releasing pretty soon so yeah just wanted to let everybody make it sure everybody's aware of that all right let me go ahead and just jump into a terminal and start playing around okay so the first things i wanted to show everybody is just give a quick kind of look and feel of you know what it looks like and feels like to use app painter okay so if you run help you get a description of what apptaner is i've already kind of given that to you you get some options here which are mostly there to just

um just uh control output to the screen here are here's really the the meat of the help section here are all the different sub commands that you can run with apptaner obviously there's lots of different things you can do apptaner is a very big full-featured program with lots of different options and that's its philosophy that's what it's supposed to do is be a very full-featured uh you know place to use containers i'm going to focus today on things like shell run exec i'm going to do a little building and then towards the end hopefully if there's time i'm going to show you some examples that

use the sign and verify commands so i'm not going to cover all the different things that apptanier can do all right um so just to you know make sure that everybody's level set i'm going to show you the the most trivial most easy example with linux containers so first i'm just going to show you i'm using rocky linux here because rocky linux is a really great really stable operating system right let's say for instance for some reason i wanted to swap out that operating system with something different uh you know this is the command um in apptainer to you to shell to open a new

shell in a container i'm going to grab that container from docker hub so um apptaner is perfectly happy to use uh oci containers from docker hub or from any other oci registry that you want to use and i'm going to get just the standard alpine um the official alpine container okay so i'm going to get a new shell great and so then once i've got that new shell i can do the same command that i just showed you and you can see that i'm now in alpine linux instead of being on rocky linux so i just swapped out my operating system with a single command

basically and that's that's the essence of containers and now once again if you know if i want to uh i'm just kind of giving you a look and feel of that painter right now if i wanted to do the same thing but i didn't want to bother with actually shelling into the container i can execute individual commands like so with this exec keyword and so now instead of shelling into the container and doing a cat ncos release apptaner will enter the container execute this command and then pop back out and give me the result since that's you know a little uh quick easy thing quick

easy way to do the same thing okay now that's that's neat but most people are not really interested in just swapping out their operating system with a new one for no good reason most people want to use containers because they have applications which they don't have installed on the host system or don't want to install in the host system for some reason and they want to have have quick and easy access to them right so i'm going to give you kind of a trivial example but you can see how this can scale up to more difficult examples i'm just going to run the python command

on my host system and it's going to tell me that python's not found on the host system uh now this example is trivial because it would be very easy for me to install this command if i wanted to but you can imagine maybe i want to run some you know complicated artificial intelligence workflow or maybe i wanted to do to run something that's got a lot of dependencies or a complicated um build tool chain that i don't have installed on my system or maybe it's some software that has it was written by you know a scientist in a lab that you know doesn't have any

packaging behind it so if if there's something like that that i need to run um and there happens to be a container uh available in a place like docker hub or some other registry um i can just go ahead and grab it and run it and so here's how so i'm going to get the default python container from docker hub shell into it with this single command okay now i'm in the container and now all sudden if i run python i have python simple as that okay so now i have to exit twice because once to get out of the python um environment and another

time to get out of the container i could do the same thing a little bit more easily with the exact command and then i'd have to only have to exit once you've already seen that i'm not going to show that to you again okay all right so um so that's that's kind of like a few little basics about uh about containers for the remainder of the um of the demo i'm going to have i'm going to cd into this demo directory and i'm going to be showing you these same files kind of over and over again i've got a directory here called pem which i'll

talk about later in these other two files let me show you something um that's a little bit different than what you might have seen if you're familiar with other container runtimes now job painter has this pull command i'm going to go ahead and pull once again that um alpine container from docker hub but when i pull it i actually end up with this new file and that's a look that might be a little bit different from what you're used to seeing with other container runtimes uh and this new file has a sif file a singularity image format file and now that that file kind of

is the container it encapsulates the container so i can do things like shell and shell into that file or you know use that file as a as an argument to shell and so now i'm in alpine using that sif file now i can move if if i decide that i don't like the name of that file for instance i can move it to another name i'm going to go ahead and do that and it still works but i have to give it the correct name so okay so i mean this is great because now if you can manage files which i hope you can do

you can manage your containers right i could copy this container up to an s3 bucket i could put it on a website and have people download it i could scp it to another system i could email it to somebody since it's so small um so this just makes this kind of like takes that layer of like having to rely on some program to manage your containers away and just lets you manage your containers by by yourself and that's just kind of the tip of the iceberg of what sif allows you to do i'm going to go through some of the other use cases of cif

towards the end of this talk all right so now um that i've kind of covered real real basic just sort of look and feel of apptaner i want to start getting into some more advanced um i want to start talking about those three things that i mentioned earlier in the talk that differentiate apptaner uh from other container runtimes in this space the first of those is it has a simple and sensible privilege escalation model and so i want to start getting into that a little bit all right so let's let's look and see some details about who i am on this system right i am

apptaineruser that is my username on this system and my uid is one two three four gid is the same and i belong to the groups attainer user and wheel okay um i'm gonna go ahead and shell into that alpine sif uh container again and i'm going to repeat and run id again and this is a major difference between um Apptainer and other container runtimes i'm the same user inside the container with the same uid same gid and i belong to the same groups okay so let's um let's look and see if i can elevate my privileges i'm going to try to run a pseudo command

i'm going to do sudo who am i that's going to say it you know doesn't it doesn't have sudo installed inside of it but that's that's fine i'm a little bit tricky and i have a container up on docker hub in which i've installed the sudo program so let me go ahead and run that instead okay i'm going to jump into this container and now i'm going to go ahead and try to elevate my privileges now you might have anticipated that that wasn't going to work because i already told you that whatever privileges you have on the host system those are the same privileges that

you have inside the container you can't elevate privileges once you enter the container but what's cool about this is that it doesn't give you the standard um you know this incident will be reported uh kind of error message it gives you something different it says hey the no new privileges flag is set uh which prevents sudo from running as root and if you're running in a container you might need to adjust the container configuration to disable the flag so what's happening under the hood here is apptaner starts a process which is responsible for spawning your container and that process when it starts that process it

starts it with the no new privs flag which is a kernel flag which allows the colonel to say i'm not going to allow you to elevate your privileges in this process or any of the child processes uh another uh kind of um guard rail that apptaner has in place um you know one of the most basic things about containers is when you enter a container you have a new file system you have a new root file system um the way that that works is that a new file system is mounted to your operating your your host file system and then the mount name space which

we'll talk about more later you pivot into a new mount namespace which allows the kernel to present that file system to you as though it were the root file system when apptainer does that operation it passes the nosqid option to the mount command which prevents you from being able to use suid programs like sudo within the container so the point of me telling you all this is that it's not like there's some you know complicated security privilege you know model which is kind of tacked on on top of apptaner to make everything secure instead apptaner just uses um you know the the kernel and the

file system features which have existed in linux forever and which are really well known and dependent upon to you know make sure that you can't escalate your privileges inside the container so it's really simple and straightforward all right um so let me show you one of the uh interesting side effects of this whole you know you are the same user inside and outside of the container um so i'm going to go ahead and i'm going to print my working directory this is a little surprising and i'm going to talk more about this in a minute for now just sort of accept that even though i'm

inside the container i have the same directory as i had outside of the container i have the same files too once again that might be a little bit of a mystery right now but i'm going to go ahead and uh touch actually i'm going to put some text into a file all right so i'm going to i just put the text bar into this file foo i cated it and notice that all these files even though i'm inside the container are owned by Apptainer user that's me i'm going to exit the container now and if i list the contents you can see that i just

created that file foo and it's owned by me it's not owned by root or nobody or some other you know weird entity that i have to chain and try to you know get this file back on by me and this is a really convenient and you know nice side effect of having you be the same user inside the container as you are outside of the container okay so that's kind of like a really quick overview of you know what it what the privilege escalation model and what the uid model looks like with apptaner what about that thing that i just showed you that i said

was kind of a mystery so i showed you that i had the same uh directory inside the container as i have outside and i was able to view the same files well um this moves into the second uh aspect of apptaner which differentiates it from other container platforms in the space and that second aspect is intelligent integration uh with the host operating system or integration over isolation um apptaner was designed to be used by scientists or others who want to run workflows and it's you know really basic if you want to run a workflow usually you want to do something with data right you usually

want to read some data into the container write some data out so it's an intelligent default to have specific directories bind mounted from the host operating system into the container by default and to make that the intelligent default and to make it so that the user has to opt out of that if they don't want that so by default the home directory is by mounted into the container at runtime uh current working directory temp bar temp and then some others like uh proxis um and dev which allow the container to do things like access devices on the host system and things like that um you

can turn this behavior off if you want to so you know if you uh you know wanted to do something like you know uh let's see if you wanted to use a contain or they contain all um options you know you could say i don't want all those uh directories to be mind mounted into the container at runtime or you can use the bind option to buy mount different containers or different uh directories into the container at runtime so that's all configurable but it's just about having an intelligent default okay can you edit the intelligent default yeah you can also change the default uh you

can do so within the configuration files in Apptainer um or if you wanted to you can set an environment variable so that you always have the same directories by mounted and you could even you know add that to something like your dot bash rc so that every time you you know you uh you um jump into your environment and start a container you always have the same files same directories by mounted into the container thank you okay so i want to talk a little bit about so there's another way in which apptaner there are several other ways in which apptaner intelligently integrates with the

host system and one of those is through namespaces so i kind of glossed over namespaces a little bit earlier um in the talk and i want to talk about them a little bit more detail so namespaces is are a feature of the linux kernel um and essentially all they are is there are some resources and the linux kernel can take and partition these resources out into different um spaces into different name spaces and then they can present those to the user in such a way that the user can only see you know whatever partition they're in and that's pretty much namespaces in a nutshell and

there's this is a kind of a list of all the different resources that the colonel can partition out namespaces this way um i already talked a little bit about the mount namespace so that's how you can get a new file system that's how you can see your file system as though it were the root file system there's others like the network namespace which allows you to partition off your network from the other network on the host system um the pit namespace which can give you a new pid table and you know so on and so forth by default most container runtimes enter most or all

of these namespaces because they're trying to achieve maximum isolation um this was one of the great things about apptaner though is um that uh you know the the the users that apptaner is trying to hit are are you know scientific users who want to use aptenergeron workflows and so because of that they might want to do things like use the network on the host system without providing some sort of you know specific configuration um or see the processes that are running inside the container on the host system so the only namespace that apptaner will enter by default is just the mount namespace and all the

rest of them continue to be shared with the host system unless you specify otherwise unless you specifically say otherwise and once again this is all configurable um and let me just let me just show you uh a quick quick little demonstration of why that's useful to a scientific user for instance all right so many many um scientists use python to program in and many of them use jupiter notebooks so i'm going to go ahead and start a container from docker hub that uses jupiter i'm in the container and then if i want to actually run that notebook um the command to do so is in

opt i'm going to start the notebook i'm going to start it running on port 8888 notice i didn't do any like port mapping from the container to the host system i just entered the container just as is um and i'm going to specify no browser because this container doesn't have a browser in it so i can't you know connect to it with a browser inside the container because this container doesn't have a browser so if i do that um you know the server starts the jupyter notebook server starts uh i can go ahead and just open this link now i'm on the host system i'm

outside of the container and i'm accessing jupiter notebook through the container through that 888 888 port um but i i didn't have to configure any any port mapping or anything like this um if you're a scientific user who just wants you know just wants your jupiter notebook or you know whatever your application is to work and just wants to be able to connect to it this is really easy and this is really helpful you can just use a container that has this in it and you don't have to provide any special configuration um if you're uh a developer creating um you know microservices then this

is actually kind of maybe the opposite of what you want right but the good news is this is still all configurable so you can still set up a network namespace you can still um you know set your port mapping up and do everything that you need to do it's just once again about having intelligent defaults to help the the types of users who are normally going to use apptaner um i'm just showing you this little this little python script here so you know once again i'm in that same directory that i've been in this whole time i've got this this is the actual container that

i pulled earlier this is that file that i just created and then i showed you this little python script that i have it's not really a script it's just a single command um and it just says this is executed by python inside the container i'm going to be talking more about that later i just want to kind of tease that i've got that script here okay that's fine while we're talking about intelligent integration with the host system um before we get away from that i want to go back to my name spaces man page and i want to talk about one very special namespace that

is the username space is special because it allows the kernel to map a uid on the on the host to a different uid within a namespace um and so that's that's kind of kind of an interesting and different thing that you can do um once again apptaner does not enter the username space uh by default but if you wanted to enter the username space you could do so like this however it turns out that you know most people who want to use the username space want to use it for one specific reason and that is to make themselves root inside the username space right you

want to map whatever uid you have to zero and that's that's usually what you want to do so app painter gives you a little shortcut that allows you to do that the fake route option um and if i run that now my uid is zero i root inside the container and so i want to show you how this works too because this is a little this is a little tricky actually okay so now if i print my current working directory this is all kind of a little bit of a fiction and i want to show this to you because it could be confusing at first

so if i print my current working directory um supposedly i am in root's home directory right but if i list the files that's fishy right i've got that demo directory that i showed you earlier now magically in roots home directory a cd into that uh i can see those same that same group of files that i keep showing you over and over again right so is this really roots directory but notice that they're all owned by root now everything is owned by root okay so i'm going to make another file i'm going to call it made by root i'm going to show you that that

actually worked it's owned by root right okay but now i'm going to exit and there's my made by root file which is actually made by me Apptainer user and that's all uh handled by the kernel that's you know just just username space is handled by the kernel so um that enables you to do you know things inside the container which would otherwise be impossible but it also prevents you from you know editing files on the host system which you know you shouldn't be able to edit okay good so now i want to talk about um this is good so i'm i'm going through this fairly

quickly and so i hope to have some time for questions towards the end um we do have one but i think you're going to cover it anyway so i haven't i haven't had it brought up yet well what is the question i want to make sure i'm going to cover it so it's actually asking about checkpoint restore is that supported in apptaner oh i'm not going to cover that actually okay well then i'm glad we brought it now yeah so that's not uh that's not part of um something that i was gonna cover in this demo um i don't know if i'm allowed to say

it so you might have to redact this from the uh from the from the live video but is this on a delay um i think that there's a rumor that we're going to start pretty soon with an uh apptaner education series and um if we do that which you know is just a rumor at this point in time i think we're going to be delving a little bit more deeply into specific features like that and it may be that the checkpoint and restart workflow which was recently added to apptaner there's a rumor that maybe that might be one of the first topics that we go

through in detail um but for now i mean you didn't hear that from me uh and i'm sure you know you better get this information out there before it's deleted off the internet um need to be confirmed now i will say that that that uh functionality has recently been added to apptaner there is a checkpoint and restarting uh workflow that can be carried out in Apptainer and um it's not part of my uh demo to go through that today but yeah it's pretty cool stuff okay um is there anything else before we jump into the the final aspect of apptaner which differentiates it from others no

not yet okay i mean this isn't my opinion the final the biggest thing that differentiates Apptainer from other container runtimes and that is sif the singularity image format all right so to start this part of the demo off i want to start by building a container um so you know i've been showing you these files over and over again but i'm not actually showing you what they do and anything about them so let's let's look a little bit at some of these i showed you the pi script dot pi uh file earlier let's look at this python definition file so a definition file is to

singularity what a docker file is to docker or pod man or whatever i guess build uh technically um so this is the way in which you you instruct apptaner or singularity to build your your container um in this uh apptanor definition file i am using a bootstrap agent called library a library is a registry that stores sif files natively basically instead of storing oci so this is like uh the sif version of you know the docker docker hub um and you can have different different libraries i have one you know pre-configured that i'm going to be using here but you know you can point to

different libraries i'm going to be pulling this container called god love d secure um in boontube ionic and so that's a kind of an old version of ubuntu and i'm going to be using this fingerprints keyword that i'm just going to ignore for now and i'll tell you more about that later so just forget about that for now the post section is a scriptlet um that once you download that container the build system is going to pop into the container and it's going to execute whatever you've got here so i'm just going to do an update and then i'm going to install python and then

the run script this is that piece of metadata that tells the container what to do when you run it i didn't run a container earlier but i'm going to talk a little bit more about that here in a minute and uh so we'll see what that looks like basically whenever you run which is a command and apptaner whenever you run the container this is what it does in this case it's just going to call python um and it's just going to hand python with this funny little syntax whatever the command line is that i give to apptaner it's going to go ahead and hand

that right onto python okay so let's see what that looks like let's go ahead and build it first i'm going to do appear build python.sif from dot python.deff and it says whoa you can't do that you're not root so i could uh become root and do that but that's not a great thing to do for various reasons so instead i'm going to use this nice little uh fake root um option that i showed you earlier to pivot into the container uh and become root in a new username space inside the container okay we do have one question if you don't mind what that is downloading

uh fake recovery user user space and secure host root name space um not sure if i understand the question um so what fake root does is it leverages the username space which is um a kernel feature which allows the kernel to map the uid and gid on the host system to a new one uh in the new name space and because um because the kernel is involved here the colonel is able to uh to keep straight you know what the context is so if you try to access files um even if you're in the username space which are really owned by root on the

host system you're still not able to access them right you can only um you can only access files which exist within the context of the container and not on the on the host system does that answer the question so it still does secure the host um it doesn't allow you to access the the root um file system on the host but it allows you to access the root file system within the container um you know and sort of pretend to be rude essentially i think it does dave if it doesn't if you want to add more context and ask it a little bit deeper maybe

but if that answers your question great thank you for the question thanks dave yep cool and so now i've built this container python.cif um and so i can do i didn't show you this before but i can do this uh this run command python or applicator run and if i run it it's going to do whatever was there in the run script okay so what was there in the run script was to run python basically but that's cumbersome to you know use that entire command if you've been paying close attention you might have seen that these sif files are highlighted green and that's because um

they have the executables bit set so you can actually execute these guys this is one of the cool things that you can do with sif so if i just run it like so it's going to give me python and i can pretend now that this sif file is just an executable sitting on my host system like any other executable because i gave it remember that kind of awkward syntax to just pass the command line from apptaner directly into it i can also do things like that and they work which is pretty cool and furthermore um i can give it the script that i teased a

little bit earlier i can just pass the script directly to that invocation of my python sif and it's going to go ahead and work just as though it should so that is a very powerful thing that you can do um just by encapsulating the container in a single file um you can see how you could leverage this to install programs uh on behalf of users for instance and then let your users run the containerized programs as if they were just programs right on the the host system um and that's that's something i think that a lot of people actually do it's very cool okay now

there's two more things i want to cover but um is there any is there any questions about that before i go further on nope continue on okay two more things i want to cover which make this the sif format just really great um there's two questions that we're going to answer uh with the sif format the first of those is how do you trust containers and how do you run containers that you've downloaded from potentially untrusted sources okay and then the second question is how do you run containers in untrusted environments okay so the first question how do you trust containers that you've downloaded from

untrusted sources you do that by cryptographically signing and verifying your containers i'm going to show you a little workflow to uh get you acclimated to that okay so instead of explaining this one first i'm just going to kind of throw it at you and hopefully you can you know kind of kind of see what i'm doing and get an idea of what's going on here i'm going to create a new key with apptaner this is part of apptaner enter my name my email address my real ciq email address if you want to talk about that painter or whatever let's go for it um this is

some demo key interrupt passphrase it's i just i just generated a key pair and as a convenience um that workflow is is part of apptaner you can see so it's it's one of the commands within apptana okay so i'm going to go ahead and list my keys i actually have two keys on this system there's one i created a while ago when i was first putting this together and then the one that i just created a second ago um i'm gonna go ahead and sign let's say the python container that i just created um really in actuality if this was production i wouldn't sign this

container um because well actually this one would be okay to sign because i started this container from uh from uh from something i trust and i'll show you why later but i'm going to go ahead and sign this python container it's going to ask me which key do you want to use i'm going to tell it the one that i just used that i just created they asked me for passphrase now it's signed and now if i want to verify that the container has actually been signed i can do so with the verification command that's going to use the public key material along with the

container itself to tell me basically that this container is a bit forbid reproduction of the original one that was signed by the entity that had this fingerprint and that fingerprint's very important that fingerprint is really the way in which you know that this container was signed not just by anybody but by the same entity you know um that originally signed it the one that you are interested in knowing about okay who cares right i just i just signed um and verified this container on this the same system of course it didn't change you know of course it's the same well why is this important okay

i used to work at the national institutes of health uh in the high performance computing center and one of the things that i did while i was there is i created a whole library of containers that i vouched for and i said nih staff can use these containers as starting points and can be sure that there's no malicious code in them and nih users can also use these containers and be sure there's no malicious code in them and then i went and i stuck them up on a public library that nih doesn't control um so how can i be sure you know and how can

anybody be sure when they download those containers that they haven't been tampered with that that library hasn't been um hacked that somebody hasn't come in and replaced all those containers with you know their own containers that want to buy bitcoin or you know do whatever it is that they wanted that they want to do um so let me go ahead and download one of those containers okay so this is uh this is once again it's in my secure collection it is um it's an alpine container i'm going to call this container secure alpine okay so i've downloaded this i'm downloading this container and now i've

got this container you know how the heck do i know that this is a this this is an okay container to use and the way that i know is i can verify it and it says that this thing was signed by david godlov using a production key a million years ago this is my um this is my gmail address and this fingerprint is very very important i happen to know that this is the fingerprint that i originally used to sign this so now i know that this container even though it's been sitting out in the wild somewhere and i downloaded it and i don't know

who's messed with it since then i know that this container is a bit for bit exact reproduction of the original one that i made you know some odd years ago um now what else can i do i can go ahead and resign this same container and ask me which key do you want to use i'm going to use the one that i just created again oops i didn't enter the password correctly good to know that works um and now if i verify the container again i've got two signatures on it so you can stack signatures one on top of another like this so this is

really cool because now you can take you can create a container and you can say okay my um software development team signs off on this container my qa team signs off my security team signs off and i signed off and i'm not going to run this container unless i have all four of those signatures or however many signatures you know i want i told you earlier that i was going to gloss over this fingerprints command in the python definition uh file that i made earlier well that's what that fingerprints command is doing so you can put that uh that fingerprints word in your header of

your definition files and now what this will do is if it downloads this container and it finds that this container has been tampered with since it was first created it's going to bomb out and it's not going to do any of the rest of this definition file it's not going to use this container so yeah you can totally just throw this container away not use it if it's been tampered with there's also configuration files an app here that you can use to um blacklist or whitelist which you know white listing is probably more useful in this particular context but you can basically say i'm not

going to run any containers on this entire system unless they've been signed by you know the following 10 entities and if i know that those fingerprints are all there everybody's signed off all the teams are good then i'll run those containers so it's a pretty powerful thing you can do and so that's how you answer the question of how do you trust containers downloaded from potentially untrusted places now how do you run containers in untrusted environments and the answer that question is you encrypt your containers so that nobody can look at them and see what's running on them okay so i'm going to show you

that workflow really quick um for this you actually do have to be root uh this is something that we're working on changing right now but for this currently you still have to be rude well let me hold on let me show you these files again there's this mysterious pem directory that i've been showing you over and over again but now i'm going to actually use it so uh what that has is it has a couple of rsa keys a public and private rsa key which uh are stored in the pem format and i'm going to use those you can actually just use a password if

you want but it's not very secure i'm going to use those rsa keys to encrypt and then decrypt my container okay so i'm supplying a pen path i'm going to use the uh let's see the public key material to encrypt my container and then i'll use the private uh key material to decrypt it later i'm gonna call this encrypt.sif and i'm gonna build it from the alpine so that's another thing that's pretty cool that you can do um since your containers are just files you can actually build from the files which are already on your system and when you do that you're essentially just converting

the container from one format to another it's going to take a few seconds to do even though it's a very small file system because it's got to actually encrypt the file system okay so now we're encrypted and so now i'm going to do apptainer i'm going to run true and it's going to say wow you can't do that because that's encrypted okay so let me supply the pen path uh it's gonna be the private key material because it's the secret it's actually gonna let me run it i do that it's going to run it runs it ran silently because the command that i ran was

just true and so now i can just to make sure that actually did what it was supposed to do i can echo the environment variable which tells me the exit code of the last process and it was zero so it it worked out and did what it was supposed to do so that answers the question of how can you um run a container in a potentially untrusted environment uh it's worth noting that so obviously this is just a file so when you encrypt it um the file system is encrypted sitting on disk it's also obviously encrypted in transit because if you move it from one

place to another you're just moving the file from one place to another so there's no need to decrypt it um it but it's actually even the file itself is actually even still decrypted or encrypted even when you run it which is pretty cool so there's no need to decrypt it on disk and then you know pop into the file system or whatever where somebody now can go look on disk and see what the file system is it's all decrypted in memory so that's you know a pretty cool feature okay so this was just a really high level um kind of teaser look and feel and

what are some of the key features uh within apptaner um once again there's a rumor that we might be going through some of the features of Apptainer in more depth in some of the future webinars but i can't confirm or deny that uh so i guess yeah there's a few minutes left if anybody has any more questions i'll do my best to try to answer them we don't have any at the moment thinking wait a minute maybe i missed it when you said it dave but you can have you can have a container signed and encrypted i think you said absolutely that is the case

yeah so i've just encrypted um this inc.sif so let me just go ahead and sign that as well okay so now i've got assigned and encrypted cool very cool and yeah and it's worth noting too um there's sometimes some confusion uh about encryption versus cryptographically signing and so sometimes people sort of conflate those two i think that a good way to tell those two apart is to think about them as doing the opposite things so when you sign your container you're signing it with your private secret so both of these are asymmetric right when you sign your container you're signing it with your private secret

which is private to you and then when you verify it you use your public uh key material which anybody can have right and that means that anybody can verify your container because it's just all they need is the public key material to verify it encryption works the opposite direction when you encrypt your container you do that with the public key material right but when you decrypt it you do it with the private your secret that only you have because only you want to be able to decrypt the container afterwards so there's similar technologies but they do basically the opposite things from each other that's the

way i keep it thank you very much thank you yeah i don't see that we have any more questions um look again no all right well just one thing you don't have anymore not bad sorry real quick um so if we don't have any more questions i don't know if i've ever completed a demo without running um this container i just want to make sure that i run it once uh there we go there you go perfect that's great dave i really appreciate you going on and doing this that's fantastic uh guys we really appreciate you joining us today and we look forward to talking

to you again at some point in the future if we have a series that is rumored to be or not to be really appreciate it guys go like and subscribe we will see you again next week thank you thanks