Apptainer: Securing Containers with Signing

you have a key it's a an asymmetric key so you have a public and a private component and then through some out-of-band mechanism you exchange your public keys with other people that you want to correspond with um and then you can submit uh assertions of trust into some kind of key server um and there are a few but we'll be dealing with kind of the most standard uh i think it's just openpgp.org or something like that for for apptainer um and so through your relationship with other people you can say i have trusted this person i know that i have their key but you can

also trust that other keys that they have verified are also likely to be the right person or in this case the the right author of a container or the right organization but there are other ways to get keys as well you could imagine uh downloading a pgp public key from an https protected website from a company or something like that and then from that point you can verify that containers that you've pulled down are valid from the point of their publication so having you talk about keys and keys from here keys from there so obviously a lot of different keys could be involved how do

you deal with a bunch of different keys is there somewhere that they live how do you get them where do you put them yeah so generally speaking there's some kind of database on your local system uh in what's generally called a key ring but then the public key material you can just upload because it's public to any random key server that you want and then you can use those key servers to pull down other public keys as well but locally on your system you'll have a key ring of public keys of other people and your own that you trust to various degrees or with various

degrees of certainty um and then a private key ring that stores the the secret material for keys that you own that you can sign with and there there's a key ring in like i have the the pgp keychain app installed for mac os but then apptainer has a key ring also that you um that it will use by default both for verification and signature thank you for that so i know you're going to show us a little bit of this but from from a user perspective maybe i haven't done this before and give me an example of when this would make sense or why and

i'll open this up to everybody i mean greg i know you have thoughts on this as well but tell me tell me something when and why um so from my perspective the the point of containers in general is or one of the biggest points is portability and the ability to kind of publish a workflow and then other people benefit from that workflow and so there's this kind of typical uh other kind of workflow a typical way that people work of you know you develop a container and then you push it up into a container registry or an artifact registry of some some kind those may

be private registries and then maybe the signatures aren't as important because you have to authenticate to the registry but lots of these registries are public and you know with with standard web stuff we we decided to protect that uh you know we as kind of the general internet community protect that through uh tls signatures x 509 signatures on https websites where you can verify the identity of the the server that's that's serving that material to you um pgp signatures on a container are a way to do that uh in this other space maybe you trust the website or the the container registry that you pulled

a container down from but now that container is just a file that's sitting on a file system floating around who knows where it came from uh one of the benefits of the apptainer mechanism for uh for signatures is particularly with with sif container formatted files is that the uh the certificate is bundled with a single file so there's no separate signature file that you have to manage or keep track of it's just if you have a container and it is signed you can verify that um that it is signed by the entity you expect or someone else no it's interesting it's all there together in

one file i'm not having to log around a bunch of different things and i hope that you'll show us how we can see that here in a minute so what i can yeah i can add in a little bit to that so uh early on as is you know containers were starting to really gain in popularity uh and and you know we were just starting with singularity at that point we were just starting to integrate with you know the the registries that docker had up there you know before oci was a thing it was just docker hub and we were basically pulling containers down from

that interface uh the questions started coming up over and over again which is right we don't just go and install random rpms on our system uh that are not cryptographically signed that we don't trust right we may be pulling packages off of various mirrors for updates but they all contain a signature and each one of those signatures can be verified by the material the the public key material that we have on a particular system so it doesn't matter where we get an update from what mirror it came from we can validate the authenticity of that package and that update through this this asymmetrical uh key

method so but what are we doing with with containers right we're in many cases we're downloading containers that we we don't know where they came from we don't have clear visibility on who has touched them what is the provenance of that data and the material within those containers and um and then worse we're actually executing those on giant computers giant super computers in some cases um and in some cases even worse we're even doing that as root it's like yeah okay hold on let's back up a little bit we got we got kind of a major security vector here going on that we have to

be thinking about so the idea of this was how do we trust how do we guarantee the trust of any of those those containers and all of the material within those containers in such a way that we feel like we can we can leverage that we feel like we can we can we can trust that not only is it can we guarantee you know kind of who it came from and where it came from but we can also guarantee that it has not been tampered with uh since then so these are really good reasons why i think everybody should be thinking about you know cryptographic

trust and validation of anything that you're going to be running on your computer or leveraging um and it's just to keep keep going on this for just a second it's not just linux i mean if anyone has a mac and you go to try to down an apple download an application you're going to have to and it is not cryptographically verified by apple as going through the trusted mechanism you have to jump through some hoops just to get that thing to work right this is this is a much simpler interface and it works extraordinarily well and it integrates with standard mechanisms that we're currently using

within the industry for dealing with uh key material everything from standard pki to standard hkp protocols etc etc so um that's kind of some of the background there i just wanted to throw in no that's great and i appreciate it one of the other things that i wanted to point out i know jonathan you kind of spoke about this last night is we're talking about cryptographic signature we're not talking about encryption that's a completely different topic and and different conversation to be had but right now we're talking about cryptographically signing a container right that that's correct and apptaner does support uh encryption of containers at

rest and in during execution uh but it's it's handled through a completely different mechanism so i expect at some point we'll have a demonstration about that but for today uh we're gonna keep it simple and just talk about pgp signing very good i just want to make sure that we clarify that before we get started here so uh i guess go ahead and go ahead and start showing us the generation in exchange yeah great um so i've just got a terminal open here um with a very simple container this is just a hello world container that i i threw together real quick it's based on

the the alpine base package or base container um uh mostly this is all just so it'll be very small and easy for us to oh sorry i'm getting some disturbance around the share um it's really small and easy to uh um to upload and download so uh quickly for the demo so the first thing i'll do is build this container let's just uh one thing to note is that because apptaner supports this fake root mechanism uh i can do this just as my regular user uh so i'll end up in just a couple moments with a sif file on the other end of this and

uh here we have a really basic zip file it's it's built out of parts and and we'll learn a little bit more about that in a moment um but mostly we can see that it's got the squash fest image that's the majority of our our file system and then a couple other things here that that are part of the container but we can't verify it so if i say apptane or verify that sift that we got it has no signature so there's nothing to verify so we can here within apptaner and there are other ways to bring keys in in fact i'll show a little

bit of that in a moment but we can generate a new key right here within apptaner and use that to sign our container so that's a painter key new pair we'll fill in some metadata and this can take a moment uh but it's pretty quick part of the issue here is that i've got this running on a relatively small vm so it doesn't have a lot of cpu power available to it but it doesn't take too long here to generate a new key seems very similar to creating ssh keys yeah except for we don't have enough as much of a progress indicator for some reason

there we go so with our new key i mentioned that app painter has a key ring in it we can see the keys that are in our apptainer key ring with this key list command and we'll see that we have two so key number zero here uh is my real uh pgp key and we'll use that in a moment um but key number one here uh is the one that we just generated it's our our demo key it has a fingerprint or the public key has a fingerprint that we can use to identify it with key servers and each other sometimes you use that fingerprint

to reference this key to apptaner sometimes you use the index but it's pretty clear from context which one is appropriate so from here with our new key we can just say sign our our new sif file it asks us which private key to use it's asking us to specify by index so we can just tell it to use key number one our demo key and because we're using the private key of our key pair to sign we have to enter our passphrase you in most cases should have some kind of passphrase that protects the private key material again because this is all about trust so

we want to protect that private key so that when people receive a container that has a signature on it from this key uh they trust us as well as they expect us to uh maintain security of our private key as well or our our passphrase around that private key so that way the private key is never just sitting decrypted on disk uh in case you know there's a security incident on the local workstation that we're working on let's say so now that we've signed it and verify our container uh we see that our verification happened locally we're doing this with the local key in our

local uh key ring we see which uh which key has signed it and it gives us fingerprint here and then shows us uh there are ways that you can sign individual portions of the key and there are sorry individual portions of a container and there are reasons for that that we can go into although uh it might be more of a discussion at that point but here we have our entire container uh verified uh with that key uh so the next thing we can do uh is show that uh oh actually there was one more thing um if we list our container again we'll see

that we've grown another object this here is the signature itself and it exists outside of this group one and is signing all of group one got a copy paste error here so it's it's signing uh the group one set of objects in this container and the next thing we can do then is sign it again if we so wanted to and this time we'll use the zerowith identity which is again a real one that i i maintain and imported in out of band from the gpg keychain app so now we've signed it again and if we list our container we see we have a second

signature here and both of those verify in this case they're both in my local key ring again so here we have our first signature with the demo that we had before and our second signature uh from the second key so jonathan on that if i had one of those public keys could i could i execute it then or do i need both so container execution is a different uh a different question so by default apptainer will allow you to to execute containers regardless of whether or not you've been able to verify the signature you can define policy in the configuration for requiring that the containers

that you execute have uh one or more signatures so that's that's part of the uh execution control list i think ecl and in there you can say for containers in this path uh it must have one of this list of signatures or it must have all of this list of signatures or it may not have any of this list of signatures and control execution that way right now in the verification logic or the way that that apptaner verifies containers it just verifies the signatures one at a time and so something we'll see in a moment is if i didn't have this uh if i didn't

have this key in my local key ring and it wasn't available through a server then verification would fail and it wouldn't go on to verifying this and it would just say that verification has failed even though we may have that second signature i think that that's uh a little bit of an opportunity for for change uh in the interface but there's nothing about the cryptographic signature that necessitates that that's just a specific eccentricity of the way that aptaner handles multiple signatures right now but in theory you should be able to verify that a container has been signed by any of these keys regardless of their

order or how many of them okay thank you so now that we have this we have kind of everything we need to pass this container on to someone else so the first thing i'll do is push this container up to a key server that we're using internally we'll use it for our purposes of our demo here this effectively just uploads this container up i need to log in first effectively just placing this container up on a server uh that forest will then be able to access uh and pull this container down from and then once that's done we're going to demonstrate two mechanisms for uh

container or for key exchange uh one is passing around your pr uh public key out of band i'm actually going to download the public key off of this system and pass it to forest over slack just it's a file and i'll just hand it to him by dragging it over there but then we're also going to push my my regular public key up to a key server and this this key has been pushed there before but it's it's an item potent process we can do it repeatedly with the same result and if i had added let's say email addresses to this key it would just

update the record on the key server but we can do that now and then we can export another key here i'm i'm specifying an ascii armored key but that's that's only to make it a little bit more palatable if we were copying and pasting the key around in plain text we're going to export a key into a file and so here i'm going to export key number one which is that demo key that we generated earlier that gives me this demo key.pub file which you won't be able to see this right now but i'm just going to copy that off this server onto my local

system and just drag it over to forest cool super secure slack communication yeah private keys it's a pre uh a pre-negotiated trusted communications mechanism and so and with that i'm i'm basically done at this point so um i'm gonna copy this over because i'm uh one more time because i may have gotten it confused with a a version of this key that we were doing before in test but forest now has a copy that key and and should be able to take it from here and you should be able to actually share the whole container over over slack too now in theory this container for

sure would be fine that way because it's it's not very big but containers can get large and and perhaps i wouldn't want to download and drag a multi-gigabyte container over to forrest over slack i think they might not like that either not if you like your slack messages from prior to 10 minutes ago yeah all of a sudden my slack bill is going to go up a lot you guys seen people sharing containers over everything from google drive to dropbox to um ftp i mean of course scp-ing them around as well but having the file the whole container in a single file makes it really

easy just to deal with and that was always one of the goals going back to early singularity days and thus the name singularity uh single file so uh yeah all going all the way back since then it was always kind of focused on the single file methodology and and ideology uh and and to the point that was raised earlier regarding the cryptographic um validation what's really nice about this is again we're handling this very similar to how like rpm would handle this so there's no uh external services that are necessary to validate against uh everything is contained within the actual uh image file so it

makes it very easy to deal with yeah i wish we could actually get a show of hands live of how many people are using ftp i interacted with one recently for some genomics data that i was pulling down um so they are still out there yep and in recent past life i had to build a multi-server ftp service for someone exactly awesome i believe there are numerous numerous labs still using uh ftp as the art as the interface to their archive storage system yep exactly that yeah all right forest cool um i just want to note first off before i jump into this that when

uh jonathan you sent me that key snippet over i got a little message on slack saying that you'd send a pgp snip like it specifically recognized that it was a pgp public key snippet and also as much in the notifications so apparently uh they've planned for it on their end um but yeah so you can see that uh so so yeah we'll jump in here to um my component of this let me just copy the container url that i'm going to be using here oh yeah it's the same one from before of course but yeah cool so yeah so i'm just hearing an empty directory

you can see that i've just made this so we can have a place to pull that container down into and pull that um well the key i'm probably just going to download into downloads but um just to have a place that we can do this demo from so first things first i'm going to go ahead and get myself a copy of that container so i'll go ahead and do an obtainer pull and then provide that same url that jonathan just pushed that container to so we'll go ahead and do that you'll see we get this message saying that we're downloading the image if i go

ahead and do an ls you'll see that it's there that was very quick you can see that i now have that same container jonathan just built on his machine over here on my machine if i do an obtainer sif list just to point this out you can see that we have the same components inside of this that we had before just that definition file this was built from a couple of json bits here that include some metadata that was built into this file when it was created um so like the environment the labels stuff like that we then have that squash fs file system that

contains uh you know the container file system proper and then those two signatures that jonathan signed this with so you can see this container comes down on my side obviously still with those signatures if i go ahead and do obtainer verify however actually i think that might be obtainer obtainer verify it is so if i do obtain or verify and then i provide this you'll see that this fails to verify as jonathan had mentioned um even though one of those keys is out there on a public key server because i haven't imported his um uh that local key that he created here into my own

key ring the result that we get is uh it's saying that this fails to validate um because as i mentioned it can only validate one of those keys from that internet location i still need to import that other one here locally so i'll go ahead and download that key that jonathan sent me which is right here just give me a moment so i'll go ahead and download that you can see i can do an ls latr home slash downloads do grep demo key you can see we have that key there i'm actually just going to go ahead and move that over into here as you

can see we have the key and the container file here i'll go ahead and actually import this key into the key ring here so that app painter can recognize it so go ahead and do abstainer key import and then just provide the path to that key file you'll see that we get the message here that the key with that fingerprint has been successfully added to the public key ring and if i go ahead and rerun the well actually first off i'll go ahead and do obtainer key list and you can see the two keys that obtainer knows about here um one of these is basically

just a test key that i put in here for something else uh long lost a lifetime ago um and you can see the other one here was just created and imported right now so if i go ahead and do this uh obtain or verify on this container again what we'll see is that we should instead this time get a message confirming that all of the keys on it can be verified um because once again i've imported that local key into my key ring here on my laptop and then the other key that this assigned with is sitting at that open pgp key store an obtainer

will reach out and verify it from there so we'll go ahead and run this and you'll see we get these two outputs here indicating that we have a local key being verified we have a remote key being verified so this is the one i imported this is the one that came from uh openpgp and you can see that we have a listing of the objects that we verified here so once again we can see those same objects that we had in there prior to this uh and we can now tell that those are verified so if i just want to go ahead and show i

can run this container you'll see that i get the quick little hello world message out of it um but yeah that is container signing so outstanding thank you forrest one of the other things i wanted you to talk about jonathan is the format of a sif i know whenever you do the obtainer sift list it kind of shows you the different pieces of what's inside that file you talk us through what that is a little bit and then talk about the groupings and kind of how we can use that yeah i can um run through that really quick director johnson but i'll run through that

real quick uh let me uh stop sharing and share this other screen that i have here so this right here is kind of the standard image that we use to express exactly what is inside of a sif and exactly kind of what that internal format looks like and you can kind of see how we represent the ability to both evolve a container after it's created in a verifiable way as well as the ability to cryptographically sign the entire container itself so as i kind of touched on there we when we looked at that sift list we had four different components in there we had a

dev file two json files and then that squash fs file system inside of a sif you can see this image describes the structure of it so i'll kind of map what we saw there on the command line to what we see in this image the global header and the descriptors we didn't really have a ton of visibility into there those are kind of more i believe metadata that you would have to go into you know a hex dump or something like that probably to take a closer look into um but for these uh other components here we can see recipe definition and labels and environment

um those map to the three different uh components that we saw outside of that file system when we did sif list um so that one that mentioned that was the def file um is actually the recipe um that the container itself was built from um the labels and the environment are bits of metadata that can be added during container um uh their metadata can be added during the container build so labels i believe jonathan may have shown briefly at the start of this what that container definition was built from and i believe he put like his name as the author in there as a label

so labels are just metadata that comes with the container that kind of describes certain user customizable aspects of it the environment is the actual uh you know environmental variable setup that is present with that container um so that's going to be like the path the ld library path custom environment variables you've set things like that will all be contained within that portion of the stiff the immutable runtime container image is the squash fs of the actual file system that contains everything the container is meant to contain is within this becomes immutable because fest is a compressed file system so you can't really alter that meaningfully

after the container is built and then obviously you know signing gives you another layer of verification that nothing has changed within that container but that immutable runtime container image when you go ahead and build one of these apptainers it pulls down the base image that the container is going to be built off of and then um you know within the post section it goes ahead and runs any of the commands that you put in there so you know dnf installs compiling software doing downloads of different things that type of stuff all of that is run and put into the container file system and then when

the container build is done the last step is that it takes that file system and basically you know make squash fs with it um and you get this uh compressed um file system image basically that is uh that represents everything the container was meant to contain on like a software kind of application level um basically the file system level um so that's in there as well you'll notice that by default when we looked at this at the start there wasn't a signature block there that's because obviously there's not a generic signature that goes with every container that is only shown and present there if the

um that's only shown in present nervous signature is actually uh attached to the container and so up here at the top with cryptographically signed um we can kind of see what we had represented when we did um that verify with the key and how it showed we had all four of those components the def file the two json files and then the squash fs there you can see how when we ran obtainer verify we got that output out that confirmed for us that those four um components were all verified as a part of that verification operation because they're all signed with that key and so

that's where you can kind of see this concept of the signature allows for the cryptographic signing of everything else within the container and while it's not specifically the focus of this demo um just to explain what we mean by evolvable here within obtainer you can add a x3 formatted overlay image essentially to a built container that adds a writable layer to that container these are typically immutable as i mentioned the squash fs isn't really meaningfully modifiable because it's a compressed file system so you can't really you know get in there and do anything with that but if you wanted to have some type of writable

space for a container like for example you have some type of workload a high performance computing workload represented within a container that you want to have outputs written to a place that they're also packed with that container you can add a writable overlay whether that be as an x3 image or a directory on your file system that the container can use for reading and writing or not reading writing sorry they can use to write to um this kind of touches on some of the uh optimizations that obtainer provides kind of surrounding uh how it stores data with inside of it um just to give the

typical use case if you're running uh you know some type of high performance computing system with uh you know thousands of processes starting up all of those doing you know small file i o python imports that type of thing um your uh file system can bog down with doing all of those uh different small metadata operations but instead if you're running out of a container and all those metadata operations across your shared file system on your compute cluster if instead all those metadata operations are just directed to the squash fs inside of the container and it basically just becomes reading offsets into that squash fs

these read operations happen much much faster and for the same reason if you're using an x3 image overlay the writing happens a lot faster because instead of metadata operations on actual file system it becomes just reading offsets into an overlay image basically um so that's a bit of a long-winded explanation but that's what's inside of a sif that's what can be inside of a sif as well if you make certain modifications to it but you can see how the format allows for as you know we've touched on in various capabilities um a lot of very interesting features that are good for hpc um and yeah

it kind of contributes to why we do things with obtainer and not other uh containerization platforms within the context of a high-performance computing environment that's fantastic for us and it brings up an interesting point i know greg you talked about this yesterday and being able to have a real-world example of being able to do ai training into a container like that could you tell us kind of tell us that story about what that really means having having the ability to have that rightable space inside the container sure is that for me or forrest i asked you but i don't i don't care of course wants

to tell it jonathan anybody i think what we're um what we're going over here is kind of a use case we discussed yesterday for the ability to selectively um sign different components inside of a container you may have noticed that when we were doing all those verifications and lists and stuff there was in addition to kind of a component id for each for that dev file those two json files that squash fs there was also a group id that was associated with those and you'll notice that when we were looking at those signatures it says that um as jonathan pointed out it shows that we've

signed with that key the objects in group one this kind of leads to an interesting use case where if you have for example evolving data that you want to move with a container um like for example you have a machine learning use case that you need to have um data constantly being or new data constantly being fed into um like maybe you're developing your model you're developing your data set you want to be able to say easily pass all that code and all that data around people in your research group as that data still develops and you know those data sets kind of take shape

something that you could do in order to provide verification there is sign all of those group 1 objects inside of that container so assign the actual base image the definition stuff like that but then add an overlay into a different group inside of that container that is then unsigned but remains mutable as a result and so you can have the base code in the base container that you know say your model code and stuff is stored in being immutable and being verified and signed by people in your research group but you can actively still modify that overlay that contains say you know the images and

the data that your model is supposed to run over you can still easily modify add to that remove from it whatever you need as your kind of whole code and project take shape in a way that as i mentioned keeps the original stuff still verified but provides access to editing that i just want to mention as well that my artistic abilities in google slides have come a long way since then i'm actually uh that that image it's funny it brings up memories for us it's it's an older image i created it for i think it was super computing uh presentation uh i know yannick committed

it to the to the repo but yeah that's that's that's my old artiste work right there um the image is actually based also on an elf binary image so it lends itself to a lot of functionality in terms of being able to extend and do different types of things with that image and something we didn't mention but we've mentioned before is that the image itself is executable so if you have a singularity file container and you've specified a run script you can actually execute that container like a binary you can put it in your path and it will actually execute the the stack within the

container um like it would any other program so it's it is super cool a lot of functionality with regards to that and it's quite extensible jonathan i think you came off muted but i didn't stop blabbing no i i i didn't know how far you were going to go into that but i that was one of the first really aha surprise moments for me was seeing precisely what you just said that the because of the use of the an extension of the elf format you can just dot slash your container and it runs that's pretty cool that's very cool i think we had a couple

of questions that popped up i think justin was the first one so why is there a difference between a local and a remote key so in general this is just the reality of of the exercise that it's just reporting whether the key is stored locally on the computer or was pulled down from a key server and apptander by default right now is configured to use openpgp's kind of general public key server so remote keys are coming from there but you may want to know when you're verifying something where that key is you can pull that key down from the key server i think it's obtainer

key pull and then the fingerprint and and it will be copied into your local key ring and then it would be reported as a local key and in general you could imagine you know it's just public key material it should there shouldn't be a barrier to uploading that key to any key server and so why does it matter like what but even in this scenario like i passed our demo key to forest out of band because i don't want to just throw random uh public keys with an email address for no that are never to be used again up into the key server so we

have that flexibility whether it's for some privacy reason maybe you don't want that key up on a public key server or just for convenience you you just want to be testing something uh to work just with local keys similarly in general those public key servers are right only and or you know you write to them and the keys never go away they can be revoked they can be marked as inactive but they they are never deleted um but it's still an external service that you're dependent on so if you have a production level service that depends on the ability to verify a key you probably

want to pull that key locally and be verifying it with your local copy rather than expecting to always be able to go out to the internet and pull that key down off the the remote service so it's it's just the the specifics of where the key happens to be located right now and then reporting that to you because it may matter in some circumstances just to just to follow up on the last point that you just mentioned um it is important i think to know because not all levels of trust are equal so let me give an example of that uh the key that you

sent to forest over slack right you could have been not just sending it over slack but you could have also been doing a verbal verification of that fingerprint with forest specifically so typically keys that are locally installed are ones that people consider most trusted these are the ones that i now know this is my trusted key ring and that has an implied greater sense of security and trust than one that i'm downloading from a service on demand over the internet so uh really if somebody wants to have that level of trust what would happen is uh forrest would basically call up jonathan and say i

have your i found your key on this public you know key service i'm pulling it and i just wanted real quick just to verify your fingerprint is this the right key and you know doubt you know just verbally go over the last six characters or however many characters you want just to make sure that you trust that key and then when you import that into your local key ring you know for a fact that that key came from jonathan and now anything signed with that private key can be trusted with that public key so it really just demonstrates that as well as jonathan was saying

to let you know how we are verifying any particular container and i think the colors were green for local and yellow for uh if i remember correctly for remote which basically it's green is good locals maybe assuming remote is you may have a little bit of a warning that you want to pay attention here and maybe you want to validate that you know by hand uh and then of course if that's something you're using a lot you can import it one other facet i just throw out there was we're talking about local versus remote keys uh we already mentioned execution control lists and ecls are

a really important way to to guarantee that the containers you're using and the containers that a system administrator on a particular host uh allow um if you're if you're going as far as to implementing an execution control list for a particular container a set of container fingerprints you probably want to make sure that those fingerprints are actually local um not that there's an easy hack here but it is theoretically possible that a malicious user can redirect that request so it goes to their fingerprint you know a pki type service and um and then get get somebody to run some malicious code it's very uh it's

a remote there's a lot of under uh underwork that needs to happen for an exploit of that magnitude but it is in theory possible so uh if you're going to go as far as to execute you know execution control list you probably also want to make sure that you're you're properly managing those keys and validating those keys locally in key rings and in fact the execution control list feature doesn't even interact with remote keys at all uh um for those keys to be verified they have to not even only be in your local key ring but obtainer something we didn't demonstrate is apptaner has a

keyring for the user but it also has a key ring globally for the system just for public keys and uh ecls only validate based on the keys that are in that global key ring so it's not enough for you to have it it's not enough for it to be present it's the the systems administrator that configured ecl in the first place has to have imported those keys as well i completely forgot that you're right though i remember it now thank you very nice so we have more questions coming in one of them's been sitting there for a little while a little bit kiddos kiddos uh

when would you use sif encryption compared to sift signing and are the two features complementary or do they support each other like why would you do one versus the other and not both so uh signing is to to verify that you you the intel to verify the integrity of the container that it has not been modified uh since it was published uh to some degree of trust through the key material that you have from the past um but it doesn't protect the content it doesn't um obscure the contents of the container from the the person who receives it uh from uh you know from taking

it apart seeing what's in the container if you need to do that if the data that's stored in the container is itself secret that's where you would use encryption um and you should be able to sign containers and encrypt them at the same time and and some uh software environments kind of do those together by using the same keys to sign and encrypt at the same time uh aptainer seems to use a different mechanism just straight rsa keys um for uh for signing rather than the pgp mechanism uh that's something i personally need to look more into i don't know greg or forrest if you

have more specific experience talking about them there's kind of a neat feature that you can do which is you can actually encrypt so let's say for example we went through this process uh forrest obtained jonathan's public key so we can now if jonathan were to share a container to forest again we can validate that forest knows for a fact that this is um trusted forests can now create a container using the reverse process and encrypt it using jonathan's public key which forrest has once that has been encrypted jonathan would be the only person in the world well whoever has access to jonathan's private material private

key material and encryption or decryption key um jonathan would be the only person who would be able to actually open up that container and look at the contents of it so you can now think of kind of and different use cases where you can now verify trust and integrity of not only the contents but who who actually sign that you can layer signatures to manage provenance of containers uh manage it as it goes through let's say a dev sec ops cycle right here's the author signs it um you know maybe the security team signs it maybe the production team signs it and you require all

of those signatures for a container to run on a particular infrastructure uh so you can manage all of that but that's the signing right we're not actually uh encrypting the data within that container so we can do encryption of that data plus the signing and you can imagine the level of security and trust that you're kind of building in that so forest could be not only encrypting the container that he's sending back over to jonathan so only jonathan can open it but forrest can only can also sign that container hand his public key over to jonathan and now jonathan knows it came from forrest he

knows it hasn't been uh messed with and jonathan's the only person who can actually open up that container and and evaluate and leverage those contents that's very cool very cool all right george we got a few questions from you i think the first one was what open source key server can acting or integrate you want me to with it since i'm already talking unmuted so uh apptaner is leveraging standard protocols here so hkp and i forget exactly what hkp stands for you all can look it up but hkp is the protocol that we're using to actually move the keys around it is a standardized protocol

which basically means pretty much any key server that or pki infrastructure that leverages hkp as that protocol will automatically just work with obtainer so we would be able to immediately just start you know pulling pulling keys pushing keys and this is why we're using a very common uh public key service to do this so it's it's very easy to integrate this with with any kind of key server so that was the first next one is is it possible to build and sign containers from a ci pipeline another great one um i'll take it just because i'm on a roll at this point but i do

fully expect jonathan forrest and zane to to add if i mess anything up or miss anything uh so uh yes absolutely uh obviously there's some integration that you would have to do into your ci pipeline with regards to secrets and unless those those uh your private key is not encrypted which i don't recommend especially putting it up into a ci pipeline um or infrastructure so it'll probably have some sort of secret associated with that and you're gonna have to leverage your ci pipeline infrastructure to properly manage that secret and that key material but assuming you do that properly yes it's absolutely possible and i kind

of just alluded to it as i was talking about a devsecops pipeline where basically you'd have maybe a get operation that causes a a ci to go off and build your container then you have multiple pieces of this pipeline or multiple processes in this pipeline to do things like not only build but now actually run various security compliance validations against it you can do container scanning against it you do all these different pieces and again you can be signing them every step of the way such that the resulting container is absolutely guaranteed to be trusted and and you can host that once it's been signed

you could literally host that anywhere as a matter of fact people are actually hosting this within registries that are part of or associated with ci pipelines so various ci infrastructures now are actually offering uh oci based registry support and singularity works directly as optaner sorry apptaner works directly zane i'm going to blame you for that one uh attainer works directly with these various um uh registries uh that are supporting auras and auras is i think it's oci registry as storage if i remember correctly what auras stands for get it awesome and uh it allows it basically allows us to store and persist the the singularity

blob directly and that was the mechanism that we actually demonstrated just a moment ago uh with jonathan sharing that container to forest he was actually using uh oras to do that so a lot of ways you can integrate this into a ci pipeline and this is definitely the trend that we're seeing from most organizations in terms of building their containers and integrating those containers with hpc uh and then they do that in an automated fashion such that their containers are always kind of always up to date and then when they go to integrate them into the hpc system you know they can have uh either

automated ways of doing that or if we're talking fuzzball fuzzball will automatically pull from a particular tag so if you're building from a ci pipeline as something like latest or tag is latest or the tag is production or whatever you want you can always ensure that you're pulling that version of that container in an automated way and the infrastructure manages all of that an hpc system it's one extra step but it is um so it's ci not really cd um but or manual uh deployment so md cimd that's what it is uh but basically it gives you that ability to to automatically be building those

containers signing trusting those containers and then sharing appropriately with your infrastructure as i think we just had another question that popped in is it possible to lock down execution to only allow containers that have trusted signatures to run i think jonathan you touched on this a little bit earlier but i i would rather you answer it again yeah so this is um the execution control list function it's just a config file uh that exists globally for the system so it's not a per user setting uh but within kind of the the general etsy apptainer directory structure there's an ecl dot something file that allows you

to specify one to many blocks of configuration for containers under a certain path and you can give either a list of signatures or a list of fingerprints for keys that must have signed or may not sign so you can block containers with certain signatures um you can require that all of the signatures in a list are present on a container or you can give a list of signatures any of which would allow it to run um and then you can lock that down for you know only verify containers that are in slash temp or only verify containers that are in uh certain directory structures and

require that your your containers be in certain places as well very cool thank you so we're actually running up against time here guys so it's gonna wrap up but i'm gonna leave it open to you guys if you have any closing thoughts you want to add greg you have to go last so i'm gonna start with forest um no nothing too much for me uh we have that picture of jonathan as the super computing mascot oh nice throw that up there but i'm not sure if we want to but um i thought that was kind of funny but that does actually look like jonathan to

me throw it up oh sorry i need to probably share this my apologies it's uh right here if i can find where i put my stream yard in among all these links too many tabs not enough time i think that's jonathan so this is me many years ago um that's awesome there's a lot of story behind that but i'll leave that for another day that's awesome that's cool all right jonathan do you have anything to close with here no uh yeah first time on here really happy to be with the team and and glad to be finding ways to contribute and and work with all

of these uh apptainer and and more and more fuzzball it's all very cool thank you for joining thank you for putting the demos together too appreciate it absolutely craig i like how you saved me for last is that because i talk so much i'm just gonna help you because we're hiring so a couple points um yes we are we are hiring um and we are very happy to have jonathan and forrest with us forest has been with us for a little while um and uh the team is growing so if you know if this is stuff that interests you and you you want to be

part of it definitely reach out to us let us know the other thing i'm going to make a comment on is the amount of times that we uh call it call apptainer singularity and this has to do with the fact that it started life a singularity i created singularity back when i was at the department of energy at lawrence berkeley lab and joint appointment to uc berkeley i have to make sure i give all credit where credit's due to my employers um and uh singularity came to be over there we prototyped it we brought it out to the community and it's been a fantastic tool

for a really long time there's been corporate involvement with regards to singularity and we want to uh but but moving forward we always want to make sure that the project stays in the community 100 and so uh singularity has been moved into the linux foundation and as part of that move the linux foundation asked if we could rename the project uh just to make sure that they have the ability to protect um the the project and the name and and the branding of that project um as best as they can and singularity as you may know is a term used all over science research and

everything else so we uh uh we put it out to the community and the community uh spoke and voted and apptaner is what was chosen it was chosen because it is an application container system and so it made a lot of sense but that is why old memories muscle memory inertia is very very difficult to break at this point because there is many years of legacy of calling it singularity but the project is the same it is it is taking on a a new life as apptaner but it is a absolute simple transition there's another webinar that we did specifically on that and um and

uh being associated with the linux foundation having the linux foundation run that project is of massive benefit to the community first and foremost and uh and that's the message really i wanted to get across so see i told you i was going to talk a lot sorry about it that's exactly that's great and the part of the linux foundation is actually getting more exposure and exposed to other things and allowing for more integration so it's just growing faster and faster and having more people interested in it so thank you for sharing that craig again we're hiring check out the site if you're interested in any

of this stuff like and subscribe so you can stay in touch with us and if you ever need any help or you ever want to talk to us just reach out on the site we'd be happy to schedule a consultant a consultation with you and talk through your needs what you like what you don't like and just thank you for joining us and we will see you next week thank you bye everyone thanks bye