Getting Started with Apptainer: Simplifying Container Deployment

singularity image format and I think it's cool like that we're they we're we're kind of talking about that today because um you know I was trying to think of a topic uh for today's conversation about you know what are we going to talk about during the um the uh container education uh webinar and I started to think you know um well I started to think about this and that and the other and then I I thought it finally of this topic and it's kind of funny because this this idea of a a a single image container format is so kind of fundamental to to one

of the differences um that that sets obtainer apart from other types of container formats that you don't even kind of think about it if you have been using aper for a while it's it's it's like such a fundamental thing that you tend to miss it if that's you know if this is something that you that you use a lot but so because of that I don't think we've actually done any kind of like a webinar on it and actually talked about it and um it's it's an important topic that we should talk about um it's a it's a main differentiator from for Apper from the

other kind of container platforms out there the idea that your container is actually just in a single file that you can move around from one place to another so that's what is it's the singularity image format and it is kind of the um the state of of the evolution of this single file container format which um aptain or previously Singularity has always pretty much had so you say pretty pretty much always always had okay so this was the the basic idea behind Singularity is that you know previously some of the most popular container at that time was more of layered images is that accurate right

right so yeah and and it continues to be accurate that that you know the most popular um container platforms aside from Mainer um use what we what we talk about as occi which is the open container initiative it's a set of standards that dictates how um containers should be saved and they are saved and stored as you know we commonly refer to those as layers but what that really means is a bunch of tar walls that each have um file system information that can be um layered over top of one another with an overlay file system to create the final container at runtime and that

system has got you know some advantages um one of the main advantages is data data D duplication so if you have like a lot of containers that are all based on the same uh base operating system then you only need one tarball for all those different containers um that has your base operating system and then you just have layers that sort of tweak what's in each one of those containers on top um so that's one of the advantages uh you know one of the disadvantages is that you kind of you you know you need um tooling to be able to you know take all these

tballs and put them together appropriately and make the container out of them you know um and so because of that you can't it's it's not you need like uh an ecosystem to like move a container from one location to another um you need you know it's not straightforward to just like execute a container um you can't like treat your container as an executable and there's other things that we'll get into that you know you get advantages um from using like a single single file format did we want to give a little shout out to who kind of started and created and put the um C

on the sure yeah yeah yeah um well you know so I've got kind of an outline here so in in the background we have like kind of an outline that we're working on um so yeah let let's uh yeah so uh I was gonna say later on when we talk about because I I kind of want to talk about the history and kind of go back in time a little bit but yeah um when we when we get to the actual development of CIF itself um there's a there's a fellow that used to be very active and did a lot of development with u what

was then Singularity Yanik kot and so Yanik basically took the requirements of like you know what we wanted as far as you know what the community wanted and and so on as far as what a uh a a singularity image format file should look like and he used those requirements and kind of came up with some of his own requirements too and created a really flexible and extensible um file format that you can use to do a lot of different things and so um yeah shout out to Yanik uh shout out to yanic that' be cool if he was watching right now and another thing

that he created also so if you've ever compiled um Apper from Source you know that there's a um there's a system that we use called make it uh which is kind of like different than most other projects and you you you uh you run the command M config and that's that's yanic too so he created that that system to compile Apper he's awesome yeah it's been a while since I've chatted with that guy so hope you're well yanic and thank you for all the good work that you've done so all right so Dave you want to talk a little bit about the history you said

that there's some benefits in the you know what's commonly referred to as the layered format um these tarballs that you were talking about but there are some disadvantages as well that's kind of like where Singularity kind of was Bor is that what you're going to talk about now I wanna I want to hear that history well yeah so like um you know we we often talk about kind of the Genesis of uh Singularity and kind of you know in HPC and how it it was first created and I've told you know I kind of I talked to Greg about this uh a day or two

ago in slack because sometimes I tell the story and I don't know if I get it actually completely right so it's my understanding that that's accurate I tell the story too so this is good let's clear it up what's the true story so what what is now obtainer was originally obviously called Singularity and I I had known that Greg called it singularity in part because of this aspect right because um you know the because of the single file image format and I but I I kind of knew it had something to do with a company that he worked for previously and something something so I

asked him about it and he said well he used to work for Linux care and Linux care had this thing called a a bootable business card and so what that was is like um youve probably seen these things they're you know um they're prob they're relics now but uh you know like a little tiny CD with like thep and the bottom shaved off this sort of looks like a business card with like rounded edges okay and so this this little business card you could go to like conferences and stuff with and you could um you could give it to somebody as a business card but

then this business card that had your name and address and whatever on it you could also stick into a computer and you could boot your computer from it into like recovery mode okay and the the way that that worked um you know once again hopefully Greg will chimee in in comments or something if I mess any of this up but I think I've got pretty much the story now the way that this worked is it had you know it had a a single file on there um that you know well had it had one file that had the the image that you could boot into

on the you know you could boot the computer into a recovery image and that single file was actually named Singularity and so he was kind of like thinking along those lines of okay we're going to have this single file that has the entire operating system image in it and that's going to be you know the the that's going to be the container and so because of that he kind of called it Singularity and Singularity also fit with you know the the entire kind of ethos that we were uh you know working with at the time um so that was the orig and so um and

then the original so version one of Singularity I went back and I was looking at like GitHub and version one actually had a different file format called sa um maybe you know I didn't ask Greg about this but maybe like Singularity app or something like that maybe a you know a uh foreshadowing of the the Apper name um but in any case that was a that was basically just a compressed cpio um archive and version one of Singularity was was it was almost kind of like a proof of concept kind of like uh get get people in the community interested and it wasn't really until

version two that we started to see stuff that really looked like the Apper that we currently know so anyway there was this um Sapp file for a while and then with version two um I think that the primary file format was ext3 and so when when was this was this 2015 uh yeah I think around that period of time 2015 2016 in that time frame yeah so then um ext3 was the uh was the um the format that we used for a while which is you know that's a that's a common format for like um a little bit older but like for Linux file systems

and stuff um ext3 had some problems uh I don't really know all the technical details but I do know that it's kind of a sparse file system um and that it was it as you copy it back in forth from one place to another sometimes it it you corrupt the image so it was not unusual to like you know um move this from one computer to another you know maybe one time out of 10 find that the the image got corrupted and you had to redo it again so it had some issues with it um and then you know at some point um you know

that uh Greg kind of had the thought and you know some other community members um started thinking too that squash FS would really be the way to go so um the squash FS file system um is really it's it's used a lot for stuff like you know live USB booting and and a lot of kind of similar applications and it's cool because it's compressed uh it remains compressed there's like kernel drivers to you know to run run the the image um without decompressing it and stuff like that so it's it's really convenient and you don't have this problem where you like move it around and

it becomes corrupted or anything like that so um you know that's what we use for a long time or that's what we continue to use is squash FS um but then and then this is kind of what you were asking about before so so I think that uh a lot of us um who were developing um at the time Singularity started to think about what are all the different advantages that you can get from having this this single file be your container and you know one of the advantages that was there from just right from the beginning is that you can take this single file

and treat it like an executable and you can just run it and then that works you know based on the way that it's designed and architected but we started thinking more about like what about attaching other pieces of data uh to this single file you know what about like um putting metadata in the file as a different uh there a different data object within the same file that moves around with the file what about like signing the container and attaching you know the the public material of the signature to The Container itself so that when you copy it from one place to another you don't

have to go find or have a server or some other you know infrastructure set up to have that signature go with the container it just goes along with the container um what about encrypting the container since it's a single file I mean you just do that and encrypt it and you know you don't have to decrypt it or anything like that when you run it so you know I think that um we really started to think a lot about uh you know leveraging fully leveraging the advantages of having this single file and that's where that's where yannet kind of came in and said um you

know not not only do I want to try to uh you know leverage the current use cases that we currently have in mind but I want to try to come up with a file format which is extensible and can be used and like you know and flexible that can be used for maybe future use cases that we haven't even thought of yet and so that's kind of what CIF is so CIF is this um it's this file format which allows you you know it's it's got um a header some Header information which has like pointers which basically tell you where the different parts of the

file in of of other files inside of it are and it can have one or more um you know like uh root file systems inside of it in in squash FS format which actually uh are the container but then it can have other data blobs inside of it too things like Json for like metadata or um it holds the definition file signatures um and you can just put arbitrary data in there too and you can put other file systems in and do things like have overlays and all kinds of stuff like that so when you you were saying that there are some benefits to that

what are I mean you just you didn't really like give any detail like is there some detail you're like oh yeah we've got these certain use cases that this is really good for but we're thinking about beyond that like what are what were those original use cases that Singularity really shined for yeah so um I think that like uh you know know the original use cases so it's it's kind of funny because I think that um a lot of the things that we're talking about are things that we've actually demoed before uh in you know in in other webinars and but I I think that

what we've kind of glossed over is that it is this single file format it's the CIF image that enables these features so I think yeah what I'm kind of trying to do here is connect the dots between stuff that people already know like the the signing and verification process the um the the ability to encrypt your containers and run them encrypted and um you know things like um having writable overlays attached to your containers and I'm trying to connect the dots between all those features and the underlying technology which is the sift technology which allows this stuff to actually happen um you know we've thought

about too I you know I don't we haven't in we haven't um developed anything like this within uh Apper proper but you could for instance you could think of instances in which you've got more than one container within a single um sift file so and you can do that you can put more than one uh squash FS root file system without any problems into a single Sif file and so if you were to carry that forward um you could think of and we've kind of like bald and thought about stuff like this although we haven't really implemented stuff like this but you could think about

this is from from like um an orchestration point of view where um you jam maybe a whole bunch of different containers all into a single C file and then you do something you you know maybe your your primary container is able to extract those other containers from the cph and is able to orchestrate them in some way you know that it's it's because this is such a a flex ible and extensible file format that you could you could do stuff like that that was not um originally you know part of the original intention behind creating it but it's it's like it's flexible and extensible enough

that it it affords a great deal of creativity and you can come up with a lot of different ways to use it so is this CIF is part of oper or is it a separate a separate thing that's a good question so um the CFT project while it's kind of tightly tied to aper is a is a different it's a different project and it's a different uh repository on GitHub and um obtainer depends on it but it's it's a different it's a different bit and in fact um so so CIF became kind of like the default container format forap tainer 3.0 uh Apper 3.0 or

well at the time sorry Singularity 3.0 um Singularity 3.0 represented a really major rewrite so um up to that point in time Singularity had been written kind of in a combination of C bash and Python and you know which is cool because it was like Community effort and all that sort of thing but um we we kind of went through and we we converted it all to go and you know there are reasons for why why we did that go is kind of the the native language if you will of a lot of different container tools and so to leverage a lot of the tools

that currently exist in the ecosystem it was best to rewrite that and go and then as part of that effort too um you know as part of that rewrite there was also this new container format that we adopted which was um CIF so that was 3.0 and then there's a cth tool and I'm going to be kind of demonstrating that here in a few minutes and kind of you know going through some demos but there's a CF tool um that allows you to do things with the CF file like view its contents uh add new contents delete contents change things about the contents and originally

that was a separate project but um we we ended up folding that into Apper and and bringing that functionality into obtainer itself so now there's this um command group with an obtainer that all falls under the umbrella of cph and you can use that to manipulate CF files wow cool okay so we were having a demo today Dave godlo this is very exciting yeah okay so always exciting it's always exciting when I demo because you never know what's actually gonna happen that is true that is true you guys we we'll never let you live it down one time he was like with his phone you

gotta go to it can you guys see it can you yeah yeah that was something okay but it looks like I'm gon to be able to share today so I I have um I'm always safe now so and the way in which I'm safe is that I just bring up a Chrome browser window and I I use the SSH terminal in the Chrome browser window because even if everything else on my computer is messed up and I can't share I can always share a browser tab it seems like so there you go this is the way I do it now just to be safe but

you can't even tell the difference it looks really nice right oh a side note too any of you guys you uh watching if any questions come up you want some clarity you want to ask Dave something like go ahead and pop that in there we'll get to questions in a minute cool so this is the CIF tool that I was talking about earlier so we've got this CIF um command group and so here's all the available commands that you can use with CIF right so this this is the these are the things that allow you um so you can add delete you can dump I'm

going to be showing some of these things but not all of them um you can list the contents of a cph so let's just kind of get an idea um I'm I'm going to go through and just play around a little bit with a CF file and show you guys some stuff that you can do with this CIF tool um to do that one of the first things I need is to just get a cph file so let me go ahead and and download one from dockerhub now um you heard us talk about oci earlier um dockerhub you know is a place that you can

get oci images and if you just grab one using the docker command like so you know and then you just grab an image what'll actually happen is you will get the the tarballs you'll get the layers of that image and then Apper will convert that to a C file for you on your behalf but you don't have to do that so now there's a neish protocol that you can use called oras oci registry as storage and that allows you to push C files to dockerhub and also pull existing C files from do dockerhub and so I've got um some C files that I've pushed up

to dockerhub uh I'm going to go ahead and grab one my username and I'm G to grab one called Rocky Linux and this is nine and I've already downloaded this so it's cach so we don't have to wait for it to download but um you know no matter what it was going to end up being a CF f file when I downloaded it but in this case it was actually a Sif file before I downloaded it so this this command just basically downloads the existing SI file um so that's kind of cool when yeah that is actually really cool when did oras like become a

thing is that a a project like an open source project that people are like hey this would be cool if it existed and they just created it yeah that happened um ah man that happened like four or five years ago it's it's been it's been around for for a while um um yeah and the original idea was to just push not containers but just random stuff just data blobs up to um oci Registries that are that are there to serve containers and um you know we there there was a pretty large company that got interested in using this to be able to push and pull

um sift files up to places like dockerhub and so we ended up working with this company um that begins with a micro and ends with a soft so and uh we ended up integrating uh oras uh into um ater than Singularity so that's why you're able to do this so easily uh through obtainer yeah that's great I I remember we used to get a lot of questions about like wait a minute it's not Docker right why am I getting things from Docker like what it's translate how is it doing that like is that safe you know there's there was a lot of kind of hesitation

around that well in this obsoletes so there used to be um a few projects to try to create oci style Registries that instead would host um SI files and this kind of obsoletes those projects because it's like now you have an oci registry you can use it for C files yeah um this image happens to be signed so I've signed this image um and because I've signed it I can verify it the uh key material is up on a remote server and um you know so I'm able to so so basically if I go and I I double check this fingerprint and I make sure

that it's the same fingerprint that uh you know I originally signed the image with then I can be sure that this image has not changed since I created it okay so that's one of the advantages of using CIF is signing and verifying um so let's have a look at this Cy so I'm going to do a CIF list and I'm going to list the different parts of this C file so if we look we can see that this C file um has four different data blobs in it um the first of these is a definition file that's the type it is here um there's a

generic Json uh that Json has got some like metadata associated with it that we'll look at here in a second there's the actual squash FS the actual file system that's the the actual root file system that makes the container and then there's a signature block and that signature block exists because I signed this uh before I pushed it up to dockerhub and and that's the way in which I can verify this because the signature block exists in the cth file so let's have a look at some of these parts so I'm going to do a obtainer CF dump and I'm going to dump this ID

one out of the image and when I do that that's a def file so I'm going to actually get the definition file that I originally used to create this container and in this case this is identical to doing an well it's not identical I don't think actually but it's it provides the same information as doing uh an Apper inspect def file and then the name of the image but in this case we're getting this actually from this metadata which is outside of the file system and it's just you know it's just in the CF so we don't have to like open up this file system

at all to see this this file I'm gonna do the same thing with um that Json and so this is this is the second if I can scroll up here this is the second uh second piece here the second group is this Json file so I'm going to go ahead and dump that and if I do that you can see it's a mess right because all the white space is messed up so what I'm going to do is I'm going to reformat that through this Python program which allows me to do like a pretty print and see the Json in a little bit more of

a clean format and so if we look at this wait so is all of that inside of the container like this ability to do this Json translation that tool no that tool is actually just on my system so I've got python on my system and so what I'm doing is I'm I'm using this pipe to say okay all this stuff that I'm dumping out here pipe that into this new python command which is going to format it in a prettier way okay but if you wanted to you could put that in the container yeah you could do that yeah if because if and then if

you wanted it to be there for it sure the next time but but then it would then your your signature would be different right the the fingerprint would then change and you have to resign it yeah we're going to go through that in a few minutes yeah but you can alter you can alter these containers and when you do of course you mess up the signature and they they're you can't verify them anymore which is as it should be yes right but so this this Json basically has uh a bunch of the metadata that you would you know see inside of a container so it's

got the environment file that's used um in this case lcall equals c is the only environment variable that I'm adding um it's got uh these um like the labels so so there's metadata that you can add to a container um you know that that's here so i' I've added this author uh you know me um into the into the metadata um and so it's basically got all it's got the Run script the def file you know all that kind of stuff let's look really quick at the signature too so if I just list the contents again you can see the the signature is id4 here

and so I can dump the signature just by using the id4 here and so this is public key material and this is the public key material that is used to verify this container and it travels with the container and you know so that's kind of convenient and then you can even dump so if we go back up here and look the the file system itself is ID number three and you can even dump that uh but I'm not going to dump that to standard error so you can redirect that I'm going to redirect that to something called Rocky 9 squashfs and so what this is

going to do of course is all the standard error is going to go into this file so I've got this new file Rocky 9 squash fs and guess what that's a working container it's a squash FS file system so I just basically extracted the squash FS from the CIF file and brought it out just as a bare squash FS uh file system and obtainer recognizes that and obtainer can still use that the only thing though is if I wanted to like use the ver the verify command that I used previously um on that bar squash FS image it's going to say this isn't a cth

file I can't there's no magic number in the C in the file showing me that it's a cth file and because of that I can't verify it um so Dave Rush has a question I don't know if you wanted to put it up but I'll definitely ask it because there does look like a lot of jibber jabber uh going on up there so his question is does the CF file is it text or binary it's binary yeah it's binary um it's got I mean it has like it has text in it obviously because of the Json and um you know because of the like you

can I'll show later on you can basically put just arbitrary data into it um and that can be text but it it it in and of itself is a is a um a binary okay so um let's see only Neo can read that I'm gonna I'm going to go ahead and uh get rid of this Rocky 9 squash FS thing here um and let's see uh let's see oh yeah I want to go through I want to go through now um kind of showing you a little bit more how signing verification works so I already showed you uh if I list everything here you know

that I've got this signature block here now one of the cool things about um CIF which is a little different from some of the other container platforms is you don't have to stop at one signature right you can sign uh you can sign a container with multiple different signatures so I'm going to go ahead and layer another signature on top of this and I have to enter a passphrase and now if I do an obtainer verify so what would be the point of multiple signatures yeah good question actually before I before I do that verify let me list it again and now you can see

that I've got two signature blocks instead of one and so what the point of multiple signatures would be let's say you've got containers that you don't want to run unless you've got uh your developer team has signed off on them and your security team has signed off on them you got a QA team and you know you can just keep on going down the the road and and say I've got these five teams that I'll have to verify this container for this critical environment before we actually run it and so you can do that um with obtainer and so by default now if I do

like obtainer verify uh it's only going to verify if all of these different signature blocks if it if if they all check out so I've got this remote um key material once again because I signed this Rocky image before I pushed it up to dockerhub and I'm pulling this uh this stuff down from a remote server and I've also got this local key material which is the the kind of the silly demo key that I just signed this with so um let's see so what is it is it verifying is it verifying like is the SI binary code part of what it's verifying yeah so

it's verifying um that no bits have changed since since the signature was applied so the the sign when you're signing it you you basically take the private key material and you use that with the the contents of the C file to generate what the answer should be and so when you verify that you take that key material and you compare that up against the contents of the CF file again to make sure it's been unchanged and if it changes at all then it won't verify anymore it'll say oh well it signed but it changed since it got signed so I can't verify this anymore I'll

show some examples of that here in a second yeah yeah yeah i' love that like because then obviously you'd want a report back right like what has changed yeah right right right so let me go ahead I'm just going to um show you what it looks like to remove some data now from this uh from this file so I'm so I'm listing it again and this last signature block I don't really like that I want to get rid of it it's kind of a a silly key so I'm going to go ahead and do obtainer c delete or Dell uh five this is ID number

five from the image and then if I list it again we're back to square one this is the this is the SI file that I originally downloaded um yeah and then yeah if we if we verify it again just to kind of show it'll still verify but it's just it's just one one signature now that it's verifying because that's that's all that exists there all right so um I'm gonna go ahead now and add some random data so um let me just create like a note so let's say I got a grocery list here right so I want to get milk and cream and hamburgers

and brisket okay I'm coming over good hopefully you're you're noticing kind of a bovine theme to this uh shopping list here um so let's say I just want to add some random text um so the way that I would do that is now I do Apper well let me let me let me show you how to do this so first we're going to do obtainer sift ad and I'm going to get help on that because this is a little bit more complicated than just adding it the reason it's a little bit more complicated is that I have to give it a data type I have

to tell it what kind of data is here and that can be it might be a definition file um you know it could be you have to give what your grocery list you have to give your grocery list a a data type file uh when I add it to the CIF I have to tell the CIF what kind of data I'm adding okay so it's so and in this case I'm just going to say it's generic data doesn't have any specific type to it it's just some stuff um the other thing too that it's good to do is it's good to associate it with a

group so you saw earlier um you saw earlier that if I list this there are these group IDs so all this belongs to group one one and so I'm going to go ahead and Associate this with a group so that it knows that this is part of like you know it's associated with all the it's it's not really because it's just generic data but I'm going to go ahead and Associate it with this other stuff and that's going to tell like if I try to verify this and if it's not part of a group the verification procedure will fail and it'll fail with kind of

like a nonuseful error where it just says you've got data in there that doesn't belong to any group and so I don't know what to do with it okay so let me go ahead and so Apper CF add I've got this command I think saved here so the data type is seven that's that generic data type that we've got here um the group that I'm going to add it to is one uh and I'm going to add it to this SI file and yeah it's note to self okay so now if I do an Apper list I've got this generic data at id5 now it's

just generic raw data it's my grocery list I just added to f file but now I know what you want to see Rose you want to see me try to verify right so let's let's see what happens oops so when I applied that signature um there were only these three blocks here and that those three blocks is where the signature was applied across that this fourth this fifth one does not did not exist right so it says yeah that you know you've got some unsigned data in your container and I'm not going to verify it so even in the where it says the ID group

link you know even that little very nice spread out for the human eye to see very easily you can see that what was added is underneath the signature is that also an indicator of like where things and what could possibly have changed um I guess it could be because I could delete that signature and sign it again and if I did that then this uh generic raw data would be moved to ID number four and the signature would be moved to five but I think that's just you know I wouldn't depend on that um I guess there could be situations in which the signature could

end up being earlier in the list but it could still be applied over everything okay cool so let me go ahead and delete my grocery list I don't need that in there um well that's actually an interesting question if you put something in but then delete it will it still give you the error that something has been put in and out even though like it still looks the same as it originally did let's find out okay this is very interesting and so now the Sith file is bit for bit the same as it was when I downloaded it right so it's I can verify it

again oh that's cool okay so I see how this makes it very sharable in like scientific spefic environments where people want to take the container do all kinds of crazy stuff with it get whatever reports and output that they want because they followed exactly what other people did but then you know get rid of all that if they want to right let me let me show you one more thing too so and I'm gonna set this up a little bit before I just start typing so another another kind of interesting thing that you can do with this um right so so squash FS files are

uh readon that's just the way they are you can't you can't write to a squash FS file because it's compressed and that that's just not what it's for um so how do you write to a container well a lot of times what we do is we don't actually write to The Container We Buy Mount directories from the host system in we write to those but sometimes you want to write you know something that that actually moves with the container so you can use overlays for that um you can create images that um then you apply to The Container at runtime and then they save they

save the changes to the container and then you apply them again when you run again and those changes persist well with CIF you can just take one of those overlays you can just jam it right in the cth file and so now the changes to the container move around with the container and so that that that's a that's an overlay that goes into the Container let me show you how that works that's cool so you have an option to do that you don't have to do it yeah right and you can you can backtrack and get rid of it when you're done so so I'm

going to go ahead and create an overlay I have to tell it the size and this is I'm pretty sure it's in megabyte so this is going to be a a one gig overlay that I'm going to add to this container and by giving it this syntax it's going to tell it not to just create the overlay on disk but actually to add it to the sift file okay when I do that it's going to fail and it's going to fail because it says this is signed and I'm not going to add some writable partition to ass signed container right because that's a bad idea

so I gotta get rid of the signature first so I'm G to do an Apper um C delete I'm pretty sure it's four I should probably I don't know verify that so uh let me just double check real quick uh well I mean it's up there I know it changed it might have changed a little bit because I it didn't change it's fine oh yeah because we got rid of the shopping list yeah let me just double check I didn't just mess up my demo yeah so I still have the file system that's the important part and the signature's gone so now if I do

that Apper uh overlay create again command and then I list it again cool I have an ext3 overlay which uh ext3 remember is that file system that we used to have um the sparse file system it's got some drawbacks one of the things that you have to like you know tell it what size to create the file system but it's writable that's one of the advantages so now I can actually write to this uh container so if I do something like Apper shell and I'm gonna use fake root so I can elevate my permissions inside the container even though I'm not roote on the system

and I'm going to say writeable so I can actually make changes to the container um you have to do that even though you've got this writeable overlay because otherwise obtainer out of an abundance of caution will Mount the overlay as read only all right so now I should be able to do something like touch a file at the root of the file system and if I look at it there it is I just created this file called Foo and if I exit and then go back in and look again it's still there so it persists across um across different uses wait how did you get

it to persist because that WR overlay the writable overlay right right right yeah so um the way that it the way that I was able to write to it and the way that it persists is that it lives that that Fu file now lives in this ext3 overlay okay so let me go ahead and sign this container again so of course you know I had to get rid of the signature to make this work and now I can sign it again and you know everything will be cool after I sign it so um and now if I list it again now I've got a signature

block and if I verify we can see that nothing has changed since I signed it which is pretty cool you signed it yes yes but go ahead go ahead ask the question yeah but you obviously changed something in there since you got it down so so since I signed it's nothing changed what if I change something so I've already touched Fu what if I touch bar and I create another file [Laughter] Fubar indeed so now bar exists Fu exists and guess what this container has changed since I signed it right so if I try to verify it again oops it's gonna say no sir data

object Integrity has been compromised yes this is wonderful however what if somebody who you were using their container but they didn't want you to change anything in it or get rid of the signature or put a new signature and they want to know that it's the same container that they gave you 20 minutes ago before you did your magic yeah so and what if I'm trying to fool them is that what you're saying yes so I can sign it again and that's cool but it won't have the same fingerprint because I don't have I don't have their public key material or private I'm sorry I

don't have their private key material if I could steal their private key material then I could I could create a a container with the same fingerprint but otherwise this is why I always say that the fingerprint is so important anybody can sign a container and if you just check that it signed that doesn't mean anything you have to go back and you have to look at the fingerprint make sure it's the one you're expecting to get so that is very that's a lot of numbers and letters so like is is there an easy way to scan that no this is this is cryptography so so

there's there's no so the the signature um the validity of the signature requires um the private key material which is a secret that hopefully you will keep close to your heart um and it also requires uh you know the contents of the container when it's signed so it's it's it uses those two things to generate that that fingerprint and then that's you know that's a fingerprint that you expect and so ideally you would share that fingerprint if you want to be really paranoid you would share that out of band somewhere too so right now um if you go up uh to to dockerhub you will

see that the fingerprint that I originally um showed as part of this container I've put that as a comment up on Docker Hub so that people can go and they can look and they can see that fingerprint and then they can compare it that's got a problem with it and the problem is if somebody compromises my dockerhub account not only could they swap out all those containers but they could change that comment and change the fingerprint so ideally If people really want to be paranoid um I would email them the specific fingerprint that they need or I would write it down on a piece of

paper and mail it to them and they could write it down on their hand and you know double check it as you know that's but that's you know that's the kind of thing that you have to be careful about yeah yeah cool I love that it's good to have options as as well because not everything needs to be you know secure like that so okay so say then you gave that container to somebody else okay we're not really doing demo anymore but there probably I mean there probably I know we really are up on time but there probably is a way to look back at

the history of everything that's happened since that container first got pulled actually not oh interesting um yeah so that's one of the reasons that we always tell people that they should make changes in definition files and rebuild containers from scratch because that definition file kind of ends up being being your like the record of all the things that happened but for instance when I just added that grocery list there um to The Container um you know there wasn't anything there might be if you dig through uh there's metadata associated with those this is something I don't know that off the top of my head but

you might be able to like look and see when it was added and get some information there but there's nothing really explicit and obvious that says that this is you know something that was added after the fact and you know change the container yeah yeah cool wow Dave that was awesome if there are any questions we can we can pop them pop them up now but um yeah we were just kind of talking about like because we like to do this once a month is right um have some kind of container education Series where we kind of dive into uh what we're doing in aper

world and yeah I I Remember You Came you're like what about cth like we don't really ever talk about that it's just kind of in in the background yeah and it's once again it's like so fundamental that if you're used to dealing with obtainer all the time you don't even really think about it it's like it's almost invisible to you but yeah it's it's uh if you if you come from oci world it's like a totally foreign concept and something that yeah we should be totally exploring and talking about so random question because actually somebody just emailed me the other day saying that he was

trying to download I think it was Rocky n onto uh USB and he's like it's too big I can't do it I'm like okay well let's talk about where are you getting it from anyway I had other questions but um you kind of started off the conversation talking about a little D little CD driver disc um so is that like how how big is Apper like I mean I guess it depends on what's in it but because it's a single file like can you easily use like a you know a physical yeah so that's a little different though because um so what what they're probably

trying to do is get an ISO from our website that um and create some bootable media on a USB drive and so um like you can you can you can't really so okay um I got a lot of thoughts all at once so bootable media uh needs to be like has needs to have the entire operating system in it including the kernel and the kernel modules and an init system and all that kind of stuff containers don't have that containers leave that stuff out because you don't need an init system because you're going to use the container platform to bring the container up you don't

need a kernel because you're just going to grab the kernel from the host system and put that in and that's what's going to make your container performant so usually containers don't have that so so um yeah I mean an Alpine Container for instance might be uh you know 500 or fewer I don't even know probably fewer than that Megs Alpine container is pretty small um so they can be very very slim but they're not complete now um we have another product called werewolf and werewolf is pretty cool because what you do is you use containers to provision uh nodes to provision computers so you take

a container and then through the magic of Pixie you uh boot a node over the network into that container image and so for that you need that container image to have inside of it things like a kernel and the modules and the the init system and all that kind of stuff so you have to build Special containers that have all that stuff you know there I'm sure there's probably a way once you've got that container um to to convert it into like a bootable ISO that you could put on a USB and use to boot your laptop or something like that but I don't I

wouldn't know how to do that I would have to do some research to figure that out and probably talk to some of the other smart people at ciq to figure out how to do that but so it's not you know to answer your question it's it's um you know it's a little bit roundabout I don't know what our isos look like how big they are um downloading from from uh from from from the rocky website um off the top of my head but they're probably you know several gigabytes in size at least yeah yeah yeah I thought that was interesting I asked for a little

bit more detail that it was um it's not very details like wait where are you getting it from what are you doing um Dave God love thank you so much I'm glad that you brought up werewolf as well because um I mean there certainly is uh synchronicity in the various open source projects that we support and you know add value to at ciq and so aper is one of them wherewolf is another Ascender which is our automation platform uh and of course Rocky Linux um so you guys want to talk about any of those and you want Dave to come in and and and do

some training because he is now the training Master what's your what's your new title yeah that's it I mean it's it's either that or like um Guru or you know uh his Holiness okay okay you know what I like that because you are Mr God love anyway yeah I don't have a title yet I'm we're working on that there there's another question that came in that oh yes yeah is it possible to use CF uh with werewolf for cluster provisioning and the answer is it's totally possible um which might surprise some of the people who work with werewolf uh at ciq but no no I

you can do that and the the reason reason that you can do that is because you can point werewolf to a a bare directory that's got the um the file system in the directory and then you can use it to import from that directory and so with obtainer you can convert a cth to a bear directory by building it with the sandbox command from the CF file so you do something like abtain or build-- sandbox name of my directory name of my and then it would it would dump it would basically dump your C file the file system inside the C file out to a

be directory and then you can use that uh as the base for your image for werewolf so it's a little bit it doesn't it won't ingest it automatically the way it will for an oci um probably that's something that you know should be added to SI or should be added to werewolf I think uh but it's not currently an option well I think you know a guy so add that to werewolf right awesome all right you guys so make sure that you find us on C iq.com we are happy to help make sure that you engage with us um you can also find the uh

various slack channels and open source communities on our website as well if you want to kind of like jump in there and and and test it out we'd love for you to um to chat with us on there all right well thank you so much Dave God love you are amazing thank you all for the sift love and thank you obtainer community for doing all the good work that you're doing thank you Rose yeah hope you have a great day yeah thanks you too all right talk to see you next week same time same place [Music] bye