Container Education Series: Your Ultimate Guide to Apptainer

it's right it's a new day year it's a new month we are and we're we're ready we're ready to party is our first webinar of the year no we did last week no okay yeah we were there last week yeah I was there uh maybe in spirit Spirit yeah I think Greg Greg solell did one with some uh Ascender automation stuff right that's right we're always doing a lot of lot of cool stuff around here all right Dave so say hi tell us who you are what you do I will but I want you guys to go first I want to know where you're at

and what's going on roseen Zayn I okay okay okay okay so this is not a fake background this is a real window and this is real snow on top of this building that came down last night six inches because we had a bet and I actually got out a measuring tape at the hotel last night I was like how many inches do you think this is but I think you measured it before it finished it looks more like eight this morning there's more do you have a measuring tape right now no but we can go back and get okay we're going to have to remeasure

then okay okay so anyway when I last measured it there was six inches of fresh powder that came down on us we were at the hotel together we were having a good time we opened the door we like did botch ball like down at the hotel we were having a good time having you know food and water and the door opened and there was like a blizzard and the snow was coming in and we went out we made snow I got Zay you guys I got him right in the right in the gut side yes in the kidney area I was feeling very powerful so

we are in Reno Reno Nevada this is where uh Ci's main office is I think we have another office as well in Seattle in Seattle exactly Seattle Belle this is the OG office here in Renos so office the big office so right hopefully you guys can come down and enjoy it and you can see I'm very excited about this snow like I'm from the Bay Area California like the only snow that we get is like once every 10 years we'll get like just a little and everyone goes oh my God so yeah to be where there was H six inches of snow that came down

then wake up and it's so beautiful and of course I'm like eating the snow on the walk down here they're like don't eat the snow Rose to e that seems a that is where we are fantastic yes it's super fun no that's great we actually have sales team and several other people here just hanging out spending some time together getting ready for 24 so it's been a great week yeah absolutely making sure that we have you know nailed down what it is that you guys are looking for and we are always open to iteration we want to know what our customers our clients our community

is wanting from us I mean obviously we provide a lot right so you go to the website and you kind of listening to these webinars you know that I mean not just obtainer right which is the containers we got the werewolf we got rocky we got Mountain we got Ascender like there's a lot going on so if you guys yeah you guys have you know questions you want to know about anything like please let us know reach out to us on the website and ask questions now comment oh now that's right yeah that's true we are live don't yell it's you see that you wouldn't

have to be told that but actually we stopped in the side of the road we were stuck in a blizzard with my son like years ago and he did he totally like on the side of the RO got and my was laughing he like games do not see okay I'm sorry kid we're live that was awesome I'm sorry it's totally real back back to Containers yes so Jonathan you're off mute who is yeah hi everyone yeah I'm Jonathan Anderson I'm a Solutions architect with ciq work with Dave and Forest here and uh I'm I'm eager to both uh hear what some of our customers frequently

ask questions about adaptainer R and and learn about that uh but also if uh anyone watching has any questions of their own please put them in the chat we'd be happy to to talk about those as well uh your questions are frequent as well I'm sure there are no no small or foolish questions here Boris I haven't seen you in I feel like it's been months it's probably been a little bit I haven't uh I haven't been around the webinar for a second but like it was SC was the last time I actually saw you it's been a little bit say we haven't hav been

on a meeting a call anything like that we had ourselves closed stuff like that so good to see you again good to see you introduce yourself my name is forber I'm a Solutions architect here at ciq with Jonathan and Dave uh I'm a certified container enjoyer as well uh I got started in containers working at my previous institution deploying software in them for researchers stuff like that um but I am uh huge fan of Apper been using it for a while and excited to see oh there we go see um you got you got the sticker so it's like it's it's official uh oh what

what's going on over there no he's he has certified obtainer rack on my rock I've got I'm a fan so yeah um excited to see what we have to present today let's dive in Dave let's ask questions and I've got no AB tainer stickers handy to show off which is super lame I feel bad now you're just a living breathing Apper sticker right wait a minute he can find a llama can't find an Apper sticker oh yeah you know what we got to make something app tainer that's cool for the kids true yeah okay marketing get on it so yeah I'm gonna I'm gonna dive

in here and just kind of go through the the the frequently asked questions that we have up here on the website now but yeah if you have if there's so I kind of put this list together and I was talking it over with some of the other Solutions Architects and um I think Jonathan was the one that was like you know what is a container like what's and you know I hadn't even thought about stuff like that but you know obviously that's the kind of stuff that we should also have in here so if there's other stuff that's obvious that gets asked a lot or

that you have a question about and you didn't think of please like put something in the comments or you know like Rose said send us a message and we'd be happy to answer those questions and maybe put them up on the on the website as well um but yeah so the first one is just like what is a container and um you know this is something that I I tend to kind of gloss over because I I work with containers so much that I kind of forget that to some people they might be new and that's totally fine but um just as you know you

can go through all the technical kind of explanation of what a container is or you can just kind of think about what it's good for and you know what the concept is and I think that's easier so the concept is just um a container allows you to create a new environment basically and commonly what they're used for is to stick an application or a service into a new environment along with all of its dependencies and that when I say all of its dependencies I mean its entire file system environment variables everything and then you can run it with all those dependencies and it makes it

super portable reproducible um you know there's a lot of advantages to doing that so that's in a nutshell kind of like what is a container so I don't know if uh anybody else has any comments on what a container is maybe my explanation is a little too shallow one thing that I I like to point out just to to help people with um with kind of understanding terminology and what people mean in certain in different contexts is to differentiate a container image from a a running container sometimes people when they say container they mean like that ephemeral idea of an an environment that is executing

and has a program running in it and Apper actually makes that a little bit more straightforward to think about because the container image is just a file that's on your file system that you can run just like any other program um some other container ecosystems and environments that that image just kind of exists it's like the cloud it's out there somewhere it's it's in a registry or it's in a cache somewhere in your system um and it can be useful to to understand that there's a container image which is all of the instructions and the data that is packaged up and ready to go and

then the container as it is executing on your system I probably oh maybe I shouldn't ask a question don't to still any Thunder I'm a little there are a lot of different container formats out there right there well there's a few major ones right but obtainer itself doesn't really care it can pull in from any of those right so that ephemeral thing that you want want to run it lives off in the cloud like Jonathan said in a registry you can pull in even if it's not an app painter container yeah and even make it concrete you can make it be an app painter container

image just on your file system and make it a little bit easier to work with Forest you came off mute oh I was just gonna say that um I was just going to reiterate yeah you can take essentially these containers that exist in some other ecosystems as ephemeral layers and compress those all together into into an actual ual tangible on disk um movable image with obtainer um so you end up with uh basically treating all of your containers not as something that's managed by like a system service but you end up treating them as files on disk that you then run commands against like you

would anything else um I uh find it useful to differentiate containers in VMS not to Ste one CER but just to mention um a VM is a whole virtual computer running with its own kernel its own Hardware stack that type thing it's got a whole Dev tree um that is representing all of its devices and things like that a container it also most importantly has its own kernel that it's executing on uh a container just very very high level um is essentially swapping out the user Space versus this kernel space of your system um so while you're still maintaining the same kernel when you're running

a container you might be swapping out like the user space components uh because you're using this container and so in the end uh you're not creating any new hardware especially an Apper where it just brings in a lot of the existing Hardware from the computer um we can talk about that in a little bit but it integrates like these Nvidia devices things like that within the container itself um just uses the ones the host already provides uh um like I said you you're running your own kernel you're running your own device tree um when you have uh I think I lost my TR out but

that's essentially what I was trying to say you're running your own or you have like devices you have a kernel stuff like that all within a computer or a virtual machine a container is just a hot swamp user space that uses what's already there as far as the kernel and the hardware devices go um and it's not like running something entirely separate from what's already on that system it's using essentially the kernel space components that are already there but just swapping out a user space on top of that thank you Forest yeah well I think that's really cool too that you came up with that

because that was kind of like the next question that I have on on the frequently asked questions is like what's the difference between a a container and a VM and you just encapsulated it really nicely um and and from and once again there's like you can explain things from a technical perspective or you can explain things from a user's perspective like when would I use one or the other and from a user's perspective it's like well containers are lightweight and fast and once they get started up which takes you know milliseconds um they execute at bare metal speed uh you know depending a little bit

on your platform that you're using and so on but um versus VMS that you know there there's a lot that goes into how performant a VM is going to be but it's never going to be as performant as VM as a as bare metal pretty much um so but uh VMS are more flexible right you can run Windows in a VM on your Linux laptop or vice versa uh you can't really do that with containers with the VM you're tapping more into those like hypervisor type features than you are like the more user facing components and software applications that a container is meant to provide

access to I mean could you though like could you put all of those things inside of a container like the kernel and its own libraries and all those other not the kernel you can put it in the container you can put it in there but it's it's not gonna do it's not gonna do much well that you know it's cool though because you can put it in the container and then you can provision a a system with it with it using werewolf that's true and and we use that's why we use the term container uh to refer to werewolf's node images uh because we are

building a container just like Cainer um but we install a kernel in it and then werewolf knows how to pick that up and use it but in a it's not running as a container at that point it's just using the container image and this kind of makes a good side point um about kind of the security of containers when you're inside of it you obviously making changes to whatever kernel is running that you know is basically impossible because it's you know underlying typical aspects but the container the essential point the container can't change what's running on the system itself it can only change what the

users see is running on that system and what applications they have access to so you know for example even if you did have you know some type of situation curently in a container there'd be no way to get the host to switch over to that um because especially with obtainer where you have a lot of um kernel flags and things like that that are applied to the running of containers that make uh any type of permissions escalation impossible beyond what the user already has access too um it becomes really difficult for a user to do anything with that container malicious on the system um it

essentially becomes a layer for them to run in a trusted manner their untrusted applications that they might want to bring in in a container and we do have a question David is a good one actually David Rush talking about mapping ports from processes within the container to the actual host I think this comes up quite a lot it's something you typically want to be able to do right right yeah hey David it's good to good to see you to hear from you um yeah that's actually that's one of the frequently asked questions in here um I've phrased it a little bit differently it in on

the website I think I've I'm scrolling down to see how can I access ports for running Services inside my container and it's it's this is one of those um questions that a lot of times I get you know when I'm doing like arm in a class or something and there's students there that uh are used to using um another container platform like Docker or podman they're like okay cool well you know I'm starting up services inside the container how do I actually get at them how do I access the ports and um with obtainer it's laughably easy because uh by default Apper does not do

any network virtualization at all it doesn't enter a new network namespace um so because of that if you you start you know service running on I don't know Port 8888 for instance uh within your container and then you start a new window and you try to access 8888 on the host there it is the port is is it's the same because there's no there's no network virtualization which which is happening now this is an advantage for the majority of HPC users who want to do things like spin up Jupiter notebooks or you know um spin up uh maybe an R Studio server or something like

that on a compute node and then they want to access it um and they don't you know necessarily uh you know want to try to configure the network to to access it they just want to access it it would be a disadvantage if you were trying to use containers in the way in which a lot of times uh containers are used over in the cloud native space for microservices so if you're trying to orchestrate a bunch of containers into a bunch of microservices um to work together to provide an application then you know the fact that each one of the the containers that you've loaded

up on you know a single VM are all sharing the same um the same network with the VM that would be a disadvantage and so that's one of the reasons why obtainer kind of excels in the uh HPC sphere and is you know less widely used over in the cloud native sphere where a lot of this obtainer technology originally got started Dave you mentioned that that um aper doesn't have any network isolation by default I think it can have it though and if you if you do create an isolated Network for your Apper can you still map ports like that to get in and out

of your isolated Network yeah thank you for bringing that up for a more complete answer I was going to mention that too and then I forgot um you can so you can so um you can enter a a new network Nam space as an unprivileged user with Apper and when you do that all it really does is it walls off your container from the host Network and that's pretty much it uh so it basically just breaks your network what you need to do then is you need to configure network interfaces inside the container to be able to do things like Port mapping and stuff like

that and that's all privileged operations with an obtainer that's what I was going to ask if it's privileged do you get access to the only non-privileged ports or can you actually but if you're having to go back into do it as a privileged user then you obviously do get access to the privilege ports yeah yeah you have to do everything when when it comes to networking with an Apper if you want to do anything other than just like you know break the network which you can do as an unprivileged user um if you want to actually do any any real work you have to be

a privilege user to do that um and yeah and once again that's that it kind of goes back to that's not really a focus of Apper it's not something that um obtainer is really you know trying to provide uh so it's not something that's really been a focus of development yeah and that's true like you talked about the convenience reasons of wanting to run something like a jupyter notebook and just have it there and not have to worry about mapping ports uh but it's also part of the performance characteristics of fainer to a lot of the per performance degradation in kind of a cloud native

container environment comes from the flexibility excuse me the flexibility of all the uh the network isolation and having to do IO through the the virtual Stacks that connect your isolated network from the to the outside world um so yeah we use the phrase a lot fit for HBC we Sorry to talk over you Jonathan um we use the phrase a lot and I'm going to um jump into it right now because it's another one of the frequently asked questions integration over isolation we kind of throw that around a lot and that that's kind of like um one of the aspects that we're talking about when

we throw that phrase around so within HPC you know obviously the P stands for performance so you you you're really concerned about performance when you're looking at high performance Computing and so in order to ensure the Optimal Performance um we you know with obtainer we tend to focus on either integration over isolation or if you like intelligent integration where we don't just assume that everything should be virtualized but in instead we try to pick and choose what we virtualize um and match that to you know to the normal HPC user what's what's the use case that the that is going to be most widely applicable

to an HPC user and let's just virtualize the stuff that's necessary for that you don't have another one I have a question for you Dave shoot don't it some point you were working on being able to if I remember correctly you trying to VNC into an Apper container like actually running an X server inside the container and sure yeah yeah we do that um that's that's possible have you seen people do it like is there a reason to do it other than I can think it's pretty cool but is there is there a use case for it yeah so um absolutely I mean um you

know one use case would obviously be just to have a desktop within a container which is running on like a compute node or something like that um so you know that's that's one reason to VNC in another big use case is and this stuff gets kind of complicated and um honestly it's a little bit hard to instantiate but it can be done um another use case is that sometimes uh you want to you know visualize Graphics that are complicated you know you you want to render uh three-dimensional graph Graphics that have like shading and texture and all kinds of stuff like that um exactly because

I can sometimes that's the reason um and and so you know uh or you know going with the because I can thing maybe you just want to prove that you can play video games on a compute node um and so uh when you do that you need uh you need to be able to render Graphics using the GPU on the compute node and then you need like a transport to be able to ship those Graphics back to your local system and within Linux there's not a whole lot of ways to do that um one of the most well definitely the most common way to do

that is to use um a specific type of VNC called turbo VNC which has been optimized uh to try to and I'm G to you know plug Daryl Commander the guy who uh who um maintains turbo VNC here um so one way to do that is to to use something called turbo VNC which basically just takes jpegs and um compresses them and ships them back to your system really quickly so it's something that can be you know rendered remotely and then shipped back to you and this is a VNC client that allows you to do that so that's one reason to do that very cool

thank you and you know that and there can be applications you know so I come from a neuroimaging background or Neuroscience background which includes neuroimaging and there can be applications that like visualizing the data is part of the analysis and it really you know to avoid shipping the data back and forth a bunch and wasting a lot of time and a lot of bandwidth it can be really important to visualize it where it's being analyzed and so that there can be applications for which this is really kind of necessary um Mo molecular uh modeling is another application that you might have to visualize in the

same place where you're actually doing analysis very cool thank you Dave so where where are we at on this list how many questions and answers are there just curious did we even start on the list I know there's kind of a lot there's kind of a lot um yeah I don't know I you know I have a couple of demos that I would like to kind of show just s because they're fun yeah yeah so maybe maybe we could kind of do that so one of one of the questions that kind of popped up it popped up in the community recently and I've seen it

pop up several times before for and so because of that I decided to go ahead and do a frequently Asked question on it and um you know try to answer it but the the question is all right well I've just downloaded this container from dockerhub or something and I'm trying to find I'm trying to run some software inside of it and I can't find the software I don't see it anywhere so you know where is it how do I find it well this is not you know you can run into this problem for a variety of reasons but one of the reasons which is kind

of common is one that I'll I'll kind of show show here so I've got this little uh demo environment set up and I have this this um directory called hidden files so let me let me go ahead and CD into that and if we look here at wait we were on mute you have to repeat that I say I like how you already have your your shortcuts for LL set up your aliases oh yeah so I'm gonna I'm gonna look here at um yeah so this is actually just running on my uh this is running my my local machine but the browser is providing SSH

which I know is is kind of freaks Jonathan out but whatever this is this is just this is a browser with SSH one time I did this and you were like what uh anyhow if we look at the definition file for this container um I want to point out a few different things number one I'm building this container from a rocky Linux image that I originally built uh from the rocky Linux mirrors and I'm checking the fingerprint when I download this from dockerhub so even though I'm using dockerhub um as like a place to store this base image I'm not putting any trust in you

know dockerhub or in the authors of any Upstream images when I build this base image this is basically all just uh Rocky mirrors and and you know trusting myself that I did this properly all right but that's an aside the really the thing that I want to show here is number one I'm adding a user called hidden inside my container I'm creating a new user inside the container when I build the container which is usually you know this is an unusual thing to do with obtainer it's less I think it's a less of an unusual thing to do with um with a Docker or podman

or something but it's it is kind of unusual but sometimes you'll run across stuff like this U I'm just showing in the build that the hidden user has a new home directory and then I'm creating a couple of um execut able just bash scripts I'm putting one in that new hidden users home directory and it's just going to Echo out this message to say that it's that you know this script has run from the the hidden users home directory and then I'm creating another um script and I'm putting that in SL root and this is the kind of thing that can cause a lot of

havoc and confusion uh for users when they convert especially when they you know if this kind of thing is done within a ER file and then the user is is you know pulling that from from dockerhub into obtainer they might be confused as to why they can't find things so let's look at this really quick so now let's say I do an Apper shell I've already you know I've already built this so I'm not gonna I'm not going to go through the um exercise of building it again but let me go ahead and shell into it and then let's say I want to run that uh that user.

sh script I made that's not gonna that's not going to run because it's not on my path but let me try to find it so I'm gonna say okay what was it hidden um users. sh oh I don't have permissions to actually access this so if if I look at it the home directory is there but it's owned by somebody else it's owned by nobody now because the uid that this was assigned to when created the container doesn't exist on this system so appers just said okay well that's going to be owned by nobody and I don't have permission to actually get in there and

do anything with it okay so so that's so I you know if this was um like a suite of software that was actually supposed to be installed in a in a user's directory you might think about you know somebody might have created a service account and dumped it all in there well you know this can shoot yourself in the foot because you might not have permissions to actually access that service account yeah can you like run ID and just show like make explicit who you're running as and why you're running as that user yeah so this is kind of a an idiosyncrasy or not an

idiosyncrasy it's actually a a core um architectural design decision of Apper is that you are the same user inside the user as you are outside you might have noticed up here that I was demo user that's my real uid on the system1 and if I look at let's see p password so here I am I'm demo user and that entry was appended to Etsy password to create me in the container when I ran the container but here's this other user whose files I'm trying to access they're the hidden user and their uh uid is 1,000 so because our uids don't match up um this doesn't

work now this this can cause all kinds of foolishness right this this depends a lot on how your system is configured too so you might be running this at home and you might have no a you you might have no problem actually accessing this even though it's in the home um hidden directory and that might be because your uid might be 1,000 on the host and so you might actually have permissions to to access this or it can also depend I've found on how your system is configured as far as username spaces go so I haven't completely run this down but I was trying to

do this on a different system and I found that I didn't have any problem ACC accessing this and I think it was because how uids were being assigned inside the container and outside the container through the username space um Linux kernel feature so in any case you might run into this problem and you you might not but it's something definitely to be aware of Dave I'm I'm forgive me if this is getting a little bit too much in the weeds but I'm confused why it's showing up as nobody instead of just hidden since hidden is in that uh password file there uh once again it

might be the way in which I've got this configured I don't I don't know um okay it might it's probably got something to do with username space and the way in which username spaces are configured on my system right okay fair enough all right so I'm gonna exit this and so you know if I go ahead and try to access that file that I put in Roots directory you might you're probably not surprised I can't do that either let me go ahead and exit this and I'm gonna try to Shell in as fake route let me let me do that because that that should give

me the permissions to access let me pull this that's not found here okay let me um let me go ahead and try to uh access that script now so if I do that now not user let's do home hidden okay so now I can run that hidden. sh because now it says that I am root inside I used the fake root option when I entered the container and now I'm root inside the container and so now obviously I've got permission to access these files let me go ahead and try to run um that other uh that root.sh that I created says there's no such file

or directory why is that okay let me try to look and see what's in root well what's in root um now is all the stuff which is in I just exited the container my home directory so it has taken um it has taken root my home directory as demo user and it's kind of made that look like it's the root directory right um even if I I'm not going to do it right now but even if I entered this container as root so if I did like pseudo Apper and then issued the container command what it would do then is it would take the real

Roots directory on the host system and overlay that into the directory at runtime so you're going to find it's really difficult to try to find that software which was installed in SL root and that's kind of a common place not real common but um that's a place where you might find that software gets installed inside the container root um and this can be a real problem for you so if you're confused if you're trying to figure this out and you don't really know you know where this where this stuff was installed one thing that you can do really quick and easy after you've downloaded a

container is you can you can convert it basically to a Sandbox which is just a be directory and you can do it with this command and once you've done that you can see that hidden files has now been created and I can CD into hidden files root and I can find that's where my software is and so that can at least give you a clue as to you know this is where the software has been installed this is why it's confusing um you know I need to maybe change the permissions on this route directory so that anybody can run stuff from within it or um

I need to you know re recreate the container and install the software somewhere else or you know there's there's various different things you can do this catches me off guard all the time because I I I can be kind of a file system stickler I think the normal thing to do in in kind of the oci sphere the the web services sphere is to mount or create things in just a slash application name or whatever just create directories in root not in Root's home directory but in root and then manage them there but when I'm making an Apper uh definition file I want to just

store stuff in Root's home directory because it makes sense to me because root is building it and then I get caught off guard that I can't access that because you're not running his rout when you are running the container normally I think there's an option and Dave you can correct me if I'm wrong that uh you can tell Apper to not Mount the home directories when you execute is that is that a runtime thing or is that a configuration setting only uh that's a runtime so there's no home um so let's see I actually haven't played around with what would happen if I used fake

root and no home together uh am I still on the same yeah B root no home where am I oh so now presumably let's look and see what's in Roots home directory see I think what actually happens here I might be proved wrong here so maybe I should just be quiet I'm gonna be quiet and then we'll see okay no it it it does work so I think what happens though um if you do that without fake root is that it creates a new empty directory for you and bind Mount that bind mounts that into the home directory so if this happened to be in

I guess 101's home directory uid 101's home directory I think that this actually might be confusing as well because you might find that the home directory is empty and you might expect to see even if there was stuff in the container in that location right okay interesting so yeah so basically my recommendation is install stuff in opt I think opt is usually yeah the best the best method to you know make sure that you don't run into these types of problems do you still find that people stick stuff and opt I feel like that kind of went away for a while everybody's like anti opt

all of a sudden I still do that's I I like opt okay I I don't honestly I do not like the way this I mean there's there's probably a Linux Guru who's gonna like yell at me about this but I do not like the way that when you install software within Linux it takes like your configuration files and dumps them in Etsy and it takes your you know your binaries and dumps them in bin or user bin or something and it like spreads everything out across the file system because then it's like okay now I want to uninstall something how the heck do I do

that I hope that whoever created this created some you know uninstall uh script along with it they got installed with it or I hope that I use dnf to install it or you you know what I mean so I like opt because it just keeps every I mean that's the point of it right that's that that's the point of opt is that it when you've got an entire the entire tree is supposed to be in one place underneath that's you know that's what you do is you install opt yeah and I see opticus the open HPC guys their their software uh repository targets it all

the applications you install with open HPC go into slop share or something like that and then that's expected to be mounted across your cluster there's also user local you know but if you use user local I mean I think that that might be kind of an expectation too is that you're you're G to have like user bin and user Etsy and that things are going to be spread out across there so you know it it but you you don't have to do that you can just install everything within a subdirect for a particular program in the subdirectory if USS are local I don't know like

I said I'm going to get Linux gurus yelling at me about that's what I ask I feel like it's there are people that are very anti opt and I I've always used op so that's why I'm just curious where you guys fall on that one I'm Pro opt you heard it here first awesome well yeah yeah oh go sorry I was G to say there is one other demo I've got prepared but I you know I don't want to cut off your train of thought Rose well yeah I think that we are going in different directions here I was like we could talk forever but

I think we kind of got to like wrap it up at some point but if you think that this demo was like really important to like wrap up everything that we've been talking about I think that you should go ahead and do it but to yeah okay F think I want to do it anyway it'll be quick one of the other frequently asked questions that actually has kind of a different answer these days than it used to is can you run nested containers and so I just wanted to show a quick demo so that's become um in in in so once again this this is

going to depend also on the configuration of your system a little bit okay what did you say can you run what in a container nested containers nested containers okay thank so can I runer inside Apper yes can you uh and the answer used to be nah not really uh because Apper requires privilege to run and um within a container we explicitly block your ability to uh to escalate privileges because of a security you know as a security feature that's changed recently because Apper now relies on username spaces to elevate Privileges and these can be nested and yeah you can run nested obtainer containers in their

default configuration like in Apper default configuration without any problems so let me just um so is this also because I can and I want to or is there like any reason that anyone would want to do such a thing um both there so once again there there are practical applications that will require you to run nested containers um but you know uh it it can also just be because it's cool and you know I wouldn't sure that I can do it for example really quick not to interrupt your train thought Dave but if you're running like a cicd system and you want to be able

to for example automatically build appers to upload to your container AER registry it may be useful to have Apper inside of an Apper that you can then execute remotely to do like remote container builds the advantage of this being that then you can also keep your Builder image you know as a container and make easy updates to it easy changes you can deploy it out to have more um cicd Runners if you needed to um they do like Docker in Docker for example um for some of these cicd systems so this would be really useful in doing like obtainer and obtainer for those type of

systems as well let me just share my screen real quick so I've got it ready to go um here we go and um you can see here I've changed to a different directory I've got this nested dode file let's just have a look at it real quick all right this is kind of a a really quick and dirty little thing so I'm actually I'm downloading Rocky Linux version nine that's my base container and then I'm you know I'll update it and but I'm installing Apper inside of it and I'm just installing it using dnf and then my run script is to run uh a new

container and it's actually Alpine so you would expect so I've already created this container you would expect you know if I just shell into this container um and if I do a cat Etsy OS release of course it's going to be Rocky because this is a rocky base container but if I run the container so now I B I jumped back out of the container and recall that with obtainer you can run containers just using this kind of cute little um uh like um curus stick because they're executable so I'm going to go ahead and run the container and now if I cat etos release

I'm in Alpine because when I ran the container yeah so when I ran the container it popped in executed another app tainer command to run Alpine from dockerhub and this used to be like something that was impossible unless you ran the container as root um but now yeah it's in its normal configuration uh because Apper runs without privileges by default it's easy that was cool ad that is cool D that cool yeah you you ever try to see how many you can do until it just breaks is that a yes or now now chall somebody's going to yeah g do it right now it's happening

right now yeah I I assume you could do that pretty easily in a loop and you probably run into a um I'm gonna I'm gonna assume that you're gonna run into a uh limit on the number of nested name spaces that you can enter that's imposed by a configuration within the Linux kernel very cool deep rush so when you build the Rocky container the container was already built with a nested Alpine container no let me go back um let me show you again um yeah I'm gonna share my screen once again if I can get it back up so no so it actually what it

did is um right so so it's built from a rocky Linux base and then um I installed app within it and then the Run script what it does is it executes this obtainer shell uh Docker colon Alpine so what that does is it says either grab the container the Alpine container from dockerhub and run it or if you've got it cash somewhere just run it from the cash uh in this it's able to get it out of the cash because it's in your home directory which is right right and my home directory right and so maybe that's maybe That's the basis of your question is

that you didn't see the container actually get downloaded when I ran it um that's because I'd already done it before so if you know so if I did um uh obtainer cache clean I think it's Force here we go I just clean my cash out and then if I'm if I ran that container again now we're going to download Alpine and there we go very cool yeah that made it a little little more visual for me thank you thanks for the question Dave all the Daves all the Daves all the DAV asking Dave yeah he's part of the Dave Collective that's awesome okay so I

think we're gonna wrap up we're going to wrap up right now I think we could probably talk about AB chainer for a really long time it just is is awesome so thank you for all the work that you guys is do uh in the community and of course for our customers as well so if you um want to go through the longer more extensive list on our website C iq.com go ahead there's lots of questions answers if you want to get like actually involved in the community um we will put a link to that there's a awesome slack channel app chainer slack channel that all

these guys are a part of um and then if there's anything that's kind of outside of that like special cool stuff or support or training or anything else that you want to do for uh your you know with your your company and your people reach out to us and let us know we are more than happy to um engage with you and and help support you on the AER Adventure on cool all right well you guys Happy New Year and you're amazing thanks for being with us live we will be back next week same time same place less snow less well for me it would

be less snow it will definitely be less snow but not let know should have said that hold on careful awesome all right you guys so make sure that you like and subscribe we also have a podcast yes yes flops and threads flops and threads so make sure you check that out as well and uh we'll see you next time thank you thanks everybody than everybody thanks Z thank you all see [Music] you