Navigating Container Security - Apptainer vs. Rootless Podman Unveiled

and go to blogs and blog posts and there's a little area there where you can search for stuff so just put in Apper and podman um so yes thank you for all of the that work that you did really kind of giving a layout of what are the the the benefits of each each of those and a couple the drawback so that's actually what today is one of the things that we're going to be talking about is is this um obtainer versus rootless podman so I think that I mean I think everyone knows you guys but you know it's probably appropriate to like say hi

you know say your name just introduce yourself so we'll start with Dave cool I'm just making comments right now in the uh YouTube chat with the uh links to the the three parts of the blog series uh yeah it just occurred to me I can do it myself um yeah so I'm Dave godlove um you know uh if I if you uh you know so I I've been around the container Community for quite some time I used to be a research scientist and then I worked as a um staff scientist at the bow wolf cluster at the NIH for a while and um been with

HPC containers really since they started about six years ago yeah you know we were actually on a call today talking to um a university who is really wanting to like rebuild their cluster which is something that we have been doing with people and you guys have been helping with that so thank you um not helping doing it so thank you for all of that work and one of the things that we have been sharing is that we do aper training and that was really interesting to them because it's and and Dave you really are the one Dave does the aper training I think a couple

of you are kind of like learning that too but it's starting to Branch out yeah it is branching out yes for yeah Forest just uh who's unfortunately unable to join us today he just did one himself yeah yeah not that's awesome I think one of like the the the key things that I like to share with people specifically about your experience is that you kind of came to it not necessarily from a system administration perspective that you came to well originally Singularity but the concept of containers an obtainer from a like a user right yeah that's right I mean I was a staff scientist when

containers first started um but it was you know I was a user of HPC systems before I was a before I was an administrator and um so I kind of knew a lot of the challenges and stuff right exactly so you bring that breath of of understanding and depth of knowledge and so we can we can go in that direction and give um uh trainings and and help people understand how to actually use it too so it's not like the CIS admins having to do all of that training as well so yeah very exciting cool all right Jonathan who are you man who are you

yeah thanks Rose so hello everyone I'm Jonathan Anderson and I'm here on the solutions architect team as well and my background is in academic high performance Computing also but from that systems perspective so I first touched containerization through Singularity when we had an end user that just had to run a certain piece of software that only ran on auntu and uh we we were running I think a Centos cluster at the time uh and you know generally speaking that's a non starter but I'd heard of this new thing that was you know taking universities by storm uh and thought you know I think that's the

right tool for this job and someone else on the team had tried to do it and had kind of gotten lost in the weeds because of the at the time the requirement that you build things with root and so we were trying to figure out how to get get it set up like on a laptop and then move the container over but then I found uh at the time what you needed a remote build service to try and build things with and over the course of a day had a fully working container that ran their application on the HBC cluster and it was glorious and

a new day had dawned so yeah I'm I'm really excited to learn more about the the rootless version of that because the this this idea that in order to use or build containers you have to have some level of root access still persists and uh I I've enjoyed reading some of the articles that Dave's put together but I'm here to learn more yeah awesome great thank you Brian hey Brian fan here also a Solutions architect uh here at ciq uh my background is also in HPC Administration and architecture uh I've been using uh Apper back when it was Singularity around 2017 and uh yeah it's

been a great tool I've used it to onboard customers get them up and running and where they needed an extremely fast turnaround time I was able to whip up a container for them get them up and running and they were super happy about it and uh excited uh for to today's webinar where Dave is going to drop some knowledge on us drop the knowledge Dave this is this is very exciting so I mean I can pressure smart or um do you kind of have a place where you know you want to start oh um so we can just sort of Step through uh the the

the blog post I think is the best way to do this and go through it I'm you know I'm happy to just sort of like you know uh was that the question like where do we want to start talking about this yeah yeah yeah I didn't know if you wanted me to like pose questions really or if you kind of like you know where it is that you want to start because probably just a broad understanding of like what aper is right like Define and then what pod man is like a little bit of the history of where it came from of like why we're

comparing the two absolutely yeah and maybe we should step back a little bit and just talk about it too you know so after I put this together I I started to think you know maybe um this series of blog blog posts is poorly named and the reason for that is because it's it's aper versus rootless pod man with a big VSS kind of like you know you know was like we're gonna pit the two against each other and like you know let me just uh set expectations from the start and be like this is not you know we're not gonna this is not like an

adversarial kind of um uh series of blog posts we're not going to you know try to claim that one is better than the other you know both of these both of these container platforms offer a a great deal in the way of security and both of them are you know perfectly secure uh container platforms to run your containers in um you know if you set the appropriate options and you take the appropriate measures and you do the right things there there's differences between them and um there's similarities between them and I think that's kind of like what we wanted to highlight here is like where

do they both stand um what are their postures and you know their philosophies and stuff like that as far as um different security features and also security is like when I say SEC you know I hate to talk about security a lot of times because it's a huge topic you talk about maybe let's say reasonably or sufficiently or appropriately secure rather than perfectly secure sure yeah yeah yeah I mean security is like I mean there are entire I think I say this in the blog too but there's like entire companies that they're they're you know their entire model and reason for being is container security

you know so you can make a living out of this this is you know this is complicated stuff so I I'm really just going to talk about um some particular aspects and features and things and not really you know do an extensive complete uh you know deep dive of everything that there is but yeah so it's a big topic I'm just going to cover a few slivers of it and um it's it's really not meant to be like an adversarial you know this against that us against them kind of yeah you know kind of blog post and if you ever get that vibe from uh

anyone at ciq like call us out on it because that is not intentional there's no there's no need for any of that right on so yeah so I mean I guess um we could start off just talking about the two uh platforms from a historical perspective uh and kind of like you know where where they came from now the blog post suffers a little bit from the fact that I was actually there and present and in the community for the Genesis of one of these container platforms and not the other so I can speak a bit more authoritatively about kind of like the reason for

apers being and where it came from and how it was developed and stuff I have to kind of go through whatever is available on GitHub and and you know do things like that to kind of talk about where podman came from you know I was there I I I kind of remember when pod man started up and people were talking about it but I wasn't there there I wasn't helping to develop it because it was being developed um when it first started uh in at Red Hat as a as a Clos Source product when it first started so you know I didn't have any opportunity

to kind of like go over and see what was going on but I you know I'll give you you know what I think and um you know we can kind of chat about it too maybe others got have different perspectives as well um I kind of think that both container platforms started a little bit as a reaction to Docker and what Docker was doing in the space at the time but it was different reactions and different like you know um different objectives I guess so aper uh you know really started at the time as Singularity as it was kind of like a response to Docker

but it was kind of like um it was just more of a response to like Docker and other container platforms of which you know there weren't many that were super popular at the time not working very well on HPC and I kind I've told this story a lot like you know basically um you know if you were an admin around the 2015 2016 time period of an HPC cluster you were having Advanced users come to you and say hey can you install Docker I need Docker and um so obtainer you know then Singularity was really a um kind of a uh like well no we

can't install Docker in the HPC envir environment but let's just purpose build a container platform for HPC and let's you know do it with HPC in mind and let's set intelligent defaults and let's do all the stuff to make sure that this is like a nice smoothly operating container platform in HPC as I understand it podman um was a little different uh but it was also kind of a reaction to Docker it was like okay well let's let's um so Docker has some things you know to it that we find to be a little bit unattractive uh from a security standpoint it's got this demon

that runs that or kind of like manages your containers in the background uh on your behalf uh and that process has to keep running all the time and like you know you have to be part of a special group to run to to use that demon and um and you know however you kind of put it together at some point a user has to have some privileges to run containers and so let's make a and and so and I should say too that this was um several years later this was around 2018 that pod man you know started to show up and and became uh

you know something on the scene but like basically it was like let's do a drop in replacement for Docker where you can have all the same commands we will replicate pretty much all the same functionality as Docker but let's do it without a demon and let's Leverage this kind of newish thing called the user name space which em must virtualizes privilege and let's let's instead of actually giving users privilege and um you know to to start containers and do the things that they need to do to fire containers up let's um use the username space which is like a virtual Pathway to privilege to do

this instead and um so you know and that's that's uh that is that's pretty much what pod man does so the the leveraging the um the virtualization of uh privilege is specifically with rootless podman so that's you know you can run root full pod man where you just use you know pseudo and you just run it as root but you can also run if if you're on a properly configured system and you've got access to the username space you can um you can also run rootless pod man uh and you know do it like that so that's kind of the history on the two cool

appreciate that was there anything that you guys uh Jonathan wanted to add to that I the not to add I would have a follow-up question so like I I have my own perspective on this but what would you say the day I know we're we're not being adversarial here but what what remains uh a value in aper versus running rootless aside from the security things that we're going to go through is there still a use case for using one or the other in a rootless case for for both yeah so um so if you fast forward a little bit okay so so way back you

know in 20125 2016 when aper was first being developed um there were username spaces that was a thing it existed uh but nobody used them right um they were they had had a lot of uh vulnerabilities associated with them you know and they had a reputation among admins as being um dangerous and so by default if you had at that time I think um people were running either uh Enterprise Linux 6 or Enterprise Linux 7 um and by default you either didn't have access in your kernel to username spaces on your system at all or you did and it was turned off by default and

admins were like I'm not gonna turn something on that has been turned off at the factory and that I'm I'm hearing a bunch of stuff about how you know I see cves about it and I hear other people saying how it's dangerous so like and and that's kind of funny because a lot of people you know there was a lot of talk about username spaces but it was like from a practical standpoint it's just talk we can't use the username space so obtainer had to come up with a different way to allow users um that didn't have root privileges to use containers and so what

Apper did is um it you know the developers created a set uid workflow so there's a root owned there was a root owned binary within the installation that would effectively run commands as rout um now of course uh you know everybody knows that that set u ID or you know a lot of people know most people should know that set uid binaries are notoriously difficult to um develop and to maintain in a way that's safe right so like like Ping for instance ping used to be a set u ID binary um because it opens raw sockets on your behalf so it used to be that

when like if you looked at ping it it was owned by root and like it it ran his route and it they were just you know the developers of ping were just very careful to say that the only root operation that you can do within this is to open uh raw sockets and you can't do anything else and so that's kind of like you know what we used to do with obtainers there were certain privilege operations so there was a a binary which would um selectively escalate privileges on your behalf to do certain operations and that had to be locked down and you know had

to be very carefully enforced um there's other things about um Apper that are sort of different so you know if you're not using the username space uh how do you get a user inside the container and the way that Apper does that is to just dynamically figure out your uid and to write it into Etsy password and same with your G and write an entry into Etsy groups when your container starts up inside the container yeah inside the container and that's kind of like different from you know any other container platform really it's like it's that's pretty specific to uh Apper that's the way it

handles users and then um in order to like further lock stuff down um Apper also by default mounts your file system uh with a no set uid um flag on it so that you can't you know run like for instance ping I just got done talking about you would not be able to run ping or pseudo you would not be able to use pseudo uh within within the um the file system just because that's a set u ID bit program that has Cates Privileges and you can't use it and it also in addition to that there's a starter process that kicks off inside the container

and that starter proc process um sets a kernel flag that the purpose of the kernel flag is to prevent that process or any children Pro processes from escalating privileges using any means um so all that stuff is kind of legacy and is in aper just because aper was around before username spaces were like popular and in wide use um and none of that stuff's in you know podman because podman just uses the usern name space right so now I think what you're alluding to Jonathan is like so these days um aper uh the default installation of aper now no longer installs that set u ID

bit you can do it if you want to you can install that set u ID but if you want to but by default it just leverages the username space and runs you know what what we call rootless mode um but even though it does that it still has that other stuff right it still um has the uh the no new privs flag set on its new process that runs it's got the uh no set no suid bit that it or flag that it mounts the um file system with by default and um it also handles users differently it also still makes that those entries in

Etsy password Etsy groups um and so I would I would uh like so I don't know if so that that security is kind of redundant um it's not necessarily like more secure or something like that it's just like redundant it it adds different layers of security that aren't there with podman that might be attractive to users the user management stuff is also part of you know we talk about Apper having intelligent defaults that are suitable for HPC expectations so one of those is being the same user inside the container but I I would imagine another is things like automatically having your home directory or the

current Dory available which is you know in another way I think I've heard you say uh that we talk about this being integration over isolation and just speaks to Apper having a different set of priorities from most podman use cases I think exactly and then another thing I would we haven't you know I think it it's kind of like further along in the blog series and maybe we should wait to uh to talk about it but another kind of big differentiator is when we start talking about um container and ention and containing container signing and verification and like how that happens like so so podman

allows you to do those things too um but because the container format is oci that is it's a you know series of T tarballs that we call layers and a in a manifest to stitch them all together in a certain way um it's it's it's kind of like you can't really do that at the file level so instead things are encrypted or signed and verified at the registry level um whereas with obtainer your container is a file right it's a it's a C file that's got some partitions inside of it that's got you know the root file system and some Blobs of data and so

you can just you know encrypt that same as you can encrypt any other file and um that's you know that's the way in which Apper works and the same with you know signing and verification you can you can actually attach the signature to the file itself and then it can run around and you can put it it doesn't matter anymore where the container is is or where you've gotten it from or how you stored it or whatever it's still signed you know and you can still verify it so I actually have a question about that does the signature go away once you have taken that

container into your own space right because then you can make modifications to it right like say you go to Docker Hub and you get the container and now it's yours and you're doing different things to it it no longer is signed right uh which kind of container are you talking about are you talking about an oci container or a a aper container Brian I saw you come off go ahead man uh I think it when you pull it it is signed but I think when you modify it you're you're basically it should be immutable when you pull it down right but if you're modify it

you're basically creating a new container which then needs to be signed if you're G to be uploading it um back to dockerhub for example yeah and that's true with either the oci model or the C model in either case you don't so much much modify a container image as build a new one based off of a previous image and so the the original one that you're basing it off of will will carry a signature uh but the new one would have to be resigned and then that signature would incorporate the changes and the original as part of what it's signing kind of did a a

look Dave what were you yeah I mean it's it's there there is some Nuance there I I mean so like so in oci land creating a new container is essentially creating a new layer right on top of that base yeah and um so you can get back to that container you know just by kind of backtracking and and you know as long as you've not done anything weird um you can you can go back and like pull out the previous layer but yeah the signature is kind of like you know that that's a separate entity that would have to be applied during the step of

pushing that container up to the registry that's actually when you sign so like you you sign it as you you know it's not signed on it's not signed on your system it's signed it's signed in the Manifest that identifies it you're basically signing you know in like a git term you're signing the tag of of what that container is rather than yeah and then it you verify as part of the the pull process as as part of getting the container back you verify that with um obtainer the internal guts are that there is a so so you've got this CIF file and it's got several

different partitions in it and one of the partitions is um a file system and the file system is saved in a special format which is called squash FS or squash file system it's called squash file system because it it's compressed it takes all the bits and just squashes them down and um it's kind of cool because the Linux kernel is able to Mount and run that file system without uh decompressing it so it's like you know you can mount it compressed and you never have to like explode it back out and waste all that disc space and do all that kind of stuff but the

trade-off is that it's immutable and it's like I mean there's no way to make like the technology does not allow you to write to it so if you're going to change um if you're going to change your container you you you essentially just have to build a new container there's no way around it you have to build a new container and so once you've done that like I mean you could you could through the magic of the sift tool grab an existing signature and add it to the new container that you have built but part of the verification process is to run kind of what

like a check sum and check and see what's inside the container and see if it matches you know what was used to sign the container and so if you just jammed one in there like that it wouldn't it wouldn't verify anymore it say nope this container's changed I can't I can't verify anymore so in either case you've got to resign it yeah and I think that's the Crux Rose of your question it's not so much is the signature physically on my disc when I pull a container down and is it still there if I make changes it's is that signature still valid and in either

case whether the signature is coming from a manifest from your registry or if it's encoded in the CIF if you're making modifications you're conceptually making a new container and the original signature would be invalidated you have to to resign the new content but I guess like the difference between aper and oci then is like if I have a container sitting on dis with aper what that means is I have a SI file somewhere that I can do file management stuff with and if I want to to either sign or check to see if that container is signed I can do that without any you know

external anything else else in the oci world um if you have a container sitting on dis what that actually means is you've got some directory somewhere uh which is you know is managed on your behalf which you're not really looking or not supposed to look at yourself that has a bunch of tballs in a manifest and you know entire directory structure um and you know if you want to sign that there's no way to sign it on disk and if you want to verify that it's signed there's no way to verif ify it on disk you have to have an either a local registry that

you've got running or you have to have an external registry like dockerhub that you can push and pull to in order to do because that that whole signing and verification process is part of the pushing and pulling so that's kind of like the difference so to kind of wrap that back around into the topic for today it's does so that's we're talking about Apper specifically does podman allow for signatures as well do they do that yeah yeah yeah so that's so that's um you have to kind of go through to the third part of the um of the uh the blog post post yeah to

to see this to you know to see I think it's a third third Parts yeah yeah yeah so if you go to the third part of the blog post um I could I guess I could share it real quick if that's helpful um you know I would I'm going to be like fully honest here I would um give everybody a live demo of all this stuff except I did it all in uh a VM that I kind of forgot where the VM is it now I a lot of different VMS and and there's like some setup that you have to do ahead of time that

I you know failed to do so sorry uh not gonna wait there was there was one uh live show that we were doing where something what something happened where you had to like actually take your camera and what was that I for da I don't want to talk about it anymore I don't want to talk about it you got the live bug it happens it totally happens did I share can people see this now um I don't know I think Alex you I think Alex needs to do something from from the back end cool cool yeah okay yeah giving me PTSD thinking about that I

was like oh my God it's not gonna share all right so um yeah so so so this is the third part yeah if we go up here this is like part three which kind of walks through this stuff so how we cryptographically assign and verify containers based on you know if they're if they're CFT files or if they're oci and it uses you know oci is like can be a lot of different tooling but it uses pod man here because that's what we're talking about in this blog post um so yeah so obtainer containers you sign them and verify them um excuse me there is

one part of this that I I I didn't talk about before and that is where do you get the private so a signature like a a a key has two different parts to it's got the public and the private uh part to the signature key so and you need when you're signing a container you use the private key to sign it and then anybody can then verify it using the public key material so the question is where do you get that public key material and by default um you can push the public key material after you're' done signing something and by default obtainer is configured to go look for that material at keys.

openpgp dorg and so for instance if somebody here were to pull one of my assigned containers sift files from um dockerhub and do a verify under the hood Apper is going to go up and grab that um that open uh the uh the public key material from openpgp dorg and then just you know it's going to go ahead and verify and spit out the fingerprint and you know show that it's been signed um and then you can import that and have it all local you could you could you could verify the container without having any um without having any internet connection you know so so

here's kind of like the the workflow um you you use the uh key new pair commands I'm G make that a little bit bigger to create um to create a new pair of keys and that's that pair is like the public and private uh pair then you can list that you've got that key you know so here it is and here's the fingerprint you can sign it and uh then you can go ahead and push that key just so that anybody then can ver you don't have to do this part to verify your own containers because youve already got that that key you know Sav

locally in your cash but now anybody you know who gets a hold of your containers uh can go up there and and get your container and and verify it um and so then this is the other half of that so you pull you pull an image or you have an image or whatever um and then as a as a separate kind of thing so I'm showing this LS here just to show that after you pull this image you actually have a have a file that is the container and then as a separate thing here you know you do this verification step and uh it's going

to end up telling you what fingerprint it was signed with and so on now if we go down to um yeah so if we go down to podman and we have a look things are a little bit different so you use external Tooling in pod so podman this is another this is not Lally a security related philosophy thing but pod man's um philosophy is a little bit more like a lot of other Linux Tools in that you should do just one thing and then if you if you need other functionality you should have other tools to do that so for instance um podman you can't

really build containers with podman uh this will maybe surprise some people who are used to doing podman build but actually what happens under the hood is there's another tool called builda which is called bipod man to build containers so it's kind of like everything is modularized and kind of like you know subbed out to different tools and stuff so if you want to um if you want to create a signature uh with pod man you need another tool in this case scopio so I'm GNA go ahead um use that to generate some keys and so then you know here's the here's the keys I can

actually see the keys that I've just generated uh and then you've got to set up some configurations so you have to tell um you have to set a configuration file in a directory which and it doesn't have to be in Etsy it can also be in your home directory but in this case it's it's it's an Etsy that I've set this up to say use Sig store attachments equals true and so you basically have to set up your registry to say hey when I push stuff up here I'm going to be signing it and so then you just go ahead and tag it and then

you push it and when you push it you say sign it you know with my key and here's the location of my key Dave I want to make sure I understand that so that uh Etsy containers that's a a configuration for the local registry that you're running that's a a server side setting is that right yeah yeah in this case with this example it doesn't have to be but with this example I wanted to show how to do this without you know involving dockerhub or sure you know some other uh public registry so in this case what I've done is I've actually just set up

a private registry which is listening on Port 5000 and then I'm configuring it uh once again this is the global configuration but there's also an option to do just like a personal local configuration but I'm configuring it to to say uh you know we're going to go ahead and sign that that this this is going to accept and store signatures this this registry is going to be expected to accept and store signatures this is is one of two different uh methods that you can use in pod Man actually to sign and verify um your containers the other one involves it's kind of an older workflow

and it involves setting up your own lookaside server which you know I didn't do uh because just handling the signatures like the the containers in one place okay yeah and so um you know obviously I think that's a little simpler you're going to have a registry uh you know if you're working with oci containers so why not have the registry store your signatures as well but it's I think it's two different container form or uh two different signature formats as well I think the Legacy doesn't use the same signature format and I don't know if there's an advantage to using the older format versus the

newer one or I didn't get quite that deep um but yeah so this is this is kind of the workflow and then of course then when you um when you go to get the uh the so this is a little different too so when you go to pull the container um you have to kind of say you have to kind of set another configuration this in this case I'm doing a local configuration and I'm saying that um I'm I'm telling it what type of signature I expect to be on the containers that I pull and then I'm giving it the path to that public key

material that I need in order to be able to pull the containers and so now I mean that just that basically sets everything every time you pull anything it's going to check for that signature um and so you can't you can't really pull anything that's not signed you know and doesn't correspond to that that public key material um and so then this is what it actually looks like uh so I'm just going to remove the container locally so that you know that I'm reping it and then I pull it and uh you know I don't have to add anything here that says you know here's

the location of my public key material because I already did that up here and so there's no option or anything it's just that everything that pulls now gets checked and that's a little different too like um when you sign and verify containers with Apper you know you can have you have 10 signatures on one container and you can you can do things with obtainer to check for instance that any one of these 10 signatures is present or only run this container if all 10 signatures are present or you you can do stuff like that there's not really as much flexibility as far as like at

least from what I've seen in my experiments with podman uh there might be ways to instantiate that but I'm I you know that not I can imagine maybe you could do that kind of thing with the other path with the look aide server if these signatures are something that's discret and you could just check those things independently yeah maybe it's just me surmising I have a a couple of questions outside of the signature uh context if that's all right Dave uh or did you have more to to share there with no we're going I know I'm going real real deep into like just one particular

aspect of all so so like when I think of username spaces I'm still I'm still kind of new to it I've I've used it but I know that one of the things you need are are these Etsy sub uid and subg files and my understanding there is that it's an authorization of allowing a user to use some typically High numbered part of the uid and GID uh kind of number space within their containers um can you talk at all like are there any differences between how podman and aper handle that and handle management of those files or or use of those files yeah that's a

good question um yeah so yeah you're totally right so there um the way in which the username space is ideally supposed to work is that when you create a new user um you get you get a new entry in the configuration file Etsy subid and Etsy um subg and those entries like if you use on a Linux system just like the ad ad user utility um those entries will be added for you uh automatically but it's kind of a pain because um a lot of times in HPC we don't use like the add user utility to create a new user right we we're and we

don't even we don't even give them an entry and Etsy password a lot of times we're like you know they're just managed completely through uh ldap or ad or you know whatever we've got a an external uh authentication that we're using to create new users um as far as I understand it so uh the way that um podman works it really requires that so so what that entry is doing just like you said it's giving you a range of uids or GS that your user is now allowed to use and allowed to use as an intermediate mapping um from like basically it needs a placeholder

uid or GID on the host system outside of the username space to map you to or to map you know different users within your container to or whatever it might use a lot of different uids and GS inside your container um so you know as far as I understand podman requires those uh to be useful um to to to leverage the username space into run in rootless mode Apper once again um Apper has been around for a little bit longer and has kind of suffered from not suffered from has like obtainer has been designed to run in a lot of different environments with varying levels

of support for username spaces uh all the way down to no support right and so it could be in your HPC environment that you you know you you don't have these uh these mappings and you're not going to have these mappings and so if that's the case there's a whole kind of like Cascade of um ways in which Apper will try to leverage the username space uh with less and less um with less and less um functionality basically and so um so so if you if you tried to use like fake root inside of a uh container in order to try to make yourself the

root user of uid zero but you are running you know on a system that had the username space enabled but in which you didn't have uh Etsy sub uh subid or subg configured it would try to work around that and it would try to work around that by doing things like potentially um by mounting in the fake root utility into your container um you know and that that's one of the different things it'll try uh if you look at the um the app taner documentation uh about fake root you can see there's a section in which you'll go through like here's the first thing it'll

try if that fails here's the second thing it'll try and so on and so forth and the the the upshot of that is that uh in a lot of environments you know not all of them but in a lot of environments where everything is just not configured exactly the way uh the username space wants it uh a lot of the usern namespace functionality will still work and users can still get done what they need to get done that's cool I'm also looking now because I I realized that ultimately the issue we're talking about users not being on the system and so you're not using ad

user to create them that uh they're they're coming out of L app right or some other external Source uh it looks like upstream and I'm I'm letting myself get distracted here while we're trying to have a conversation but there's a conversation Upstream about supporting um subid and subg in NS switch to allow pulling those entries in from elap as well so that's that's kind of cool um my other question was I I know that along the way you ran into some issues or confusion around SE Linux and I was wondering if you could share a little bit about what you experienced there and and how

aper and podman handle SE Linux differently sure um let's see um yeah so and that's actually you know I would I would say that this is probably once again we're not being adversarial but if we were keeping score I would say this is one place in which pod man probably comes out on top right um so the the the problem that I ran into is just by default when I mounted uh new volumes into a podman container and then tried to write to them I was getting you know SE Linux issues I've also experienced this and been confused yeah and uh you know at first

I was like you know what do you do so I'm in the VM and like I'm like ah this doesn't work turn off SC Linux worked okay now I'm happy um but I decided no I can't do that man I'm like I'm trying to you know I'm trying to get a little bit deeper and actually understand this stuff and so I started talking to um probably our most senior developer one of our most senior developers certainly Cedric uh and uh I asked him about it and he was like well maybe there's just like a like maybe podman by default sets up an SE Linux context

for you when you enter the container and I was like oh that's something to and so I started Googling around and looking through the documentation and yeah that's exactly right so there's an SE Linux context that um that it enters by default which prevents you from making any changes to volumes which come in from the host system which is kind of nice right especially if you're running a container for the first time and you don't know what some untrusted container that you don't know what it has although you did tell it to bind in your home directory so maybe you expected to do that but

I maybe not writing it well well that that's what I was about to say too I mean um your your your home directory gets buy mounted in by default right with obtainer right but not with po ban yeah yeah yeah so like so that's actually a nice feature to have because with obtainer um if you download some Rando container from dockerhub and you don't know what it does and um it's and you run it and in the in the um the the entry point of the CMD It's gotta remove you know Tilda uh Slash dot or something in it then you just blew away everything

in your home directory right and you didn't like that that's if you ran it without any without any options or without any anything it just did that with with podman even if you buy mounted your home directory in unless you gave it um the correct option to tell it not to um not to enter that se Linux context which you know I guess a lot of people probably just do by default but unless you did that it would not just delete everything in your home directory with Apper it would so like I figured out based on the documentation what SE Linux context was being entered

by pod man and then I figured out so you can do the same thing with obtainer but I've never seen anybody do it before and it's kind of like it's a pretty you know messy option argument pair to do it but we there is that that functionality there it's just once again a case of just being different you know that that functionality looks like when you say obtainer run you can specify an Linux context to join is that what you're saying yeah yeah yeah I could actually show it to you I've saved it in the in the blog post here so let me just Cee

it up here again cool so whenever Alex is ready we'll here we go thank you sir so um yeah it's just right here so um there's a security option yeah within Apper which allows you to basically pass strings and you do a kind of a lot of different things but you can um you know you can set the SE Linux context with uh with this string here here where you're basically saying this is the context I want and this is this happens to be I'm not you know this happens to be the exact same context uh based on the documentation that podman enters just by

default and it's pre it's preventing you from altering files on the host system um so if you did this now you know you like you can't you can't change files on the host system um that have been bu mounted in um you know just by virtue of the fact that they're bu mounted in so it once again it's like we talk a lot about this I mean it's the same feature set in this particular case I mean that that's not true across the board we kind of touched on some things that are fundamentally different between obtainer and podman but in this case we have the

same feature set we just have different defaults and the default um in this case in podman is is more secure is can you specify that security option like in the aper config file could you set the the podman default as an aper default a site local config that's a good question I don't know if you could or not even if it's not there that seems like a reasonable thing that could it would be pretty straightforward to add if you can almost certainly do it through environment variables okay so yeah so so you you you could make that you could force that config somehow yeah yeah

so I know that you're in the Apper um slack Community as well so that's where um if you you guys are are out there and and curious and you want to get involved with the community one of the things that I like to to remind people too is like yeah the community is there to help you but you are also part of the community so jump in as well and you know and and and be a part of it they definitely want you there so that that's easy again to find on our website um just go to the uh uh open source communities and and

go to obtainer and jump on that slack we'd love to have you um but so Dave I know that you are active in there as well as at ciq really working on some some things that are specific to certain customers if they want something customize or a training or something to work out special with oper so what are um um like some of the like best use cases that you have seen for for aper where it just is like this is definitely made for you and it just works exactly as it's supposed to oh yeah so like yeah the I mean well yeah I'll let

everybody kind of talk about this um you know but I mean it's kind of a no-brainer to me but I mean AB tainer is really built for HPC I mean really HPC use cases are where it shines um and you know and that that's a good that's good for this discussion as well because you know um it's it's got intelligent defaults and an architecture which lends it uh to to to using HPC quite easily right um podman comes from kind of a a different culture right kind of a different world so podman is um is a container platform which has sensible defaults and features and

architecture for the cloud native space and works really really well there and I would argue that you know obtainer is probably not a very good fit for a lot of the kinds of things that people do do uh in the cloud native space we've you know through development we've gone through and we've we've kind of made um we kind of went through a phase where we were making aper more attractive to that space and we got like you know the community was kind of like not super crazy about it uh you know they like to use aper for HPC that's where the community is at

so we're going to continue to kind of develop aper for you know what people want to use it for um and then the flip is probably true as well I mean um I I am not if you read this blog post you'll probably figure out pretty quick that I'm not like a power user of podman um you know I can I can get done what I need to get done but not like aper I mean I don't use it on a daily basis um and I would I would say that like you know it's just if if you're used to using containers in HPC it's

just kind of clunky you know like uh you there's this whole different kind of feel with any oci container uh platform I I think where it's like you're more hands off with your containers and a lot of people kind of you know talk about that and that being a benefit that's what you want you want your containers to just be like you know cattle that just run off and and do stuff and you just around but I mean me like I want to I want to see my container when I pull it I want to have it you know where I can change its name

and move it from one place to another I want to enter the container with shell and not have to remember you know what the particular options and arguments and all that stuff is that I you know get a shell inside the container and you know I just I just want that convenience to be able to just like be Hands-On and do stuff with the container and not like have this layer of indirection in between me and the container that I have to like tell you know something else to go do something with the container and have it done um but that's just you know that's

what I'm used to and that's why I like coming from HPC and that's you know that that's what works for me cool yeah I would broaden that just a little bit because you know AB Tander comes up from HPC but it's not the high performance part there there are performance benefits to obtainer for sure that make it a good fit for HBC but it's not the high performance part that limits aper applicability it's the the kind of Unix family of philosophies where you have tools that you execute at a command line that participate in a pipeline and kind of have access to a shared file

system obtainer integrates really well into that ecosystem of tools because that's what the HPC ecosystem is you're you're a person sitting at a Unix command line or a Linux command line and you run Tools in a script that interact with each other through the file system uh podman being part of the the kind of cloud native um philosophy in community their practice is Services rather than a shared file system so they want to run a process and then have different processes interact with each other over sockets in the network and so if you're if you're running a service or an appliance or some kind of

application that's kind of isolated and you prefer that isolation over an integration of tools that work together um then that's where podman really shines and and its Integrations with systemd uh even make that more obvious uh in kind of a modern Rocky Linux context um but you can use Apper for that as well especially if what you're running are services that are meant to integrate with that local um shared file system uh so that there's overlap for sure um but it it to me it really comes down to whether you're writing Network Services that talk of the network uh almost exclusively uh or uh you're

operating in a command line and that command line is your ecosystem yeah that was that was a nice um comparison thank you Jonathan Brian uh I think the use case that uh appeals to be the most about uh Apper is the ability to have your container encrypted on dis and this is really important in the HPC context especially if you're running you know export controlled types of code where it needs to be super secure needs to be encrypted at rest and obtainer gives you a way of actually doing that uh this is important in areas like cfd if you're running you know supersonic simulations uh

stuff like that yeah awesome I'd like to tack on to something Jonathan said too um so you were talking about the difference between aper and podman as far as like yeah and that's a that's a key difference I'm really glad you brought that up as far as like uh you know podman coming from a world in which containers are expected to like interact with one another through the network um and that you know so and then you said you could kind of do that with uh aper as well but I also want to highlight there's some there's some fundamental differences in what an unprivileged user

can do uh with with networking with podman versus aper so that's another thing so like um because of that different emphasis um like the network Nam space and network interfaces within the network namespace and all that kind of stuff are not really very well supported honestly with obtainer um so you've got to be you've got to be a privileged user to do pretty much anything useful with um with Apper uh on the other hand um so uh podman uses something called uh slurp for netfs and it allows you to like it allows you to use an unprivileged protocol to kind of spoof a lot of

privileged operations um within the container essentially it you can't do everything and the the performance evidently is degraded by doing this so it's not it's not as performant but you can do a lot of unprivileged networking with podman that you just don't have access to and just can't do it all with with app tuner once again not an issue for most uh HPC users who don't care about that stuff but that would be a tremendous probably issue especially if you were trying to do unprivileged of within a cloud native environment that would be like a problem right well guys I think that we could probably

go on forever and ever but we have come to the end of our webinar so thank you so much all of you for being here and for sharing with us thank you for everybody at home and everyone who's listening and watching we'll be back here same time same place um but always we we want to talk to you guys right so here is our our sa team plus a a couple of others Our Little Dream Team here that is more than happy to dive into you know what exactly your environment needs right and what's best for it and how we can help support you and

um so just you can reach out on our website C iq.com and we'd be so happy and also of course I'm sure that Alex you put a little Link in there for them for everyone to um join the app chainer slot Community if uh you want to be part of that community so thank you guys have a wonderful day and we'll see you next week thanks Rose thanks Alex [Music] bye