Apptainer (formerly Singularity) 1.1.0 Has Been Released

admin on their high performance Computing architecture there with our research Computing team I'm just looking for ways to kind of install packages for researchers install software for researchers that was um you know something a little bit different than what was out there and ended up finding Singularity used it to deploy a lot of workloads and uh now I'm uh you know the container guy of sorts um with regards to some of these things so yeah I've been a user of Singularity and Apptainer for a while and uh really pleased to see Dave and of course they have two major opinions in the community on here

as well it's great and so Dave Dykstra whenever Dave God love refers to you he calls you Dr Dave but when I see Dr Dave it's really both of you because you're both doctor days so I'm going to refer to Dave in this context he's going to be Dave doctor God love I'm gonna call you God love because that's what I call you around here because we have so many tapes so let's start off and talk about what what are we announcing I know there was a new release of Apptainer so let's let's talk about what the release is and then kind of dive into

a little bit so I will go with Dave Dykstra tell us about the new release all right so the the big change in this release is that it's now rootless by default so we aren't not installing this at uid Route component and this is possible now or practical because we are using unprivileged user namespaces and more places than before most especially the mounting of zip files can be done unprivileged uh completely unprivileged and and then it also does uh some sometimes use features of mounting of fuse 2fs mounting the uh ext3 files images and also uh overlay uh which most of the time Apptainer

does not need to use overlay but sometimes we if you want to like extend especially if you want to extend a uh an image you can do that with overlay and then the other big thing this time is the unprivileged build so it always used to be you had to be root in order to do the build and now it can be done as an ordinary user it's fantastic so Dave godlove what does that mean for the end user how is this going to help and some of those contexts to security and workflow oh well I mean the building aspect that Dave just talked about

is is really uh huge for users um so in the past um it used to be that uh you know if a user um you know wanted to build a new a new uh Singularity container or apptainer container they would be required to have their own Linux environment in which they were root right um so uh you know to to do that um they would either have to you know be able to spin up a VM or maybe have a cloud instance or maybe have a dedicated resource that was you know their laptop or something like that and um you know now I mean that's

that's a significant hurdle to you know an average HPC user who maybe the only Linux environment that they typically have access to is that within the within the high performance Computing system and obviously they're not going to root there so this is going to allow um people to build you know right in the production environment where they normally work uh and that's you know that's that's really huge um also as far as not needing to be root uh you know to I mean not having that suid root owned SU but part of the workflow I mean oh hey Krishna Krishna that's also uh from an

admin perspective so you know now we're leveraging um username spaces which were included I believe in uh you know Red Hat seven and are on by default in rel7 and Rocky or relate and Rocky eight um so they're enabled by default uh and so now I mean you know it's it's it's a a Linux feature which is already there and I think that one of the cool things about that too is that it extends um you know something that Singularity and appertainer have been doing for a long time which is using you know Linux feature Linux kernel features and file system features and things which

are already there and not doing anything exotic as far as trying to um you know uh put Security in in but just utilizing these features which already exist and which are already well known and well used and well trusted to enable security I think that it like it it um continues that tradition which is really cool great thank you Dave welcome Carissa hello hi everyone would you like to introduce yourself and tell us how how you were involved in the community for Apptainer hey so I'm Krishna krishnamuriki I've been working with uh Greek Godzilla for more than a decade now um we initially I was

employed at Laura's Berkeley lab along with the records and uh being involved with this project from the early days of Singularity um and over the years the project evolved more of them and I tagged along with it and when recently when app trainer was forming Greg was looking for volunteers on various activities related to it and I I put my name in I don't think I'm doing enough even though my name is there but yeah I hope to contribute more and right now I am listed as the official release manager although day Dave has been doing a lot more than me uh right now other

than that I am employed with right now I'm employed with ala Corporation and we make wafer inspection products um it's huge like powerful microscopes in other terms uh we're looking at papers at Nano scale and identifying the defects those are the products we make microscopes with huge cameras and a lot of data coming in and consuming enough clusters and we are as a company as the products that we are we are interested in our dropping containers and the organizational organization things that planar is the right way to do it and that's why even with that hat I have we are interested as any employee we

are interested in contributing to the community and being able to Leverage the product that's coming out of the community there is a huge interest in attainer who have been keeping eyes on all the features that are getting released with the current 1.1 release um yeah we want to use it the products that Ana is about to ship in the next few years it's fantastic thanks Krishna Dave I think you had some something else you wanted to add to or talk about when it comes to security yes well first I want to say also Krishna is also on the technical steering committee with me and uh

and a few others uh including Greg um so yes I wanted to add so there did used to be able even even before this release used to be able to do builds unprivileged uh but only with the fake root option and it was that fake root option required the system administrators to set it up with Etsy sub uad and Etsy sub GID so uh so a lot of most you know people haven't had to experience that their system administrator had set up that mapping for their user ID there and so now in this new release the fake root option is extended so even if you

don't have those things set up you can still build you can still run the fake root option uh and and it does that by using unprivileged username space and then also using the fake root command which kind of fakes out the let's let's build and package installs and things think that it's running as rude when it really isn't but it's sufficiently so that so that uh for building packages it works great for that great thank you Dave so how have we actually gotten to where we are so I know it started off as Singularity a lot of things have have been put into the product

the project and then over time it it moved into kind of where we are today so Dave can you kind of tell me that story of how we got here you want to know the whole history or you're in particular about this release no let's hear the history and then we'll talk about this release further like how did we get to where we are today and why is this release important to us in the beginning there was two roots that's true it's well in the beginning Greg wrote Singularity two uh we had written in C and then then it was it was it it grew

a lot in popularity in fact it grew a lot and popular already at that point in our community in the high throughput Community Computing Community as we saw a lot of potential of going especially we needed to be able to use it as unprivileged users who were running pilot jobs at at sites and then those would then run user payload jobs and so we needed this type of capability uh we from the beginning saw that things were soon then moving to totally unprivileged it didn't we didn't need set uid root and so we thought that would be the way to go and so uh some

people from our community especially Brian buckleman started contributing at that point and and we started using it early but in Singularity too and then um Greg Forum Greg left lbl and and started the scilabs company and uh they rewrote it with uh and go and see and uh then uh and made a whole lot of new features and including during that time period we started using in our community started using it on privilege and we were able to do that because we use the CERN VM file system distribute our containers we don't need Sif files so the and um so so it doesn't need to

have to mount anything so we for for several years have been running unprivileged and um but it's long been my goal a personal goal to be able to have have it be by default unprivileged because we know that that's the way the kernel support was going and because that's where uh other tools have gone and so I've I've requested a few years ago hey let's let's make it totally unprivileged and uh so uh well then uh Greg left side labs and and took the project with him and then after about a year scilabs decided to make a fork of the project they call it Singularity

CE and that was very confusing to the community to have a singularity and a singularity CE so then then Greg led us into uh to to the Linux foundation and then Linux Foundation required a name change so so we got the name change of apptainer and then so since that time then I've gotten more deeply involved and and and so I I thought my my top priority for for Apptainer was to make it totally unprivileged so I've been I've been putting a lot of effort into that in the last six months correct thank you and Dave you haven't really said it yet but I mean

um Everybody everyone should know that uh I think that it's not maybe official but you've pretty much assumed the role of the community um manager or the community liaison and also of the release manager so Dave is the one who's been running the uh the meetings uh the the community meetings that occur um two times per month and he's also been the one who's been you know setting the milestones and figuring out you know what real excuse me what releases we're going to contain and what they're not going to contain and uh kind of prodding people to get things done and so he's really he's

really kind of uh jumped in and filled that role that has become empty so thank you Dave for for doing that absolutely so before we go into that more Dave Dykstra when we start talking about being able to run things unprivileged and we talk about security what what use case problems does this solve or what is it what does it enable for an end user that is important and why you were pushing for this for so long well set uid route is notoriously difficult to to secure uh you know I think um site Labs the developers there that and and Greg and the original one

did you know took a lot of care to try and make make sure that was that was Secure but it's still notoriously difficult to secure and uh there there can be a lot of attack vectors on that and um so uh it's something you try to try to avoid when we can now on the other hand the uh these unprivileged user name spaces has had quite a few cves in the kernel that say exploits come along which can be exploited through the use of unprivileges and namespaces and so in our community we and also we've now recommending to the to the whole Apptainer community as

we advise turning off network name spaces because almost all of the cves in the last several years that have that can be exploited through unprivileges and namespaces have had to be done in combination with network namespaces so if you can avoid it and and it is used by other container tools by default podman Docker use it but if you don't need them or if they they also can be given an option to not use it there's also used a little bit by systemd but those can also be can turned off various packages of systemd used network namespaces but you can you can disable that and

we give instructions on how to do that so we think it's a it's a it's a good security trade-off it's also the difference between a small number of people looking at it versus a large number of people is a large number of people a huge number of people are trying constantly going through the Linux kernel looking for holes there's hardly anybody's doing that for Apptainer so it's a question of you know security by obscurity versus security through the experts I'll say we want to have the many many eyeballs on things because that's the way you make it ultimately secure absolutely so Forest how do you

see this change again I know you spent a lot of time building containers and Apptainer and then executing them in another place but how is this going to be different for you well in a lot of ways um things are somewhat similar as was mentioned to what you could do with that fake root option um to be able to build things unprivileged but this just provides a lot more freedom with how the user can be able to build their containers on privilege in that environment this gives users a lot more capability to build their containers perhaps from the context of a cluster um you know

as we mentioned it makes it easier they don't have to spin up you know a rude owned environment or anything like that to be able to get on and get working with Apptainer now so in general um it just makes it a lot easier for the users to get up and running with creating their own containers there's no concerns about root there's no um uh you know sacred has been expanded and stuff so it can do more now and yeah just in the end it makes it easier for the users to get their um workloads containerized and up and running on their architecture and there

are some technical considerations um kind of you know that we're exploring with different things you can do with an Apptainer um now that it has some of these capabilities around um you know cicd things like that um so we're looking into that but primarily the biggest thing I see is that this makes it a lot easier for the user at the end to um not have to be concerned with the security model they're dealing with and just be focused on getting their application containerized all right thanks Forrest so we do have a question from Sylvie so what is the biggest challenge to making antenna run

without privilege leave that one of anybody who wants to answer it I think that's a probably a fairly deep question I can I can answer that so absolutely so um well the you know at this point of Kernel development that really there was nothing stopping it except for time to develop it because they were already available uh I mean you can run the um can do mountains so the mounts is the biggest part of uh of what was needed to be able to do um to do mounted containers you want to be able to mount a squash file system for example for that was the

and and that was a privileged operation until unprivileged fuse was allowed in the kernel so that was from inside and unprivileged username space you can then do fuse mounts so there was already a squash fuse available uh which is was was uh so it really to do uh is tie that into into Apptainer and um uh then you know a lot of different combinations to test now it did turn out and this is something we didn't find until late once people gave us benchmarks to run it turned out that squash fuse in parallel mode was a very poor performance when there's lots and lots of

files to be open so you're running in lots of cores they're all opening up lots of files with the same same single squash file system that turned out to be extremely slow and but but that was fixed in the final release by uh by using a multi-threaded patch for for squash fuse so that was that was the one that was the potential killer to the whole thing but but um very fortunate to have that that multi-threaded patch ready and it was uh it um worked really well thank you Dave so Dave God love so for those who are less experienced in all of this can

you you want me kind of through this step by step of why privilege is needed I'm really glad that you called on me when you uh you know needed a less experience Viewpoint that's I mean it's really me it's about me you did the right thing um so um yeah so this is a great question so so I I was as we were talking um here I started to think to myself you know there might be people wondering right now why why is any of this privileged at all why do you need I mean so there might be even people who kind of didn't even

know this was privilege I guess so um you know as a user of an HPC system when you run the Apptainer command what you're doing is you're leveraging there are parts of the workflow are were parts of the workflow up until recently up until now that were running an executable which was owned by root and had the uh the suid bit set which was basically making it run as root you were you were becoming rude even though you didn't know it for some of the operations that you were running and um those operations were running on your behalf now now that's not that unusual

of a thing there are other programs that do that on your behalf that allow you to safely do things within the context of a particular program that normally only root would be able to do but as Dave alluded to before that's something that's that's pretty it's hard to lock that down and ensure that nobody can leverage that for nefarious purposes so in the case of app Tater um you know what what really kind of happens behind the scenes is that you've got um you've got a file system sitting in a little image somewhere uh typically within a sip file and inside of that within a

squash FS image apptainer locates that file system and it mounts it to your um to your host file system now uh and then after so that that is one of the parts that um up until recently up until you know the work that we're discussing today needed to be done as root because mounting a file system onto the host file system is is um something that requires Escalade escalating privileges then after uh you know uh after that occurs um or perhaps before I'm actually not totally sure of the order of operations but uh applicator allows you to Pivot into that new file system within a

newly created namespace which presents that that file system to you as though it were the root file system um and so there's a the uh the basic operations which are which are necessary there's other there's a lot of other things that occur but those are the big things that occur to set up um an Apptainer uh container and so that mounting operation that's a that's a privileged operation um and so you know now with these new developments there there are ways to be able to um mount a file system without using privilege and these are the things that uh that Apptainer is now um that

Apptainer is now leveraging for building for building containers so when you build a container you are installing packages so it's like you're doing a yum install that's usually a privileged operation it tries to write into the into your root file system and such so so that is a privileged thing as well so so that so um you know that's why we needed to have either either give it a fake root or a real root in order to do that and uh yeah it might be worth noting too that um the building operation is not inherently something which requires privilege uh and you know a long

time ago when we were kind of you know uh first coming up with um you know that was one of the things that I worked on too is a long time ago um helping to to put together the build commands to to you know create uh create new containers and while I was doing that I sort of had um the idea that you know build um that building a container and I talked about this a couple of weeks ago a little bit building a container entails the operation um can be lots and lots of different things you can build from lots of different sources and

you can end up with lots of different things depending on you know what you want to build and what you want to build from and not all those things have to be privileged and even currently within apptainer if you want to build a Sandbox you can do that on privilege right because all that requires is taking a image and dumping it out essentially or you know you know filling up a directory so we for a while we thought well maybe we won't require you to be rude or won't require you to build with privilege and we'll just see how far you can get and maybe

some things you might be able to build without privilege um but in practice it turned out that almost anything of consequence that you want to do when building an image you have to be a root user you have to be a privileged user to do that and it got kind of confusing to just allow users to try to build things unprivileged as we were doing testing during development so um we basically ended up putting in like a check to see you know are you root um when you when you when you go to build something but another thing another interesting topic to bring up or

an important topic to bring up is that um this came up several years ago as well uh building a container and doing so with whole root privileges is actually a pretty risky proposition um you risk the Integrity of your host operating system your host file system when you do that um you can uh I touched on this a little bit a couple weeks ago too and I was when I did the whole um thing on on building containers uh but you can make mistakes using the wrong sections using setups like setup and pre for instance you can make changes to your funnel system um and

that's a you know that's something that you can pretty easily guard against just by not using those sections but there are more nuanced and complicated things that can get you into trouble uh recently it became possible to bind Mount directories from the host system into your container at build time which was previously always uh you know disallowed and I have a few times now had a apptainer binder environment variable set and forgotten that I had it set because I was running containers and gone to build a container and said oh I don't have privilege for right here why don't you or you know what's going

on because I've used fake root instead of real root but if I if I built those containers as root then I oh man suddenly I'd be writing to my uh I'd be writing to my host operating system and even if you don't make those mistakes you almost always build a container from another container and if somebody decides that they want to do bad things to you or bad things to your at least to your computer um it's possible for them to craft containers that very quietly um do things to your to your operating system and not not to your not in the container uh you

know just with very very simple commands during the build process if you do do so it's root and so that's why you know in the past we always suggest that people use throwaway VMS to build their containers and then just discard them when they're finished with it but now um using these um unprivileged uh these unprivileged you know ways of building containers it really it makes it a lot safer and a lot nicer to build your containers yeah Dave I take it you've never you've never messed up a host system before ever no Christian I think you had something you wanted to add yeah yes

this ability to build containers and not having not needing to have the root um when I was in the role of supporting the research community at Berkeley lab that was a big hurdle in container adoption by the user Community from non-cs domains like like domains without a lot of computing and knowledge if they have to even build up to have a discardable VM for building content that's too much of an ask from for them and that was a big hurdle for all of them to adopt containers especially Singularity to build everything from scratch so they would just go you will do it using a Docker

container and try to convert that from Docker to Singularity which can be done without a root operation as they that's the part they were taking but they are limited by the docker containers they have access to so they are they need to have it's additional processing additional steps they have to get everything working in Docker do the conversion to Singularity and get onto these huge shared machines where only Singularity wasn't available for them to make use of container systems so this being able to create Apptainer containers without the root privilege is really a game changer for for situations like those um yeah I I and

yeah this is something that has been waiting for quite some time and uh thanks to the community and primary day for he being staying behind it and pushing it out making it available thank you Krista so Dave what else has in this release of Apptainer I mean I know a lot of it is focused on security and being able to do things we've been talking about but what else is a part of this release oh there's so many miscellaneous things I don't know if it's hard to say what what's what's what's more important um we've touched on the the most important ones uh there's you

know a bunch of bug fixes a bunch of also minor minor new features he also did import many of the any of the fixes that scientists put into Singularity CE uh uh we did um I did want to mention also on the build topic that that a lot of times people talk about we want to be able to build a uh build a package optimize for particular hbcs and so they want to use the compilers from the from the local hpcs and so that's another thing that's that's helpful to have be able to build on the machine where you need to to do the uh

run the run the containers ah nesting okay that's a that's that's a great question but that is enough I feel like we've done got into this one before and it becomes very interesting you know Dave and Greg I think he talked about this too but go ahead yes so that's another big big advantage of the way that this was implemented in in 1.1 and that that just about everything can be done nested inside of other you know images on other containers so even though apptainer runs things uh with no new privileges set so once you've started inside this privilege you can't do anything set uids

anything set cap you can still do all the same operations inside them well we can we can still do the the fake root build that is with this fake fake root I call it thing the real fake root is with Etsy sub uid it's a sub GID mapping that one you can't do inside because that's also a privileged operation so this is a lot of times there's this there's a whole ecosystem called the rootless containers um that's one thing is that they also um they do still require some input increased Privileges and a little bit of increased privileges is that that program that runs it

that reads from itsy sub uad and Etsy sub GID that's a little increased privilege program so we can't run that nested inside of another container because we don't allow that so but this one this kind of fake fake root can be done inside and and all the mounting of all the mounting of uh fuse file systems is done in an unprivileged user namespace it does not require the the normally I mean you can you can actually run most fuse programs straight from your outside of a container but what that does also uses a set uid root assist program this user mount but that doesn't hit

that fuser Mount doesn't get used if you are running in an unprivileged username space at the first level when you first started on privileged user namespace you have a fake another kind of fake root but just your just your one user ID is is running as root there you can do a fuse mount but then we always switched back to the original user ID once once once the user starts executing the user code so but you can then run another open another user on privileged username space and do some more mounting so yeah everything can be all the new features at least can can be

nested thank you for the question Rose maybe this is maybe I'll show a little ignorance here what would be a use case when would I do that when would I want to do that to Nest the container within a container somebody else want to answer that no go for for us um I was just gonna say that has a lot of usability within like the context of cicd systems um Docker for example has a system that's somewhat similar called Docker and Docker that allows you to use a containerized build environment to automate your container builds so you can set up within the context of a

container an environment that you know can build containers on it so the first thing I think is that this has you know being able to build out containers inside of Apptainers is massively useful for um being able to get a lot more automation around how you can build and test those like I said especially with cicd systems like those uh that you can get through you know on gitlab GitHub places like that and then in this case with I am looking right now in my with my employer um there are multiple teams developing various applications but which are interdependent and in the stack in the

application stack is too deep and there are dependencies and there are various teams developing them um Apptainer is giving them the flexibility to say okay this is our latest container and build your applications back on top of this container it's giving that flexibility and freedom to various teams to do on their own and also declare what their requirements are and how they can run easily in this nested approach that is what we are adopting here internally so the new new approach in Hardware is yeah we are adopting gpus in our computers in our Linux computers within this wafer inspection machines right now with gpus um

having the base OS operately and the Nvidia stack separate and on top of that having the framework uh the application based framework and then having the application all of these things the various teams who are doing each of these stages in this application stack and all of them are releasing their own versions of containers and we were doing it nested um and running one container on top of another container so that's another example of how we can leverage it oh that's great that makes a lot of sense thank you yeah especially when the application stack is too deep and there are a lot of teams

working on it not just one single thing who are all sitting around and talking and developing that stack but various teams with geographically distributed and they're developing their own piece of the puzzle at that point it gives a lot more freedom and less communication that needs to happen unless uh moving everybody together will be difficult so thank you Dr godlow I think you had something you wanted to add to this too yeah I mean so uh we're we're having a little kind of sidebar chat here we're talking about I mean this is basically recursion right this is this is using containers recursively and as anybody

with any programming experience will tell you recursion is very useful in lots of different contexts so uh with when you're talking about containers I mean basically anytime you want to uh anytime you want to manipulate or have some sort of um some sort of workflow or whatever which depends upon containers it might be good to containerize that thing which is dealing with containers and anytime you're doing that you're talking about nesting containers one concrete example I have I have kind of a silly little container scanner that I've you know that I've got sitting around in various places on the internet and uh what that does

is it's in a container and it's got some you know some vulnerability scanning tools and it it asks you to point it to the container that you want to scan and then it grabs that container and ingests it into itself so the container goes into the container and then it starts scanning it inside the container you know to do that you've got to be able to do a nested containers right so scanning it in terms of it actually executes the container and starts scanning it against ports that are open or is it actually no it's a lot so so this this thing is actually um

it scans for um it's it's just it's not the way that people typically think about scanning for vulnerabilities with within containers um it doesn't look for cves and stuff like that what it does instead is look for um it looks for uh actually viruses and root kits within the container um and so the way in which it does that is that it's got some you know scanners built into the container and then it it's pretty silly actually in that it it just uses um the container Machinery inside of it to convert to pull the thing out of the zip file and convert it to something

like a like a bear directory that it can just walk through and scan okay like the quickest easiest way I could do it and um it uses you know nested containers and does what it needs to do thanks Dave yep dude's extra I think you had a topic you wanted to bring up about at Tanner and podman yeah so um so this is a big uh let's say philosophical difference a different um different applications that that podman and so apptainer is really focused on being able to run higher level applications let's say number crunching applications right whereas podman is designed to be more well compatible

with Docker since trying to do system Services that's trying to be able to try to pretty much emulate as closely as possible an entire whole Linux system so that difference allows us to be we can we can do a different you know since it's a different goals we have uh different types of approaches that we can have so like this is the the whole this it's very important to podman to have this Etsy sub uid it's a sub GID mapping so that each user gets a whole range of let's say 64k user IDs that they can work with whereas most the time we don't care

we only care about one user ID and we don't care about emulating the network because we can just share the network so so there is a lot of things simplifications that we can do an app trainer I mean it's also in terms of scanning it goes I you know security way I care very little the very few very few of the cves that come out that will apply to Apptainer because we don't run Services we just don't run services so all the only thing that it might apply is an outgoing or your clients your out your clients let's say a TCP client and there might

be some possible way that back through you connect to a malicious server that might somehow go back to your come back up through your connection but so there's there's so so there's a lot of simplifications that we can do um this with with this different goals and so I think it's I think it's worth having the two different um container runtimes that are that are in use under different purposes that's very helpful Dave thank you because I think that gets I think it's brought up quite often is what is the difference between the two and that that makes a lot of sense so I appreciate

that so we are getting close on time but before we do Dave I want as someone's sitting on the outside watching open source I don't think I myself don't really have the perspective of what it really takes to run an open source project what is that um the build process and release cycle look like and how do you manage that hmm well it's a big open-ended question uh there's a big open-ended question yes I mean it's it's really about uh good well using having the right tools for one thing GitHub makes it makes it you know really quite quite easy um and a way to

be able to have anybody anybody can contribute and you can review very carefully over things you can run a whole bunch of of tests every time you make a every time you make any kind of small change it runs a large suite of tests automatically so we have it set up and uh yeah so GitHub makes that makes that uh quite easy I mean then it's also a matter of also um getting the volunteers and and managing people that's that's that becomes a big a big aspect of it and uh I mean I I uh we have you know since since scilab split off from

our community we did suddenly get a big drop in in the uh resources that had been being put into it and so um and and uh so I've stepped up now I'm hoping that there will be there will be more uh people that will maybe take over from from me some of the stuff load that I've taken lately but yeah it's it's it's it's a it is also uh I felt it it's important to have a a community community-based project and one that's not focused on on the needs of a of a for-profit company and um that's uh you know being in the Linux Foundation

is a big help for that too with uh requiring uh five uh you know at least five people to be uh to to be a committee that's in charge instead of a single individual um so that's that's another big aspect of it um so yeah it surely helps um I mean this is the rule where I learned a lot of git than any other rules that I have been at um it is a good opportunity to learn really important skills and skills uh be being on the volunteer position here volunteering for the open source projects like this there would be a lot of issues that

communities puts in and looking at those issues is another way of learning what's happening in the Tool uh understanding how the tool Works um that that would be another aspect of running an open source project like this issues that come in monitor keeping an eye on them following up on them that that is a little bit of work that's involved in running a project like this and for anyone listening and interested in learning um yeah I would recommend starting there keep an eye on the issues most of them most probably they would give some answer on them you would you see how the issue coming

in and what kind of responses what channel back and forth happens that's a quick way of learning if you are interested in contributing to the project um yeah get issue use and then if you have a lot more more time you can join the weekly by two two calls per month uh joining those calls and at least listening to what being discussed how decisions are being made um when the really scheduled discussions that we're having what features you want to include in the next is what do you want to put for a later time um listening to that kind of those kind of discussions that

happen basically calls that's the next step where you can get involved yeah that is fascinating to listen to you from my perspective so I sit through those and find find it very interesting so uh there's another question I was going to ask you Dave and now I've kind of lost my train of thought here um oh while you sorry I was gonna say what have you ever done it so I don't have time but I was gonna say while you regain your train of thought I always thought that one of the most challenging things about um uh you know doing the open source development and

help you know uh helping to uh you know guide the open source development is really um security related issues that's where things become really complicated especially with with an open source project like this because um the development you know it's it's it's all supposed to open it's all supposed to happen right out there in the open and it works the best when it happens right out there in the open but then if you do have a vulnerability especially one that you don't immediately have a patch for Things become kind of complicated as far as like well how do you um make sure that the parties

that need to know about this do know about this and make sure that nobody else does as you Shepherd this thing through and get a patch done and get it released and do all that and that that's that's um that's a that's a complicated complex kind of thing to do in an open source Community like this yeah most people don't ever think about until you're involved with something like this so I think it's really interesting from the outside I remember what I was going to ask you Dave is when you're having these Community meetings and you're seeing that you need help where so the people

watching this where do you need the most help or where is something that from a viewer's perspective where could you you really use Health today well it's just uh to um to to contribute the if you find if you run across a problem or something that you think that that that could be improved trying to work on contributing the code you can you can ask for some help um I've also I've I've benefited greatly by being able to communicate with the people who who did the original implementation by a slack you know that you can can ask them ask them particular questions that's that's a

big help so but yeah definitely if there's you know if you have an idea please submit the requests and uh yeah you know when I started working on this I didn't know go at all I started before before I started seeing it in Singularity so you know learn the language it's you can hack things without learning the language too Google is a great help for that but uh yeah the community does have a lot of uh nice people who just reach out um as you look at the commit logs or the traffic in the issues like you can see who have been responding on a

specific topic that you can find the same person in the slack and you do your private message or in the general chat uh community members do respond this respond and give you uh helpful things um so yeah more PR's community members can submit a few years fixing for specific issues as Dave said they are surely helps and also as new issues come in if you are familiar with that and you know the answers um for that any any GitHub you can give in doing quick responses uh to those each GitHub issues also there's another area where we can where the project changes have come on

yes yeah I mean we can also because it's also help to give us an issue if you can if you find if you run across a problem please let us know about it instead instead of just saying well I could work around that instead of instead of and not let us know uh I also I did have a plea for for quite a while said please people test out give performance testing on this and and uh it didn't happen until very late and then it was like oh no but so and and we did only get one one person to give us some give us

some benchmarks so we would still love to see more people run their benchmarks run there run their code and I Apptainer one time oh and with it with uh both both set uid mode and not set uid mode and see if you can uh can detect different uh differences and if they see any differences or even if you don't if you just do the test test let us know give us some results yeah that's a great way to start if you're if you're just kind of you want to begin being part of the community and you don't really know how to do it that is

a really really great way to start I mean that's that's the way I started as well is that I was kind of watching development and every time Greg would uh do a new release candidate I would I would go through and just test everything that was I mean it was possible to do that at that point in time but I would just go through and test everything that was in the CLI to the best of my ability on my local resources then I would ping Greg and slack and be like here's all the stuff I tested and here's you know what worked and what I'm

not sure about what didn't work you know and so that's that's a great way to just kind of get started get moving it's definitely something that's really needed that's great thank you guys for that um so we are running up on time I want to make sure Dave has enough time to get to his next meeting but I am going to open this up to kind of final comments about whatever you want to say Apptainer this release plea for help whatever you like I'm going to start with you Krishna um 1.1 came out with really amazing features um try it out and use it and

put in open issues if you can if you find um and dear come and join the a slack Channel monthly calls and I contribute to the code if you if you want um if you have titles available that's what I would say thank you Forest I would just say that you know here a year after we moved into the Linux foundation and switched from Singularity Apptainer I am really pleased to see still such as being made to the software and such um you know long-running developments being worked out with it so it's great to still see it there's a lot of activity there and that

you know app Tanger solves a lot of active development and a lot of great goals that are being implemented and left to go so it's great to see it active still thank you for us Dave God love so I would just say that I feel like this release of Apptainer uh the 1.1 release um it feels like a new Milestone I mean it really feels like a new kind of like era opening up within um Apptainer uh just the way things are are progressing and you know obviously it's a community effort a lot of people deserve a lot of congratulations but I also want

to just kind of call out once again Dave in particular Dave Dexter in particular and you know thank him for um I think that he had this idea kind of I mean you I think you kind of had this idea a while ago and and you stuck to it for a long time and and you know you it was a lot of work to see it through from start to finish so congratulations on getting that done absolutely Dave thank you thank you so closing thoughts yeah so this uh you know this this tool has been really really important for our for uh the community that

that I'm that I'm uh support the eye through putting Computing community and and um so that's really important to us that the that the that it you know has has a long life and so that's that was even though even though even though we don't really need that this some of this functionality I thought it was really important for it to to to be there so um I thought I I could do this you know with a little bit of help from Mostly from Cedric so tell me what tell me what needed to be done um and uh so and you know this could be

the same way for for anybody who would like to contribute to it this is a community project and and you know we're we're part of the community and and every user is a comparative part of the community so please uh contribute which you can and now let's let's uh continue to have a long life I mean I I see people people having presentations of that saying say oh we should all switch to podman well you know maybe if it changes quite a bit so that it can do all the things that Apptainer can do you know but uh but why should it you know why

not have two run times I'm not sure about that uh so um I I think it's important for it to still uh to still be to be active and and being in under active development so absolutely thank you very much we thank you guys for joining us today we hope you come back next week again Dave Krishna thank you for joining us from the outside and day four so good to see you again we've really appreciate it you guys like subscribe and we will see you next week thank you thank you