/.singularity.d - The Hidden Files

do we run jobs in containers instead of services essentially and that was a need that was part of the need that the high performance Computing world uh required and another need that they had was uh how do we run containers with more security than was allowed or available at the time and so that's basically how the project started and since then it's taken off uh really within HBC primarily but it's become a huge project that everybody just depends upon just relies on now yeah that's awesome I I love stories of um like the Genesis of how things became right it was like there was there

was a need and then there was an idea and then now I mean everyone knows of Singularity Andor Apptainer and or using it and of course I am going to interrupt you probably a million times and ask a million questions however if you are watching and you have a question just jot it down in the comment section whether you're on YouTube or on LinkedIn either way is totally fine um we do have other people behind the scenes I don't know I am definitely like a do one thing at a time kind of girl right like the multitasking of looking all the different places that is

not my job which is a beautiful thing we have other people to do that so you um ask your questions and then uh towards the end as we kind of wrap up we'll be answering all of your questions there so awesome I I should say too that there's been a I was a little wishy-washy uh originally about what the topic of today's conversation would be so I don't think we're actually going to talk too much about de files I thought that maybe we could kind of dive into um the internals a little bit of Apptainer like the containers themselves and we can start talking about

some metadata and some internal stuff there this is going to be kind of deep down in the weeds but I think it's fun stuff to look at and to think about and it'll give you some hopefully some new ideas for things you can do with that painter and also give you a sense for like just how things are working you know deep down under the covers I like it thank you for that correction of my title yeah well that's my bad because I was a little like you know maybe I'll talk about this maybe I'll talk about that I don't know okay that's awesome all right perfect okay yeah let's dive into it so what is the singularity D directory that's it right that's what we're talking about yeah yeah so it's actually it's a hidden directory so it's actually the slash.

singularity. D directory and um yeah so so like containers you know Apptainer containers um they can do a lot of things there's a lot of like like um actions associated with these containers that you can do you can like you know run them or shell into them or exact uh them there are predetermined actions that happen when you um execute the container as an executable or when you use the Run script via run and that stuff's got to live somewhere there's an environment with environment variables um there's metadata like labels and stuff like that and all that stuff's got to be in the container somewhere

it's got to be associated with the container so that is basically the way that that stuff works is that there's a hidden directory um at the the root of the file system within the actual squash FS file system that lives inside of the the C file um and that hidden directory is called sl. singularity. D and if we look in there and start poking around we can find actually like the internals of the container and what what makes it work aside from just the file system itself so why is it a hidden directory and what does that mean yeah so whenever you put a dot

in front of um in front of a file or a directory with the Linux file system it it makes it a hidden directory and I think usually the idea there is that you know when you want to list the contents of a directory those are things that you you're probably not looking for usually and so what happens is when you list the the file or the contents of a particular directory things that are hidden are not shown by default you have to add another flag to show all the stuff that's hidden and so uh because of that then you end up um it just makes

things easier to look at you know maybe I should just go ahead and I'm on a new computer and uh it's a Windows machine which is insane I haven't used windows in like 10 years um but uh yeah so I've got I've got a new personal Samsung machine that's got Windows on it and I'm going to try to share screen maybe I should just dive into that um and then if I can and then we can go through and and kind of you know start to just look at some of the stuff instead of having me talk about it cool and um if you're at

home I think we can add a little link here too to a little bit.ly for the Apptainer Deep dive for people maybe we can just like pop that in the chat there for them so that they can continue on their search and inspiration with um Apptainer okay there you go yeah I can see it cool so yeah this I'm not gonna I can't see anything except for my screen now because it jumped to a new desktop but yeah go ahead and let me know if um things Catch Fire behind me or if there's anything else I need to know about going on um so yeah

we were just talking about hidden directory we can just start off really quick just talking about hidden directories with Linux so if I do like an ls- L which I have a shortcut to that which is LL um it's just going to show me I've got a couple of directories here in this demo user or douser as you may wish um I've got a couple of directories here um go and June 13 Demo which is what I'm going to be looking at here and but if I could do an ls- L A so or just a let me just do LL which is an alias

for ls- L and then give it A- a so that- a means all if I do that oh this is why hidden stuff is helpful because all this kind of gets in the way right probably what I want to look at is either this June 13th demo or go and they're here but there's all this other hidden stuff in my home directory too that I probably am not looking for when I just list the files and directories normally does that make sense Rose yeah yeah yeah I see all the dots yeah cool so okay so and that pertains to what we're talking about today because singularity.

D is a hidden directory let me go ahead and CD into this June 13th demo directory and I'll show you that I've already downloaded a few containers and um what I did is I downloaded this LOL Cal siif container which is a container that you know I'm sure most of us uh who have been around the Apptainer commun for a while are probably familiar with um if not you'll see it today I downloaded a just a plain python container to demonstrate something else that I want to talk about a little bit later I converted this lolcow dos just into a directory just so we can like look at it and go through and and you know see what's inside of it and then I've got some other containers that I created using definition files there's one I call tree.

Def and there's one I call them. and we'll talk about those later as I'm trying to demonstrate different things um so I don't know um do you have any any questions to begin with rose or should I just kind of dive in and start talking about stuff I guess you should just dive in and start talking about stuff because I'll probably take us off on a tangent okay well I mean feel free to follow tangents too if you like but you know yeah I mean I guess I was just wondering I mean if if you're gonna open the door like when you say you downloaded

this so are these containers inside of containers or where do these containers live you have like a special demo environment and that's where they all are so yeah no so this is just on my laptop excuse me my um my voice is going to be coming and going this is just on my laptop and I just have a new user that I created called uh you know well it's Up For Debate what it's called but demo user or douse or whatever um so um this this new uh user just have created this June 13th demo directory and I've just i' pulled these containers using the

commands like apptainer pull let's see if I've got one saved yeah so that so that's the command that I used to pull this lolcow dosf Container for instance awesome and then a side note question oh sorry sure go ahead yeah no because because okay so we're talking about Apptainer and like I know the history of what happened with Apptainer and singular it but then you it's still it's called singularity.

D or SL singular so yeah people yeah so for backwards compatibility anything that was named like one thing was uh most of the things that were named one thing um especially like internal uh still have that same name so for instance this is a very important directory let me just show it to you as we're talking talking about it so remember that I I um I took this C file and I built a Sandbox out of it I did that with like an apptainer build sandbox LOL Cal and then lol cal.

and I'm not gonna execute that command again but that basically takes this C file and creates a be directory out of it so let's jump into that and have a look so if we go into it and then I do an ls- L you know it's just a it's a root file system of a of a you know it's a Linux root file system with a couple of funny kind of new files new sim links and so these are called Singularity um because these have been around for a long time these have been around before Singularity entered the Linux foundation and changed it named Apptainer

and so because of that um if the new version of Apptainer which has changed his name wants to run um old containers it has to be aware of this naming Convent that that important files are going to be called you know Singularity and notice that they're pointing to this place. singularity. D and once again we can't see that because it's hidden but if I do a ll- a oh now we've got some more files here right and a lot more of those are pointing to that singularity. D directory and in fact we can see now this singularity.

D directory and if I CD into that and then I run the tree command to just see what all is in there we can see that we have um several directories actions uh bootstrap history which is interesting environment Libs and then we got all these files right these ones are kind of interesting because they have the names of the sub commands that sing that Apptainer uses I'm going to be saying Singularity a lot because it's like right in my face to say Singularity a lot but yeah anytime I say Singularity I mean Apptainer as well um um I'll talk about the bootstrap history later we

have an M directory which has has to do with environment variables and I'll talk about the details of that momentarily we've got um labels Libs which doesn't have anything in it right now run script start script script and then we've got this this you know of course a file called Singularity as well because why wouldn't we have that so um yeah so the answer your question Rose yeah a lot like I used to joke that within the singularity before we changed our name to you know before the Project's name was changed to Apptainer within the singularity like ecosystem everything was called Singularity and it was

really confusing because anytime there was any like new file we added or new directory or new configuration or new anything we just called that Singularity and it was really confusing so careful what you wish for you wanted a new name now you got a new name and now it's all so confusing yeah right because it all like you know it has to be Backward Compatible and point back to like old school stuff all right let's see so should I start talking about um the contents here or do you think there's questions before I jump into that yes I think it's great do that I will

actually go check on and see if we have any questions coming in so far uh nope no questions so far as far as I can see so carry on so one of the first things that you'll notice like let let me make this explicit if I do an Apptainer yellow help um well maybe this doesn't make it more specific there some of these commands there Apptainer got a bunch of um sub commands but there's a subset of these commands that we refer to as the Action commands and that includes things like exec um shell run test uh start if you're talking about image start so

let's just go back and and look and see at the contents of this directory again well I'll be I'll be darn so so we've got um under the actions we've got exec run shell start and test so right off the bat you should be thinking to yourself that you know there's something interesting about those so these are actually internal shell scripts that um that they're kind of like drivers they make the container work so let's look at the Shell command for instance if we do like um them actions shell we can see this script and so when you when you run a container um this

is this is actually the script that gets well actually I'm sorry when you execute the shell command within a container this is actually the script that that gets run inside the container to make everything work properly for you and so it's kind of like a pretty pretty simple solution to like run this script to make the container like to have this this uh directory full of like um back hash script drivers to make the container work but that's just how Apptainer works it's pretty simple so we step through and we look at this um we have a a for Loop and this is basically saying for whatever you know for whatever the star says for any file which ends with SH which is in the directory singularity.

D environment do something and whatever iteration we're on call the the current you know thing that's in that directory script and then so what are we going to do that that number of times well we're going to say if that new um environment variable that that we just named script is a file then Source it and that's what that dot means source so is this something that you would want people to actually be able to see clearly so yeah this is what's cool about this yeah this is what's cool about oh go ahead I'm sorry do we have a delay no it's it's dark oh is it is it hard to see yeah so it's like the background is black I mean some of the writing is white and it really pops and it's like okay I can see exactly what we're talking about where it says script and singularity.

D environment the little star that pops out really good but the other things are like black on blue and some kind of I don't even know what uh I can't remember there's an option there's an option to set the set the background equals dark um is it equals yeah here let's do this there is that better much better thank you okay um yeah yeah yeah so yeah um I thought that you were asking at a higher level is this something that your average is this is this file something that your average user you would want them to see um but you were asking do you

want people um viewers of this presentation to be able to know what you're talking about a little different it's like here let me show you this thing do you see it do you see it so okay so hopefully now we can see it a little bit better so we're saying um there's this for Loop and it's saying basically for every file which is which ends insh and is in this environment directory Source it and you know so this is basically what sets up your environment inside of your container and we're going to go through and we're going to look in this EnV directory shortly and

we're going to see how there's this whole series of scripts and they all get sourced in order and these scripts determine what your environment is whenever you run your container so that's the first thing that happens and so then the second thing that is going to happen is we're going to say we're going to test to see if there is a a environment variable set called Singularity shell and whether or not it's empty and if it's not empty and if it isn't executable then we're going to try to use that to start up the shell process okay so Singularity shell if that exist EXs that

environment variable exists we're going to use that as our current shell um and if that doesn't work for some reason we're going to we're going to say whoops nope that didn't work and then we're going to exit with an error code if that environment variable does not exist or if it's not an executable then we're just going to look for bin bash inside the container and then we're going to set a prompt a PS1 we're gonna uh export that prompt and then we're going to just execute this bin Bash with any um with any arguments that the that the user passed and this this no

RC says don't Source A bash RC file in other words don't set the users um configuration for their shell in this in this new container um else if we can't find bash we're going to look for sh because that should be in almost any container um and then we're going to try to start a shell using bsh if that doesn't work we're going to say I don't know what to do because your container doesn't have a shell in it and we're just going to exit with an error so this is one of the cool things about digging through the internals here is because um I

you know I might have known at some point in time that this environment variable Singularity shell was available but I forgot about it and so in in like prepping for this and going through this stuff I was like oh yeah there's this there's this variable Singularity shell that I can use to change what um what shell actually get started and so you can learn new things by digging through stuff like this about the capabilities of Apptainer what it can do and then that can you know that can trigger new ideas for how to use it and stuff so let's test that let's test how that

works all right so I'm gonna I'm going to go back to kind of The Parent Directory here which is I believe something July there we go June whatever date whatever month we're in and then remember that I've got um several different containers I downloaded this python. container and I downloaded that specifically to show that you can use something like python as your preferred shell so um one thing that the Apptainer developers did as a convenience to users is they changed all the singularity um environment variables to Apptainer so you can instead of passing singularity _ shell you can use aercore shell that will work as well in fact that's preferred and let's see I think it's user B python I might be wrong about that but let's go ahead and say apptainer shell python.

yeah I was wrong about that so now where where is python exist here in the container it's kind of Overkill but I'm gonna say find Das name name python to it's uh user local then python okay let me try this again should have saved this sorry about that user local pin python let's try this again okay so now if I shell into this now that I've got that that set as the the shell that I want to use I get a python prompt so the same command twice the first time I just got you know A bash prompt because I didn't have what I was

asking for wasn't an executable so it went through that logic in that script and said up you asked for something that's not executable I'm just going to give you um you know bash and the second time I did it it said oh no I found that and it's it's an executable so now you can be in Python instead so now because it's python I have to exit with uh this funny syntax in order to get out because it's a function cool so let's start looking through some more stuff here all right um let's see I've got I've got like some notes that I wrote down

um yeah so some other interesting things are if we look at exec I think this is a really easy one so all this does is it does that little for loop I told you about before that that um we're going to talk about in a minute that sources your environment and then it just executes whatever command you give it with whatever op whatever uh command line you give it it just execs that so it's back to the uh oh there we oh yeah anyway the colors were weird again when you went out and came back in yeah yeah GNA have to do that every time

um what was it I got oh sat back got it save in my history there we go so once again pretty simple it's just that little four loop I was talking about yep and then it just executes all right so let's look at a bit more complicated one and this will um this will give you some information about like some history too so if we look at I think run is one that's a bit more complicated and let me set that up for everybody okay so once again the first thing that we do here is we Source those envir those scripts to give us the

environment variables that we need um but now we've got this is a pretty complicated um uh script it's got a lot more to it it's got a lot of like if kind of test else if test kind of statements and the reason for this is because there might be an environment variable called Singularity app name and if that environment variable is set then we're going to check see if there's this directory called SF slapps and then if it's got that app name and if it does and and you know and underneath of that if there's another scif runscript and if it does we're going to

use that as the Run script and so um it's kind of rarely used but there's this whole um there's this whole like architecture in thater that was put in place a while ago um primarily by uh Vanessa uh soat or socket uh Vanessa socket um and what it is is it allows you to take multiple different applications and put them all in the same container and when you do this it actually exposes the name of the application which is kind of a groovy trick it exposes it out to the command line so that you can reference the containerized app at your command line and then

you can get it to run whatever application has been you know put in there and that's kind of cool because the way that that all works is just through these it's just through these bash scripts it's all just like exposed out to the user through the through these bash scripts so one thing that you can do is you can set environment variables and you can do it but you can also um pass um the name of the um of the the the application itself at the command prompt and I think that what that does is it actually sets the environment variable through the command prompt

and then these these scripts get triggered so it's kind of cool and um you know it's interesting little little little trick that you can do um and then I believe that in the first Incarnation um of this the the directory name was a little bit different the directory naming convention was a bit different and because this is something that happens S at the level of the container if you start off one way and you decide to make a change you have to have backwards compatible code in the container itself as we've seen already so I think that's why we have this second lsf statement that

you know also says if if this is an old container and if this directory structure is like kind of the old way it used to be then go ahead and use that as well okay all right I think that that's enough of the Action commands I'll let you uh explore some of the other Action commands on your own if you're interested in doing that I'm gonna skip for a moment the bootstrap history um that this is an interesting directory I do want to talk about it a little bit but instead right now I want to skip and start talking about um the environment directory so

we saw that pretty much all of our Action commands start by with a little for Loop that says anything in this environment directory which ends in a Dosh go through and Source all those files in alpha numeric order so that's the order is important um so we're going to go ahead and Source all these files in order um this uh this container was made from it was made from a a CIF file it was it was created by dumping the contents of a C file out into a bare directory using the um the sandbox directive so because of that it's not going to have stuff

like like so so this this this file here is for um environment variables that were added uh if the container was originally a like a Docker uh container and if it was created using a Docker file it might have environment variables that were added like that Apptainer will pick those up and it will add them here in this file um the base I'm pretty sure is empty so we'll look at that uh so I'm not sure if that's I'm not sure if that's something that's like of historical significance or if there are still um reasons that we might use this base but right now there's

not there's nothing here so I don't know that might be something that's changed if you pass certain environment variables um at runtime it might dynamically like rewrite stuff there I I kind of doubt that actually so I think this is probably just like for historical purposes we look at this should be empty as well oh no okay no you know what I think that it always has this because this is the place at which this um path has to be set this probably used to be in base and I'm I'm sure there was some bug or something some reason we had to move it but

this probably ought to be in Bas or used to be in base and we realized that for some reason it can't be anymore you know what I'll bet you maybe sometimes um within Docker maybe sometimes they rewrite the entire path so maybe this fixes a Bud there in any case if there were more variables here um that were added by the docker file you would see them listed here down below okay okay 90 environment.

sh that's the one that gets written whenever you first build the container so this is a it's kind of an important one any environment variables that you add in your definition file when you build the container are going to be here and so you can see that I added originally lcore all equals c um and I added uh you know A New Path and that that's where these get sourced and that's where they get added to the container okay um there's apps base and then there's apps and these both have to do with that same um that same SF directory structure that I told you

about before so if you load multiple apps into a single container these are the uh these are the the the dosh files that get sourced which give them their individual um their individual uh environment variables so you're different it's kind of cool because when you like I showed you before that there's a mechanism in place to selectively choose a run script uh based on what application you you you want to run inside the container there's also a mechanism in place to give each one of those apps their own environment so each one of those applications can have their own environment variables and that that's how

this works right here and then if we look kind of an interesting one yeah yeah so depending on how you call the container the container will have different environment variables set at runtime yeah it's kind of cool it is and once again it's all it's all it's all just bash scripts so that's kind of like the neat thing about it is that it's all just bash scripts to make that happen and then I'll show you another cool one so this 99 excuse me runtime bars.

sh once again let me make that visible okay so this one looks weird right um yeah so there is this uh there's a mechanism in Apptainer there are environment variables that you can set and I would direct you towards the appendix in the um the user guide there's environment variables that you can set which will allow you to prepend to the path or upend to the path um depending on how you set these environment variables and that becomes kind of important sometimes uh for various reasons you know there are certain bugs that it can fix and so on and so forth but this is how

this works so basically we check when the um when the uh uh container first runs if there are um environment variables set that will indicate that there are userdefined uh preens or up pens that should happen to the path or if we should just overwrite the path Al together and if these environment variables are set um we're going to go ahead and change the path depending on what what um what you know new directory structure what new directory path has been asked for and then at the end we go ahead and throw those variables away so they don't mess up the envir because it's not

the variables themselves that we want we want those variables to change the path so once again kind of complicated Behavior which is determined by pretty straightforward you know scripts cool any questions before I jump on yeah I mean who has the power to change these scripts developers so um okay so this is like Singularity how it's running not like I'm taking a container and I'm changing it this is like behind the scenes you want to be a part of it go into the community and hang out with Dave yeah and with and Dr Dave and everybody else yeah it like um it's kind of funny

right because this is like this is like the the run this is like the the um platform right this is how this is how apptainer runs but um but it's in the it's in the container and so it's kind of funny because it's like some of the apptainer runtime actually lives in the container itself there's like some blurring of the lines there but yeah the way that this works is that your installation of Apptainer has these files saved inside of it in go actually and when you create a container one of the steps that Apptainer does for you is it takes and it builds this

directory structure inside your new container and it plops all these files in here for you um you can go in and edit these if you want to um you're discouraged from doing so obviously because you'll probably just break your container um there have been situations in the past where I have edited specifically the environment uh scripts but yeah I mean that's if you're doing that then there's either you're you're probably doing something the wrong way which is making things too hard on yourself or you might be like correcting some bug which exists in Apptainer which really needs to be fixed at you know at the level of Apptainer and uh not just like by by hacking your containers individually all right so labels.

Json so when you build your container there's this there's a way to add like metadata to your container some metadata just already exists so if we look at this this is what this ends up looking like you got these um you know you've got these uh keys and then you've got these values and they tell you things like when container was built what version of Apptainer was used sorry I think you're telling me this probably yes thank you yeah you're welcome um yeah the boot the def file that was used and stuff the way it was bootstrapped basically um and so on and so forth

um there's a lives directory which is currently empty and it's empty because that's where if you run um your container with the d-nv or the dash dash Rockham Flags to tell Apptainer that you've got a um graphics card that needs specific libraries that's where apptainer binds all those libraries for you it dumps them all into that directory and then it sets your LD Library path so it'll go and look in that directory for the libraries that it needs um we've got a run script here that should be pretty clear what that is let's have a look at that so that just in this particular case

does Fortune Cal LOL cat because this is the LOL cow um so let me just show you wait did I just say LOL cow or LOL cat it said LOL cat and the reason for that let me show you real quick so this container does three things it generates for you a fortune using the program Fortune and it pipes that into a program called C which makes the fortune appear in this bubble as being spoken by a cow and then it creates kind of a rainbow gradient color and that and it does that by piping the result of the first two operations into a program

called LOL cat so LOL cat is the thing that makes these colors and so that's why that's the script um so the start script is pretty similar to The Run script um the start script runs by default if you use instances and if you do something like instance start um you know and then the name of the container instead of just like running the container it'll run the the start script instead this one's empty because I don't use this with an inst and then finally there's this Singularity file once again everything's called Singularity within the Apptainer ecosystem and if we look at that that's going

to be the definition file now here's where things get kind of funny this definition file this is not the definition file that I originally used to create this container right this is the definition file that was used when I executed that Apptainer bill D- sandbox lolcow lolcow dosf command and so what it did is it created a definition file for me based on that and it said oh well you're trying to build a container from a local image and the actual image that you're trying to build from is LOL cal.

and then that was the entire definition file now that's not particularly helpful if you actually want to see how this container was built that's where this bootstrap history comes in so what I can do um I've got this Apptainer zero uh file within bootstrap history so if I look at bootstrap history at tainer zero this is the original definition file which was originally used to create this um this uh LOL cal.

container so the way that this works is if you stay with within um the Apptainer ecosystem and you continue to build containers from one another that are just SI files it will layer on top of it all of your definition files in that directory so that you can go back and kind of go back through the history and see how the container was created so this is kind of cool and so I I um I created another couple of definition files to kind of like show this so the um I've been using the tree command a lot throughout this you know to to show this uh hierarchy the LOL cal.

container does not have tree installed in it so I decided to use it as a base and install tree and so I said okay I'm going to grab that container again from uh Docker Hub I'm use it as a base and then I'm going install this command tree and then I said oh okay well it also doesn't have them installed in it so let me um make another definition file and in that one I'll install them in this case I'm going to use instead of uh starting from the dockerhub one I'm going to start from that tree file that I just built so now I've got like three containers that I built in succession and if I go into um this Vim CFT file and I go to singularity.

D and I look at everything that's there I can use tree because that was one of the things that I installed um now this bootstrap history has two files in it and I've also still got let's see my Singularity file here it's in a different place now so if I said like them Apptainer oh uh I gota say bootstrap history Apptainer zero this is where everything started this was the origin of everything right so I started by building this LOL Cal file using this definition file and then if I look at the second one this is the one I just showed you where I went

through and I I start you know I started from the file that I built and I installed tree and then if I want to look at the the latest one that's that Singularity file which is at the top there we go and so we've got the entire history saved of all the different definition files that we used to create this container okay so I think that that's pretty much um really quick and dirty tour through all the different uh you know through all the different aspects of the slash. singularity.

the um directory I'm stop sharing now the hidden files I feel like you ever go into someone's house and you open up their coverboard like just just like to see like how they organize things I feel like we just like went inside and we saw how things were organized that's awesome um so we actually do have a question coming in from LinkedIn so thank you very much and the question is um do you know a use case to get this deep into Apptainer or to be able to change the Apptainer setup scripts itself thank you Christian yeah um so once again uh I have had

to change the Apptainer scripts sometimes um actually to account for bugs and Apptainer so um and sometimes it's not bug sometimes it's like Corner cases as well so um for instance there have been times where um uh maybe maybe your container needs um needs to run with Dash dnv and maybe for some reason um you already got your own LD Library path and so but maybe when you run with d-nv it preens that Lids directory to your LD Library path for instance and so and what it does is it overwrites some library that you want to use which is already inside your container this is

like a weird Corner case that I've run into before so maybe what you need to do then is you need to get into those guts and you need to you know maybe dump some libraries make some Sim links to some libraries in the SL Libs directory that might work or maybe you might need to change the order with which maybe you need to like um set one of those scripts and maybe it's got to be one of one of the environment scripts and maybe it's got to be really late maybe it's not the 90-h environment script but maybe it's got to be like the last

environment script where you prepend instead of upending the the the libraries that you need onto your LD Library path there can be different things like that that you might sometimes have to get into and work with or of course another one of the use cases is if you find a a bug or a potential for enhancement like once again I showed you you how um more than one app can be installed in the same container and can be referenced and all that is just magic that's done with relatively simple bash scripts so if you find something like that then you could you know uh not

just change it but actually change in the source code and go to the developers and create a PR and say Hey you know we could make a relatively simple change to these U bash scripts in this directory and it would enable all this new functionality so why don't we consider that now I will say you know these are kind of like the heart of the con of the container so if you have a suggestion for an improvement it's going to require a lot of scrutiny probably from a lot of developers and there's probably going to have to be like discussions in the community meetings and

stuff like that because you know this is stuff that is working and we don't want to make changes to it lightly right you could have unintended consequences and side effects that's a great little segue into say somebody does want to get involved how do they kind of jump into what it is that you're doing because you're talking about bash scripts but it's also written in go right yeah that's probably important to understand yeah yeah so the way the way that this works is that I think um the last I checked I think that these scripts just end up being great big long strings that appear

directly in the go code and then um these strings are just basically written out to files by go um when the container's created so yeah I mean it wouldn't be hard to find you could grap through the code for phrases that are in these um scripts and then you can figure out pretty quickly based on that where they exist in the go code and then you can do little tests where you like add stuff and then build containers and see if it ends up in the you know this would be a relatively easy place to get started if you're more familiar with bash scripting than

you are with go for instance this would be a relatively easy place if you wanted to get started and start messing around with with that tainer and kind of playing with it and you know seeing how to how to change things with it I will say too that there is um there's room for improvement so we have talked for a long time for many many years I remember having a face tof face with Ian kesir probably four years ago in which we discussed and this was like shortly after we had created the CF file um so the CF file is a new thing and we're thinking about what to do with it and we're like well maybe we could get rid of all that stuff that's in what it should rightfully be the file system and we can get it out of you know sl.

singularity. D and we can put it in its own partition in the sift file and then at runtime that stuff can get copied into the Container you know or buy mounted on top of it or you know what have you and it can act like it's part of the the file system even though it's it's really not it's outside it's in the you know it's in the siph uh file itself so you know we never did that as far as I know um so you know and that's that's still like something that could be done which would you know arguably be an improvement um so I'm old enough too that I can remember before this was even a thing it used to be there was no.

singularity. be directory and um you know Greg rightfully said hey we' we've got to have this directory where we put this metadata and these drivers and we have to have a place for all this stuff to work and I honestly I was kind of against it because I was like what happens and this is like a stupid Corner case which will never ever happen but it was like what happens if I create a container and I install and I create the directory. singularity.

D when I create the cont container and I install a bunch of stuff in it and then Apptainer comes along during the Bild process and obliterates all that and puts all this other crft in it like like you know that's a weird Corner case that could really mess me up and you know that could happen fine but I think everybody was like yeah but I mean we need a way to do this and I mean that's a that's like a one in a million Corner case so let's not worry about that I mean that could probably happen anywhere anytime not just with Apptainer though

right yeah yeah I mean there's there's lots of stuff like this in computer science I I feel like just generally um like I kind of feel like this is the way bash got to be what it is because somebody was like uh okay well you know David Bourne or whoever it was somebody who created this stuff way back in the day um I don't know don't yell at me if you're still alive and you created it and that's about what happened but um I think somebody was like all right look we have all these commands but sometimes we need to pass strings and sometimes it's

it's a like the shell just based on the placement the shell understands that they strings but sometimes it doesn't so how are we gonna you know how we G to fix that somebody was like just put quotes we'll just quote it when it's a string and they're like okay put it in and they did it and then somebody was like hey I got a problem I need my string to have quotes in it and they're like oh uh well we'll just um will make single quotes like the magic quotes that allow you to have double quotes in your string like cool all right and then

some but it's just keep snow on right it's like but I need single quotes on my string oh well we'll put a backslash in front of the single quote and that's an escape character and that means that you know this is don't this you know this is a a literal string you know what if I need a backslash and a single quote you know and it just keeps going and going and so You' got like all this I I swear I think that's that's how bash strings developed right and one of the challenges of trying to understand and learn like how the heck you got

where you're at oh that's awesome well you see a lot of the history in this stuff too because it's like on the code side you can always just rewrite and you can always just get rid of stuff and it's hidden from the user and it's hidden from everything so you can always as long as it operates in the same way you can refactor but this is in the image right this is in the container and on the container side there are containers sitting on Docker Hub that have been built five or six years ago that still need to run and so because of that you

can't just say oh we're gonna make all these changes and it's gonna operate the same way your code still has to be able to look at those old containers and be like oh I remember when we did things this way when we did things this way I had to you know operate and you know X Y and Z and so that's why you end up seeing like this Legacy stuff in this directory that's wild thank you Dave God love that was fun um all right so tell us how we can get involved in the Apptainer Community where is it that you hang out and do

this kind of behind the scenes magic yeah so most most of where um I hang out is just on GitHub and so like a great way to get started with the Apptainer Community is to just go to GitHub and start look at the issues that we're having um there's a whole slow issues um or you know if you have a bug or a suggestion for an improvement go there and raise an issue and if you really want to get interested you know and that's enough I mean you're you're you're a member of the community right there um if you you know want to go a

little bit further don't just raise an issue take a crack at making a PR take a crack in making a poll request uh and another thing that you can do is attend there are twice monthly Community meetings I think it's like the first and third Tuesday um of every month and we have a slack Channel and if you're in the OR slack workspace um that's referenced at various places and different websites and stuff like that so if you're a member of the slack workspace then um Dr Dave will send out automated reminders about these Community meetings and so you know you can like raise an

issue and then you can come to the community meeting and there's a time to talk about stuff you could say hey I want to talk about you know this issue I raised and see what the community thinks about this if this would be a decent enhancement or if other people are experiencing this bug stuff like that awesome all right so um our behind the scenes little magic people we'll make sure that you guys get those links to uh the GitHub repository and then also the slack workspace um Dave is there any kind of like final thoughts you want to leave people with about the uh Singularity is it slash.

singularity. D um yeah no uh it's just cool I think to be able to look at some of like because uh once again this shows you some of the history of Apptainer and um shows you some of the internals and I think you know sometimes it's good to kind of peek out the covers and look at some of the stuff a little bit um one thing it's kind of unrelated but uh as we close out here uh if you have any like requests so we've been doing this ongoing um container education series for a while and you know I'm like kind of trying to think

about what topics would be interesting and you know what to talk about and we've been doing it for a long enough that I'm kind of starting to forget things that we've already talked about and things that we haven't so like I'm GNA ask uh any of our viewers if anybody is interested in like any you know topic for a future webinar please let us know um I guess at like ciq.com

there's a contact us page you can reach out and uh send us an email there or if you want to make a comment uh on you know one of the videos or something like that just let us know because I'd be happy to go through something totally different or something I haven't thought of uh with containers and not just containers but with you know other stuff Warewulf um you know fuzzball demos uh just anything you can think of if there's anything that you want to see us do in a webinar just let us know yes absolutely 100% thanks Dave and uh I mean you

kind of did like a really beautiful wrapup there so just make sure that you guys like And subscribe and share this um you know put the little tick on the notifications we are live um pretty much every week you know uh give or take I think we might do a recorded one on Fourth of July you know just kind of I think we might take that day off but we are definitely here for you so check out ciq.com we're doing a lot of really cool stuff just as a little reminder if you guys are still in centos7 happy to talk about um ciq bridge

where we're you know providing companies those backported security updates that they need so that they can stay stable and secure while they are migrating so we are going to just publicly you know encourage the migration off of sentos um we're happy to help you with that and of course we got Rocky and Warewulf and Apptainer and Ascender and you know fuzzball on the horizon so lots of really fun stuff and we'd love to chat with you all right you guys well hope you have a really wonderful day again happy birthday Zay Hamilton thank you Dave God love for being here yeah all right see you

next week bye bye later