Getting involved with Apptainer Development

while and and you know there kind of helped to manage development even though I wasn't one of the ones who was actively developing it so hopefully I can say a few things that are um you know interesting and and useful about the the community and uh the developers even though I'm not uh currently one of them yeah and you pretty much can't go through any of the documentation for app Tanner without running across something that Dave has written uh he seems to have written almost all of it I know as a community I know a lot of people do but Dave is definitely has his

fingerprint on a lot of things in that documentation for sure so Dave why don't you give us a little bit of background on the community and kind of where it came from where it is now how I got there just tell us the story yeah absolutely yeah we thought that we would um you know spend this time uh to talk a little bit about um you know not so much like developing you know containers or developing uh workflows and things like that but more developing aptainer itself because you know I mean this is a really um it's a great opening uh welcoming community and it's

a uh you know I think this is a great project if you're interested in getting into some sort of like um software development but you don't really know where to start I think this is a great project to kind of dive in because you know the community is so open and um kind of organic and everything but so yeah we you know um I guess it uh apptainer has been around in some form since you know for like six years or something like that at this point in time um and you know it used to be so uh you know Greg kind of started the

um the project uh when he was working at Lawrence Berkeley National Lab some time ago and he sent out some announcements um he created a slack Channel he created a website and people just sort of organically got involved and started to mess around with it and um you know I think that the project has really been organized mostly around GitHub has a lot a lot of projects are today and we've really used uh GitHub to great um to to uh to Great advantage to organize the community and organize the project and everything um so you know it was uh you know Singularity for a long

time and um the let's see I believe that the um you can still find the the repo I believe it's under HP CNG at this point in time but you can actually find the old apptainer repo which was renamed from singularity um and so you can go back on that all the way back you know basically every commit all the way back to the beginning of the project almost um and so now there is the way that the project is structured there are a group and unfortunately we couldn't get any of them to uh join today uh everybody's pretty busy but there are a group

of um people that are associated with the Linux steering committee and they are the ones that have Commit access to their repo um and so but you can you know um go to the apptainer uh you know repo and and um you know it's it's now just on GitHub it's apptainer slash apptainer and that's where the the code is and that's where you can get involved and raise issues and create PRS and do all that kind of stuff so Dave I know that it moved into the Linux Foundation but what has that done for updater what does it actually mean for the project how does

that help what view is it given it yeah so the Linux Foundation is now um you know an official uh sponsor an official host of the uh of the project um I myself am not one of the members of the steering committee um and I haven't been deeply involved in that so I can't speak about that in depth um you know maybe somebody else maybe forest or Brian can also comment on this as well but I I know that it it gives it um it you know we we're always looking for ways in which to sort of make sure that aptainer remains open source and

remains Community Driven and remains you know something which is not really dominated by any um you know like corporate entity or so on you know something like that and uh I think that helps us to be able to to make sure that that obtainer stays open source so have you seen growth in that Community since it's been moved over to the Linux Foundation have there been more people coming in to help actually do development documentation I mean I feel like there are a lot of there are already a lot of users of it but it seems like maybe there's been an uptick in that as

well is that something you've seen on the on the project side yeah I think so I mean I feel like for a while there when uh you know once again maybe somebody else can comment as well but for a while you know when I was watching I tend to kind of like watch issues and PRS and and um the different things that come into the the GitHub repo I don't um sometimes I just I don't really comment on them but I yeah I at least follow them and kind of you know lurk and watch them to some extent and I feel like for a while

there it was pretty much uh the same the same kinds of names I would see over and over and over again creating PRS and creating issues and stuff and I feel like recently I started to see a lot of new names and a lot of new faces um I don't know if the actual volume of issues nprs has increased and I think that that you know partially that's because uh you know we so apptainers gone through several different phases of development as like a big long ongoing project will and one of those you know one of those phases of development it went through was really

like you know we build out a whole list of different features that we wanted to add to apptainer and we just you know we um organized all those into different milestones for different releases and we you know this is when we had like a dedicated development team and we went through and we prioritized stuff and we just sort of started adding stuff and during that period of time obviously the number of PRS and issues was just huge the volume that was going through um obtainer is now kind of matured I believe as a project and um this could always change but you know right now

I think that the user base is probably and you know I'll let Brian Forrest maybe speak to this too I think the user base is probably um pretty happy with the feature set that app tainer has and so now we're more in a mode of just sort of trying to to make sure that we um we fix bugs and we keep things up to date as as you know things on the underlying system change um and things like that it's more of a maintenance kind of uh mindset versus like you know adding features kind of mindset so I don't think we've seen a really dramatic

uptake in in pull requests and uh issues but um definitely a lot of new faces um recently for the ones that we do have it's great Forest I saw you come off mute a minute ago oh um I was just gonna say uh with regards to the community um I'm sorry Dave this opinion this one what was the the question that you just alluded to us we were just talking about moving over to the Linux foundation and kind of what that looked like for the project itself yeah obtainer is a really really mature technology it's as Dave has been in development for quite a while

now so it's definitely at the moment um it's been deployed over like 20 000 or 25 000 sites worldwide so it's very well known just like Dave says it's at this point a very mature technology and it's kind of in a maintenance mode I'm just making sure that cbes are applied to and bugs are fixed there were some major developments not too long ago with bringing rootless capabilities to it which was pretty neat um but overall uh moving into the Linux Foundation was also meant to represent kind of the goal of being able to cross-pollinate with some of the other projects there and starting to

kind of see how aptainer can fit in with those because again it's a very mature technology and at this point a lot of the work there is on just maintaining it and making sure that it has the same level of quality for users of all levels then you needed of course it is I said Thank you Forrest still got noise behind me so Dave let's talk a little bit about the source code for the project in GitHub how is it structured I mean talk to us about that yeah so um let's see maybe I could just uh share a window really quick and just um

can we please just have it shared on your screen to a camera again yeah yeah let me let me do that um all right so I mean at this point in time I think most people uh you know in the scientific community and definitely in the development community and so on have probably um you know they're probably intimately familiar with GitHub and how it works and everything um the way that uh the the GitHub repo is structured here um so it used to be that there was um this is kind of a long time ago but it used to be that there was a main

um branch and then there was a development branch and then there were a bunch of tags and so the main the main branch was at that point in time it corresponded with um with the latest whatever the latest tag was was that what the same thing that you were supposed to get whenever you pulled Maine um we've since changed that so that Maine is development and I think that so this kind of you know it seems like a kind of a small change I think but um it was uh it took us some discussion to come to this it was kind of a a big

uh you know question mark as to whether or not we're going to do that and I think that the reason for that is because um GitHub is kind of serving multiple different purposes and multiple different communities here one of the um communities that it serves is obviously the Developers who write the code for apptainer but another Community historically that GitHub has served is the end users so GitHub was used not only as a as a place to develop code but also as a distribution platform and I think that um earlier in the project we were concerned that uh when end users downloaded you know GitHub

and get more generally um is kind of uh famous for being not always you know maybe uh having a having a hard learning curve right being a little bit harder especially for non-technical users to figure out and so I think that we were concerned that um you know some of our less technical users who just wanted to use the software might not realize that if they download and install Main um they're getting you know stuff that might not be stable it might be development stuff and so um we wanted them to be able to install the main branch uh and then have that stable since

you know this has been several years ago since then I think that you know there's other distribution methods that people are using like dnf and reposing things to install aptainer so I think that that concern is is a little bit um less of an issue now so anyhow there's the main repo and that's development um or the main branch and that's development and then we've got a couple of release branches here I think that so we're currently working on the 1.1 release Branch um you know there maybe we should have a 1.2 here as well I don't think there is a 1.2 release Branch but

there probably will be pretty soon um so and then 1.0 is probably deprecated because I don't think that we're we're currently uh you know working on 1.0 we might I don't know if there's some major bug fixes that went into 1.0 maybe we would back Port them um and then so these these branches here uh just have to do with some automated um some automated uh Bots that we have running in the background um and then tags this is how we do our releases so you know um each one of these tags corresponds to some release and of course you can also go over here

and you know see the releases so that's basically the way things are structured now and the way that um the way that it works is that whenever you know somebody creates a new pull request which is submerged it'll get merged I believe the way it's currently working so Dr Dave is the one who's currently pretty much running this so I'm I'm kind of guesstimating I I used to kind of run you know all this uh but you know for some time now he's he's kind of taking this over and he's been running um the releases and development and stuff but I think that the way

that it works currently is when you create a pull request uh it will you create that pull request against Main and it will get reviewed and um merged into Main and then we use cherry picking uh to um to pull that pull request uh just the commits from that pull request over here into I guess it would be the 1.1 release and get that ready so that we can you know we can release the next uh Point release here and so you'll see that this is just one 1.1 even though we're currently on 1.1.6 so the idea is that you know whatever the latest release

is that's that's this is what is going to be 1.1.7 in this case whenever 1.1.7 comes out but if you did mention one thing you were talking about bills going other places and people using dnf I know getting this into apple or extra packages for Enterprise Linux was a very important and big deal the doctor days spent a lot of time on it we really appreciate it but uh that changed installing this from an end user perspective for a lot of people I'm assuming can you explain that a little bit I know it's probably not the most complex of conversations but it's still good to

know yeah I mean it's um well it I mean there's there's a technical aspect of it and there's also kind of a like a social aspect of it and so uh that's that's a place where um you know Dr Dave has really had a major contribution for a long time um he has been the uh the long-running Apple maintainer um you know for for years now of the latest release of uh of app tainer um and previously singularity and um yeah I mean uh there's there's the hole just like trying to get it into the you know trying to package it as an RPM and

trying to do all all those types of things um but there's also um the issues of if you're gonna package it for Apple you can't just create an RPM and just you know that there's there's a there are a bunch of um conventions that you have to stick to and you have to kind of stay up to date on what's going on and you have to you know there's more to it so I said there's a social component to it and you know Dr Dave uh for people who don't know that's Dave Dykstra who works at fermilab um you know Dr Dave has kind of

stuck with that for a long time and has been in that community and has been a part of that and has has really you know stuck with that for um really for years and I think um more recently um I uh we have uh Yoshi akasen yoshiaki Senda um who works here at ciq who's another Solutions architect he's been packaging um he's been he's been working with packaging aptainer uh in um in apt in in as a as a Deb package so I'm not quite sure where that stands right now but I think that um you know he's he's also working with that and um

I don't know Brian Forrest do you know I mean is it what repo is that currently in because in PPA uh yeah but that's live now so you can at get um at thinner uh today if you wanted to and I think there's we also have a blog post out for that as well so yeah cool stuff surprised on that well maybe it's for day two I know sometimes packages in apple or in some of the other Debian ones they're behind significantly from what's in GitHub how quickly does this follow a release cycle from the time that it's and get to the time it's an

apple is there average time frame or I mean I know different things take different amounts of time but yeah I think the time frame is like a week maybe it's it's really uh it's really fast and I think that that I think we benefit greatly from having uh you know from from us being lucky enough to have Dr Dave both packaging this stuff um for apple and also being the release manager because he of course knows exactly you know when each new release comes out and what's in it uh I think the he he actually gets um gets the RPM into Apple pretty much like

within hours or less after the release actually happens but then there is a process by which uh users have to show basically I think that they're downloading and using and testing I think it's called karma within um within Apple but uh so it has to accrue you know has to accrue so much Karma before it's like it you know it it's it's said to be uh generally available for everybody but yeah that that usually happens pretty quick and that you know I think that's what that's why there's usually a small delay between the time um that uh Dr Dave puts in an apple and it

actually becomes available interesting thank you Dave so let's talk about raising issues so somebody's working with the tool how do they get involved and start raising issues yeah raising issues is super super easy and um you know once again if this is this is something that you know most people might know how to probably know how to do but just in case there's a few that don't or you know just if you're kind of new to the space or whatever um so yeah it's uh it's really easy to just go over so once again I was just at the main GitHub uh place and I

you know I just hit issues and went right over here and um so uh oh I gotta sign up for GitHub to do that um so basically you just click new issues you have to um you know I don't know if I want to sign in right here in front of everybody hold on I'm gonna say if you're gonna do that drop your screen yeah let me yeah thanks um it's all signed in real quick and then I'll show how you can create issues yeah maybe somebody else wants to uh comment a little bit while I try to figure out my username and password for

us have you submitted issues to snap Tanner never have myself um not you Brian uh I have not myself but there is an issue there that uh is a feature that I would like that has to do with um templating out uh definition files that which would make uh reusing your definition file for different versions of a software um easier to build yeah interesting so are you going to raise that one or do you want to ask ask whoever's watching to go do that for you well maybe I'll put in some PRS as well yes after after alerting everything from Dave oh yeah so the

thing that I wanted to highlight here is um oh and let me yeah cool thank you so the thing I wanted to highlight here is when you create a new issue um we have pre-populated so you can so this is all markdown so you can just you know you're right here and then this is what it ends up looking like we have pre-populated all uh all these different maybe it's easier so what version of apptainer you know what did you expect to happen what actually happened all this kind of stuff um I will say that some people basically just start off by deleting all this

and then just you know writing whatever um the thing that I wanted to highlight here is the ultimate goal of this issue if it's possible and it's not always possible because it's not always possible for us to replicate your environment but the ultimate goal of this issue is to give us enough information that we can take um this issue and we can run it locally and we can replicate whatever the the bug or the feature request or whatever it is that you're looking for so you know do do think about going ahead and filling in all these details about what's the operating system that you're

using and how did you install apptainer and you know what are the exact commands that you used to you know create this uh to create this bug or you know what what did you expect to happen or so on you know something like that so that's why I wanted to highlight there thank you Dave so I think the next topic we're going to talk about was PR's like going through that process yeah so um you know once again too if you're like already like a go developer or something this is probably pretty basic um but you know it's it's a little different for different projects

and so I just wanted to kind of go through uh the the the the steps of creating a pull request here and the steps of getting it reviewed and how that all works so um it might be a little bit different if you're used to working on a project uh you know where you've got um you've got the ability on the main repo to create a new Branch because here the first so if if you if you see a new feature you see a bug or something and you want to create a PR to address it the very first thing you want to do is

you want to Fork the repo um what that's going to do uh uh you know and it'll it'll ask you where do you want to Fork this thing to um I've already got a fork so I'm not going to do this but um so if you're you know part of multiple different organizations or whatever but you want to you you want to Fork this to a your private space and then once you've got that forked you can go over to yours your fork and you can um you can get clone that and then the next thing that you want to do which I always forget

is you want to create a branch on that you know locally on that um on that forked clone that you have and so then you're good to go you can go ahead and you know make whatever changes you want and then when you push that um you're going to push that to your fork and then through the GUI or whatever you can make a pull request from your fork directly back to um to Maine and so that's that's the process if you're new to all that if you're new to um GitHub in particular or if you're new to this project in particular you know it's

not like you just create a branch on the main repo and then you just uh you know do a pull request from that Branch to main because you don't have the ability to create a a branch on the main repo so you have to do that fork first and then create your PR from there another thing that like new developers you know just getting started um who don't maybe don't have a lot of experience maybe don't really realize like how do you how do you edit things how do you um so you know you're going to get a review and in that review you might

be asked to change a bunch of things within your code um if you just simply uh push more um commits directly to um to your fork the branch that you have set up in your local Fork then those will automatically go up and become part of the pr you know as you push them so you don't have to worry about like what's the process for making sure that my commits you know go from this uh forked uh Branch over to my PR that that syncing just kind of all happens automatically and there are I should show you too so whenever you so we got a

few different PRS open here let's see so whenever you um let me see this one here this one's not let's let's look at this one here so whenever you uh create a pull request uh we've got some you know we've got some continuous integration which runs in the background and it runs all these tests and that's another thing that you'll want to check out um the the um the tests do everything from like a bunch of you know basic uh like actual functional tests where they actually uh you know download containers in VMS that are you know running up on the cloud and do things

with them to just um linting your code and so one thing that you'll sometimes you'll you'll you'll find is that tests fail and it's just because whatever editor you were using um didn't have like a go linter installed and so because of that I mean there's stuff that still works but there's stuff that's not like you know like correct syntactically um doesn't look right and so this linter will catch that so sometimes you'll have things fail and you'll be if you're kind of new and you know you're just starting out that can be frustrating to not know you know what's going on and why things

failed and so uh you know just make sure that you're using a linter um another thing I think is that we are uh I think that uh the project now requires that you um verify your commits and so that's that's um that's another little gotcha that if you submit a PR and your commits aren't verified then you know you'll get nabbed by one of those tests there and you know it can be confusing and frustrating try to figure out what's going on so just make sure that you verify uh your commits and that there's a there's a pretty simple way to do that I can't

remember off the top of my head how to do that but I I was able to figure out pretty pretty quickly how to do that when I needed to um and then I I jumped into here real quick because if you do have tests that are failing um you know if you see something like this then you can just go in here and and and have a look and um you know it it can be kind of daunting at first to try to figure out what's going on but um you know uh I don't know if I'll be able to figure out what's going on

in this example but usually you can parse through this and you can you can figure out okay all these are running successful okay now here's the the part at which up Lent so this might be an issue where you know once again like I was talking about before the linter caught something which was not correct [Music] I'm unsure but yeah so this is the method of going through and figuring out you know if you've got something that's failing why it's failing and then you can you know get an idea of how to fix it so after all your tests passed another thing that has to

happen is you have to have a review um and those reviews are usually uh done by the folks that are on the Linux steering committee um so those would be the people like you know Dr Dave uh Ian kind of kenoshiro Cedric um you know other folks like that that can actually re do your review and I think actually anybody can review um but I don't know if those reviews actually count toward um like like making it so that you know so that this PR can be merged yet so that's something that if you want to get involved in um you can actually I think

that anybody can go and and review PRS so you could go here and if there's anything that that looks interesting to you you could go through and test it and check it out and you know look at it and then add your review to it thank you Dave so once you pass all the tests and once you pass reviews then you know you're uh PR will be merged hopefully let's talk about actually cutting a release how does a release manager actually go about cutting a release yeah so um the way that that's happening right now um so let's see if I can find the Milestones

um let me go back into the pull request for a second so we've got Milestones set up um and this is something that uh you know Dr Dave does um so right now we're targeting uh release 1.1.7 and 1.2.0 um we kind of follow semantic versioning uh to an extent but not com completely and so the the micro release here like the 0.7 here um that's going to denote uh it should denote bug fixes and changes that don't change um the behavior so changes that don't change things having to do with the CLI or you know anything that the the user is going to touch

the minor version here so the point one here or the 0.2 here those are going to denote things that you know might change Behavior as far as the user is concerned so like you know you might have a flag that you were using or an option or an argument um that uh that you know Works in a previous release that's going to change in this new release and historically we have not changed um the major version version release unless um the the actual format changes so usually that's been and that and that's why I say that we don't we kind of follow semantic uh versioning

but not necessarily exactly so uh in the past we've changed that um because the you know we we went from using ext3 images to Sif images and we also had like a major code rewrite where we went from C and Bash to go and so that's when we went to um and when with Singularity when we went to 2.0 um but so so the release manager then can put together these Milestones these Milestones have uh you know a list of uh different uh PRS that we expect to be merge or issues that we expect to be um fixed before the next release is is cut

um and then what the you know what they'll do is they'll kind of go and this will this could change um and a lot of times as the the release becomes nearer and nearer you'll see that people are like they sort of start to clamor um for uh PRS and features and things that they really like to be added in and that can become that can become difficult because um sometimes I don't know how how much we're doing it now but sometimes you'll release a release candidate and the idea behind a release candidate is that okay we're now freezing development we're not going to add

any new features this candidate is you know what we expect to release ultimately to be and we're going to release the release candidate and we're going to expect the community to install that and test it and see if it's got any bugs basically see if see if it's got any um issues that we're not we couldn't catch via automated testing and so at that stage sometimes you'll see that people are like oh no but there's a there's this new feature that I really really want to put in and it's like well you know I'm sorry but technically we need to wait until the release you

know until the next actual release because otherwise we have to put out another release candidate and we have to start the whole Community testing again and you know the communities only got so much testing in them you know sometimes you get more or less testing out of a release candidate so um I think what we've moved to now especially since the project has become a bit more stable is um the actual releases themselves uh you know those are those are what goes out because we don't expect there to be uh as many huge breaking changes or bugs or whatever in the releases themselves and so

I think we're getting away from release candidates to some extent but the flip side of that is that we might have more uh micro releases right so you might have a bug in a in a particular micro release and you know that's okay and uh so you might do another one to take care of that and so then the actual you know process of doing a release itself is you know just making sure that all the all the changes are in they've all been you know cherry-picked inappropriately sometimes when you cherry pick a a set of PRS in you get um you know basically what

you're doing is you're taking a set of commits that were made for one branch and you're pulling them out and you're kind of messing with the history and pulling them out and putting them into another branch and because of that you can get conflicts you can get like the same file was was you know changed between these two branches and then you're trying to add more changes to that and so those have to be manually resolved somebody has to go through and actually look at that stuff and figure out um you know what what's going to work and what makes sense and um it used

to be that uh the release manager basically me at that point in time would do all the Cherry picks for the release myself and then if there was anything that I could not figure out how to resolve on my own I would go and contact hopefully the developer who actually created that PR or one of the other developers and sit down with them and try to figure out was the proper thing to do now I think that the way we've instituted it is it's incumbent upon the person who created the pr to do the cherry pick and that's good because they're usually in the best

position to be able to um you know look at look at the way the code has changed and figure out the best way to resolve any conflicts so anyway the release manager will go through and make sure that all that stuff has been resolved properly and is all in there the way it ought to be um they will there's a few little housekeeping things that they have to do they've got to bump the version number and a few different files make sure the change log is up to date uh and then basically just uh create a new tag cut the new release and um document

all the stuff on the releases page as to you know what's changed and everything there's usually a little bit of um a few emails that have to go out a few announcements that have to go in different places and then I I don't know if we're currently doing this but it's a good practice afterward to take that release branch and merge that back into main so for what you've been doing is you've been taking everything from Maine and you've been you know copying that uh as needed into the release Branch you might have some changes that ended up in the release Branch um you know

through resolving conflicts and doing stuff like that that you ultimately want to go back into Maine so that you can keep those two as synced up as possible as development continues so that's that's the last thing they'll usually do to be normal thank you Dave I think we had a few questions that got thrown up before we move on to the next topic um why you got to give away give Hub window open day can you show us the get involved with development part yeah certainly yeah so that's um so you know once again one of the things that you can do is you can

go over here to um to the issues um and you can create new issues or I mean you can just you can just hang out and kind of read through these issues this is a great place to uh you know people sometimes people have issues that are actually bugs that are actually things that need to be um changed in the code in order to work properly but sometimes people just need help right they just need this is a place where people go to get uh support and so this is a great way to get yourself um more familiar uh with the project at a higher

level because you know it's one thing to just use it for your own purposes and to fix problems that you have it's a totally other thing to you know try to get into somebody else's headspace and figure out their environment and look at what they're using it for and you learn a lot by going through you know issues and trying to figure this stuff out um and another place like I said you can go through these pull requests and you can sort of you know uh look at um you know what looks good and what doesn't you can review the code if that's something that

you're interested in in getting into um if you want to start writing code for apptainer then a great place to start off is by reviewing um you know PRS and having a look at them and seeing what codes what code changes so yeah this is a way that you can get involved um another great way to get involved is to join the uh the slack channel so uh apptainer has a slack Channel I can let's see I think we've already posted a link if I I saw it correctly I think for us put it out there oh cool thanks Forrest yeah it's out there yeah

so that's another great place that you can get involved is just sort of you know watching what's going on in slack and helping helping people out when they need it asking questions that kind of stuff oh and um there's also a it's announced in the slack Channel but there is a twice monthly community meeting and that is a really really great place you know for just an hour uh twice a month if you're interested in like you know what's going to go in the next release and what are the current um problems that the developers are thinking about and what are we looking at trying

to incorporate into apptainer for the future that uh that community meeting and you know Dr Dave announces that in the slack Channel um because he he runs that community meeting right now as well so he announces that in the slack channel uh and that's a great place to go to just kind of keep an eye on you know what's being developed and what's new in apptainer Dave is that also a place where people could go if they wanted to hear what things needed to be or where people were looking for help what Dave needed help on absolutely yeah or um where the developers need help

too so it's like we've got uh you know these changes that we'd like to see incorporated into abtainer but we just don't really have anybody to work on this stuff right now I mean that's the kind of thing that that gets discussed quite frequently in those meetings thank you all right as a user what issues have you come across it's a good question for Brian in Forrest um I honestly can't speak to any specific experiences that I've had with bugs in obtainer for the most part since I started using it it's been pretty much Rock Solid for the use cases that I was using it

for um I you know honestly I've always had a great experience using obtainer it's been one of my favorite tools since it was Singularity I um as soon as I found it started trying the you know make a lot more things containerized back in the day as far as my HBC soccer deployments one um so I would say uh you can't really think of anything too specific it's um it's been a rock salt experience for the most part for me also no real no real issues more of just like a small pain point on just managing multiple files for different versions of the same software

but uh you know we can we can solve that with a little bit of templating so yeah that's great thank you and I see someone asked if we could post the slack link into LinkedIn as well appreciate that I think one of the other questions that was asked was for Brian Forrest as well what uses have you seen or what have you seen people using apptainer for um so like I said my original experience without Tanner was using it to deploy software in an HBC environment so we had a pretty standard compute cluster that you might find that most universities in America a nice um

you know 50 150 200 node cluster or something like that I don't remember the exact account but a nice mid-size cluster um like I said when I first started I was doing deployments using kind of standard module files stacks and that type of thing I found pretty quickly after I started looking into the obtainer that the ability to for one be able to build and rebuild a piece of software to kind of do iterations on getting it to build correctly um it's both happened to constantly be deleting and kind of recreating a build environment on the cluster being able to um honestly that was one

of the biggest things that I found at the time not having to go and you know do this kind of you know CM shared apps CM shared module files Shuffle um to give my user uh you know permission there and you know just to do that type of software deployment safely with that um I think one of the most memorable things that I ever put into a container was a large stack of software for a bird research or an avian research agency called the Peregrine fund um one of the researchers was doing large-scale research into the um movement of eagles through North America and so

they were using a containerized environment built together to do kind of mass data analysis of that and then we had another guy who was using that same containerized environment to to do analysis of flight paths as birds were flying towards wind turbines so you could try to come up with models of um how to best time like turning off wind turbines so that groups of birds can pass by without being hit by them um like I said all that research was done out of a singularity container that I put together for them and that is the coolest specific example um I point to something that

uh I've seen deployed without painter that I've personally done but that's just one of many examples of other times where um obtainer proved to be the best option for deployment yeah for me uh very similar experiences to Forest uh mainly used uh apptainer to get customers up and running as fast as possible on the HPC system so this ranges from containerizing maybe a single application to run like a physics simulation to maybe to even like a suite of tools for example to run a bioinformatics application that calls multiple tools foreign thank you for us Dave I think you also wanted to give us a few

tips and tricks here for getting familiar with code yeah well this is just um yeah so uh I wanted to do a a little demo um this is maybe kind of I don't know hopefully hopefully people find this useful um let me see if I can present it once again I always have trouble figuring out what screen to present I got too many stuff too many things going on here okay so if you're like me um you know so so I'll give you my background a little bit of my coding background my coding background is kind of silly in that uh I used to be

like a scientific coding person meaning that the majority of my my coding background was done uh in an IDE in an integrated development environment now people have mixed feelings about that some people you know like me um really thrive on that and really like that and then some people kind of you know think that that's a crutch and are not so hot on uh Ides but if you like an IDE and you're new to obtainer and maybe you're new to go and you want to get um involved but you don't really know where to start uh I I wrote a little blog post um that

came out yesterday in which I detailed the steps so if I do like so you could search like ciq blogs if you do that you'll go to ciq's website and you'll be able to see um our blog posts and the most recent one that came out yesterday is this one about uh debugging obtainer and actually other other go projects using delve which is the like um the major uh go debugger and also integrating that with something like vs code and so what you end up with is the ability to like step through your code line by line set break points interrogate variables at runtime and

do stuff like that and kind of see what's going on and so I just wanted to show you so the steps to set this up are here in this blog post I've already got this set up here on this um machine and so I just want to kind of show you what that looks like and how you could use that if you're just starting off to get a good idea of what you should be looking at um so the way in which I've set this up you basically run an apptainer command at the terminal here and then that command um kind of pauses and waits

for you and then you can attach to that command using vs code let me show you what I mean so I'm just going to go ahead and run a silly command um just using you know the uh LOL Cal container which is up on Docker hub maybe I won't um I got a typo here what's my typo unable to handle the URL maybe I'm not attached to the authorization report all right let me see something else that I've got that already works there we go okay so I've got a typo up here somewhere I'm not sure exactly what it is all right um so let's

say that I wanted to debug that command I want to step through and actually see how it runs um so I've got here an alias set up because I can never remember the entire command um and the Alias is DB or debug uh Dash app and it it resolves to this entire long command and so what it's going to do is it's going to start a delve um debug server running headlessly and listening on a particular port um and then it's going to end with these two things that say okay I'm done giving you options and that's going to allow you to to give this

command uh whatever other options and arguments that would normally go to apptainer so after I've got this set up this little command here is just going to replace is just going to replace my aptainer command up above Okay so the way that that works is I would do DB app I'll install okay you know what I'm sorry guys um let me that's not going to work hard off the top of my head let me um uh go off screen and try to figure out why that's not working really quick I messed with my um I mess with my bash RC right before this and I

think I messed some stuff up sorry about that it's always fun I hope you made a backup yeah no it's been about bash RC I had an interesting conversation yesterday where someone was trying to do some things in bash and they were using chat GPT to create their bash RC it was kind of fun to play with and try gave some interesting ideas for it you should try it I uh I think have a terminal opening the gpt4 at the moment just for the record so on Deck I don't know it really would involved syntax highlight syntax highlighting and spacing for python it was an

interesting conversation I still wish we could do this where we could turn Dave's screen around again that was still one of my favorite moments that's so good so Brian of the University space I know you've been spending quite a bit of time there lately have you seen them from a non-containerized environment have you seen them starting to adopt and really kind of pick up and start looking for how can they start utilizing out here in that environment uh yeah yeah they have uh so in the environment that I'm working with right now they used to have a lot of software installed just on an NFS

share and uh we are we're currently working with them to contain our eyes both some commercial applications and also some open source stuff as well how's it going with the commercial side of it I know we've had several conversations around containerizing some of the commercial software and Licensing issues are we starting to see as an industry that that's becoming something we can do a little easier the vendors are getting a little more integrated into that type of environment um I think it varies from vendor to vendor uh as as of right now um and yeah yeah that's that's kind of how I see it um

yeah that's it's interesting whatever you start looking at places that do a per socket per core per host it gets really interesting with containers yes yes especially in something where it moves stuff around all the time is it going to work Dave do we have any other questions that'd be great um yeah I'm getting there so we could um I could share again if that's uh useful whatever you want to do I can start asking Force other questions like it almost sounded like it was raining earlier at forests it is here it was uh hailing through my door earlier because I need to fix this

seal on it but yes that is not cool all right so yeah so hopefully this will work this time so um so what this is going to do it's gonna so I ran the same command that I ran previously but I I replaced it instead of obtainer I Ran this uh DB Dash app command and then you know all those nonsense scroll by on my screen but what it's doing is it's it's basically waiting now for me to connect so I can go over here to my um so when what I've done now is I've so I've downloaded the app tainer um source code and

I've got that in this repo and I've opened up this entire directory here and within that directory um you can see that I'm in this it's it's just like the main entry point for apptainer the main package this uh cli.go and I've set this break point here and so what I can do now um that I've actually you know I'm here um is I can uh go ahead and press F5 and that's going to drop me down into the code where I can actually interrogate and see what's going on okay so now I'm I'm actually running here and so I can you know I can

step in I can step over I can step out I can do things like this and so now I can actually start stepping through the code and see what happens the first function which gets hit is this execute obtainer function um and so I can step in and and go through his and I can see that um you know this this uh variable is set to args this variable args is set to the following you know and this is the arguments that that get input uh to apptain or whenever I run it and so you can you can step through the code using stuff like

this and you know go through um you know piece by piece and and kind of see uh you know you know what's going on I was going to demo it but then I had this little hiccup and so I don't think I've got enough time right now but I'll just go ahead and and kind of say um the source code so okay so a couple things number one this is a little clunky to get set up once you get it and so there's a little bit of a hump to get over there and so this isn't for everybody some people are going to be you

know happier to just use them or something and just debug things with print statements and go through and look at things like that but if this is your cup of tea this can be a you know a good way to get started um with the code um but there is a little bit of it's a little hard to set up there's a little bit of a hump to get over there so you know unless you're going to really sit down for a little while and kind of try to understand the code and devote some time to it um you know this is this is probably

a little too much startup cost to get into that's one thing another thing is the way I just showed you so this was the demo I was going to do but I don't have time for now but the way that I just showed you um starting at the beginning and then reading through the code um that that'll work for smaller projects but for apptainer that's probably not the way in which you want to familiarize yourself with the code I would think I mean that there's a lot to apptainer if you just start at the beginning and try to work your way through it's going to

take you forever to try to figure out what's going on so I would suggest um if you're interested in a particular so say you're interested in the code that that downloads containers from Docker hub for instance but you don't really know where that code is you don't really know how to get started you don't really know how to look at it run apptainer once with the dash dash debug option to give you a bunch of verbose output and find within that verbose output a message that indicates that there that the container is being downloaded from Docker Hub or whatever as as you're doing it grep

in the code for that message and that'll give you the file that is important you know for that particular operation and then you can go in and you can stick some break points in that file and you can run the same command again you can pop right into that file and you can start looking all around and figuring out okay how do these variables get set what's the stack as far as you know where am I at where am I at as far as you know what functions are being called and and from there you can start to really interrogate that section of code and

start to you know figure out how that works and if there's a bug in it you can you know try to figure that out instead of going through the entire code I feel like that's a really good way to get started as a developer and to just kind of you know get your feet wet that's great thank you Dave so we are actually up on time Dave I appreciate it thank you very much Brian Forrest it's always great to see you thank you for joining the conversation uh join us again next week for a round table and then we will see what is coming the

week after that but thank you for the conversation thank you for watching we will see you next time bye everyone [Music]