Apptainer in Bio Sciences

hide for too long right because i think i would like to level set a little bit and just take a few minutes for those who haven't done this before and just kind of level sort of what singularity is and then i have another question that's come up that's a really good question about the linux foundation but if we could start off and just tell us a little bit what is apptaner uh kind of where did it come from how's it different than podman docker etc yeah absolutely so um just as a quick kind of historic uh for historic relevance and how we got here uh

you know a number of years ago man dave and i used to talk about this constantly as we talked about it it used to be like you know two years ago three years ago i think now we're like five years ago maybe six years ago uh customers and and researchers and scientists uh started getting wind of this new idea called containers and uh i got it i got it a lot from um uc especially so i was joined appointment from berkeley lab doe as well as uc op uh specifically on appointment with uc berkeley and i got constantly we had you know people coming at

you know science initially from the long tail so it was like political science it was library science and and others that you never would have thought would be high performance computing consumers started asking us about like can we can we go run this operating system or can we do this differently and if you just supported containers all of this would be so easy so we started looking at how do we incorporate containers into high performance computing and the first thing that we found is the existing container ecosystem is really designed to solve a problem around services and root running services in particular and that didn't

really fit well for the architecture that we were using for high performance computing which is the the traditional beowulf and where you can have any number of users on your system at any given point if you give any of them access to a root running service technically that's a security breach so we had to figure out how to solve that and that's actually what what what caused me originally to prototype singularity and within you know six to eight months singularity was used on the majority of high performance computing systems worldwide it took off massively fast um as dave mentioned you know we uh we created

a company uh called scilabs focused on that um when uh myself dave included as well as others uh left scilabs uh you know we took the open source uh singularity project um out of the company and we brought it with us to ensure that it's gonna stay in the community um and uh keep continuing down that vein what we decided is you know the best way to ensure that this stays open source and has a very long life ahead of it and cross-pollinates with various other um aspects in the in in the in the community specifically around uh oci the open containers initiative and the

cloud native computing foundation it makes a lot of sense to move this into the linux foundation and that's where we had conversations about that and the one request that linux foundation had they were very excited about the prospect of moving this in and everyone was unanimous on this but the one request that they had is please rename it um there's two singularity as as a term as a search term is just too it's used too often it's all over the place and um it's too difficult for them to protect the brand of the project and protect the project itself so they ask for a new

name something that's unique and so we put that back out to the community the community came back and said after a vote of about 50 people who said they wanted to be part of that process that apptaner was the uh the name we ended up going with and that's where we are today and i feel like i say this every time i i always revert back and say singularity so i'm still trying i'm getting better we all do i i do the exact same thing painful so whenever you talk about the linux foundation that was one of the questions that that came up that's really

interesting to me is what are the requirements that the linux foundation has or what do they place in a project before it will sponsor it or as a part of that process so there wasn't a there wasn't a huge amount aside from the fact that it is truly a community um it's not corporate controlled and if it is corporate control they ask that it comes out of the out of the company and goes into the linux foundation but we basically assign all aspects of the project to the linux foundation so the linux foundation is the owner of the project and um so you know trademarks

brands all of that sort of stuff linux foundation owns and and they now maintain it there is a charter involved in doing that which basically requires the organization of the project how think certain things are going to happen with regards to uh decision-making process what they like to call the technical steering committee which you can almost think of as the board of the project uh the linux foundation has the ability to do monetization on projects as well uh we have chosen not to do monetization um on on apptainer just because we wanted it to be you know just well we didn't see a huge need

to we may change that at some point in the future but we just didn't see a huge need to monetize on it uh for the project's benefit and um uh yeah so you basically you hand over everything over you assign everything over to them uh they they come up with the charter you agree to the charter you assign the initial technical steering committee which by the definition of the charter is the committers of the project so those who have commit access uh or those merging or responsible for emerging prs contributors of course are the people generating prs committers are the people that are responsible for

merging the committers whoever those committers are are the people par that are on the technical steering committee excellent thank you for that oh and and who is the technical steering committee you should probably mention that as well so we have a a a few different people on the technical steering committee uh there's myself there's ian kenoshero there's cedric uh cluj um there is uh dave dykstra from fermi lab and christina marica marieke from um um kla excellent it's always good to know who's who's involved appreciate it one of the other questions that i get quite often is is that tinder going to continue to be

hpc focused or are there other other outside of hpc use cases that that people are looking at um i'm gonna i'm gonna take a stab at this but i don't want to talk too much but i think this is the last probably last couple points that i may actually be relevant on as we start talking more about science and smarter things um i would say yes apptaner is going to be focused on the high performance computing side but that doesn't mean that those are all the use cases that it is going to be used for uh one use case that we're familiar already with people

using it for is high security environments and uh singularity and aptaner have some very i actually did that i called it singularity and i corrected myself at least it wasn't me i made it i made it totally smooth you probably almost didn't couldn't tell um but uh Apptainer and singularity both have some very unique ways that it manages its containers uh in in the image format and that image format allows us to do things like cryptographic signing verification uh layering cryptographic signatures as well as just encrypting that image so only people who have the passphrase or the keys to that can actually uh unlock a

container so there's a lot of high security and high trust use cases around um apptaner as well but the one thing i would just really just kind of end this with is or or this question with is uh the name actually came from application container and we we see a big difference uh in terms of usage between service focused containers and application focused containers and uh there's good use cases for both of those and one thing that i i personally believe in is always use the right tool for the job and uh if you're if you're bringing up high performance computing applications mpi applications x

or gui applications and whatnot you can do it in a service focused container but it's it's quite a bit more difficult uh it's not designed specifically for that where apptaner absolutely is designed exactly for that so it works very well at that nice thank you so i have a question and i'm gonna i'm gonna throw this one to glenn just because it's it's gonna use one of his words that that he introduced me to so if i have a doctor container how do i transmogrify cleanse word that into Apptainer i'm to try not to say the the the s word so um yeah so you

know doing having having done that a lot uh and finding a lot of docker containers being the predominant um container format for a lot of the bioinformatics uh tools out there um you know we simply you know import it into apptaner we switched in my last job we switched to app chainer i don't know six uh sometime into last year i think and you know it replaced the s word um flawlessly so we just kept using the same you know the same commands and and everything just uh we were able to import uh docker containers into uh into app uh easily we also we also

started using podman more for uh for for docker containers just so we could get a demonless you know um process running essentially so um yeah and so transmogrify is a little bit maybe heavy-handed because it seems to be a a pretty easy process to get docker containers running in Apptainer um but we ended up doing it a lot and and a lot of our workflows you know we'll search you know it's like if if you know if this is a docker container you know transmogrified into uh into an Apptainer container um so so it's not that hard it's actually it's actually pretty easy excellent

so on on that let's get into the actual conversation that i think is going to be interesting today and let's start talking about how people are using apptander today so i mean this is really dave glenn how are life sciences leveraging containers and attainer i'd like to talk a little bit about that because um i'd like to talk about that a little bit from a historical perspective because i recently had kind of a bit of an epiphany as i was writing some documentation that the use case is kind of you know it's changed around a little bit it's a little bit different from what it

was originally um so originally so one of one of the things i always kind of try to stress is uh and this is something i came to recently is that there's a differentiation in the types of users between producers and consumers when it comes to containers and i think when we first started uh you know developing um you know when when greg first started uh putting together this you know what was then singularity and is now apptaner uh and he started getting the community together there really wasn't a good way to ingest docker containers in and so if you were going to be a consumer

of containers you had to be a producer first and so that was one you know the way in which we wrote the documentation and everything was to start off with how do you build a container you know that's that's square one and um recently uh in going through some of the old documentation that we put together and updating it i had the epiphany that we had the entire documentation backwards we needed to flip it all around and uh write an entire you know section on how you use containers first and then have kind of almost as a footnote at the end by the way if

you also want to build your own container there's a way to do that too and here's how so and and that's really um you know that's that's really a change and it's really it shows how how the community has grown now and um how how containers have been adopted to the point where if you want to use a container nine times out of ten you don't need to start you know writing a definition file or a docker file or something you just you know the first thing you do is you go to docker hub or another registry and you look around and you find the

container that it is that you want to use and you download it you use it excellent glad you have anything to add to that uh yeah so i having tried to build a lot of bioinformatics tools over the years and having scars from it um i'm encouraged to see that a lot more researchers are not just you know using github and posting their code so it's easier to get instead of having to email somebody who left their postdoc to become a professor and can't you know bother be bothered to send you their code um so i'm really encouraged by that even more so now that

they're actually um posting them uh as in containers um and a lot of uh and some of the more enlightened researchers i think will post both container both containers and Apptainer and uh and docker and so it makes it makes sharing uh containers code and you know scientific reproducibility much much easier than it uh than it was about before because uh you know even compiling a different platforms you know we know can come up with different results right as small as they may be um so i i'm really encouraged by that um and then for users to you know get other people's code so it

kind of goes both ways sharing it from the original um original creator of the of the software and then people who want to reproduce it and then pass it on it's also easier for people to get you know like researchers at my last my last org would come in and say yeah we got this code we want to run and instead of the the admins having to spend a week trying to build it you know they can just spin up a container and test it really quickly for them um or they could just run it on the laptop and it's not on the hpc cluster

and so that that runs a lot a lot quicker too so i see i see the the pace of reproducible research picking up uh really quickly you just brought up a really a couple really good points um first off the the mobility and reproducibility of these containers uh you know something i i said a lot in the past and i i know that there's certain since we're talking bio i know there's a there's certain um you know pharmaceutical companies and bio-focused organizations and research that whenever workloads are being used clinically or software's being used clinically it is actually there are certain mandates according to the

fda that require that that software stack is is managed very similarly to a medical device and i'm wondering have either of you you you guys had similar experiences with regards to compliance maybe not specifically fda but compliance and and legal meds of reproducibility no not me in particular and you know i think that um probably because the nih is a little bit more on the research side versus the actual like you know applied medical side so i you know if you got more into another agency um you know or more like a hospital setting i think that you might you might get into that a

little bit but i don't know that's just my experience i don't know glenn if you've seen um those sorts of things in your experience yeah so we've uh at tgen we had several or several startups kind of be incubated and then spun out and they were all clinical um bioinformatics organizations so diagnostic uh tests and other clinical sequencing and the the code there wasn't it was it was locked down but it wasn't in containers right they had they had an auditable process but and this should come as no shock it's actually the uh the data that's actually more scrutinized with regard to compliance how long

you're going to keep it is it encrypted uh where is it going to be for the next seven to ten years or infinity how can users can contact you and have it deleted um so all those processes have to be have to be sorted out and and the software was kind of the easiest thing to do um however uh i would recommend you know if someone had asked me because a lot of these you know work started before i got there you know why don't we put it in containers so we could we could more easily you know pass an audit you know for example

right and uh and and sign things you know as opposed to just trusting that you know the bwa version you downloaded two years ago is actually you know doesn't contain any any um bad code or or or is you know just a crummy version so um it's it's it's a big deal it's going to be a bigger deal um i think as we try to get the kind of cover the last mile of of software integrity so there are companies out there that want to help researchers uh create encrypted enclaves in in computer ram so that their their code their data and their and their

models can't be altered even in in dram so i i think using containers there will be a lot easier as well to help uh to help manage those uh those encrypted enclaves so you brought up before you also brought up the build infrastructure uh and and not directly but the build process i guess i should say what i really like to see and this was part of the reason why i was so excited about moving singularity into the linux foundation is because uh the amount of innovation and capability growth that has occurred with the rest of the ecosystem around containers is massive and the fact

that you can have get operations feed into a cicd pipeline and automatically generate containers on a on a get up like a like a push or a tag is is truly remarkable and the fact that you can then automate that entire pipeline and then automatically integrate that into your your research is is fantastic so i'm really excited about that uh dave to your point about flipping kind of the documentation upside down totally agree and i'd love to see more emphasis on how do we make use of cicd for container creation generation and how could it be such that anybody who's who's a researcher scientist or

creator producer of these applications can automatically be creating these containers and the format doesn't matter i like oci a lot because it works pretty much everywhere apptaner and singularity both can leverage oci directly but you can also create sif files and then even cryptographically sign those zip files and then the second point i just wanted to make really quick was as we're talking about reproducibility of of these containers and portability of these containers knowing where you got these containers is actually really important now for the most part we've all just been kind of ignoring this and and not paying much attention to like there's a

docker there's a there's a container up in docker hub i'm just going to go ahead and pull it and use it and see how it works there's a risk associated with that if you don't have uh validation of that to some extent and we already have seen certain container viruses or malware going out there and there was one that i thought was hilarious which was uh every time somebody ran a particular container it started doing crypto mining on your own research and your own resources but putting it into someone else's you know coin account uh that was brilliant but it was a malicious container right

you didn't expect that to happen uh we need to have some sort of uh provenance management of provenance of the of these containerized workloads and and trust of these and i do think you know there's been a lot of pr forward progress moving out of oci and cncf around exactly this so i'm really looking forward to that and i'm looking forward to integrating that uh into optaner and making this even easier and better for for users thank you greg so we do have one question and is uh does anyone on the panel have or envision a standard and i'm going to say doy this time

and not doi greg since you fixed me last time four containers if you uh publish a paper uses codes systems for computation to share the code and those systems via doi for a container you know it kind of you know making me sound like the homer simpson you know the bit of dough right um uh but yeah people use both but i think the the more common uh was was is doi for that but that's a great question um and this was uh this was one of the initial um goals and motivation of a project called singularityhub which uh was started in early on through

through singularity um and uh vanessa created this project and uh that was the goal behind that is to basically have a place where you can post your and have a uuid associated with that container such that anybody in a publication or or wherever can basically reach out to there and touch that our previous company scilabs david myself ian cedric and others were all in we also had something very similar to that uh called the library so there are solutions for this um and uh in an oci i mean i see people now using docker hub almost as a as a toy for um you know

their research papers and whatnot and as a reference i think we need to do more of that i absolutely think we need to be doing more of that so uh in every paper every research paper there should be a if there's software associated of course there should be a link not only to the source code and recipe for how do they create their environment but the actual containers that they use to generate that uh that analysis and again going for reproducibility is so critical here um i don't think so you can do there there's multiple ways of of registering and obtaining toys uh for various

you know prepared papers for software releases and whatnot i don't know if anyone is using the official doi um uh infrastructure for a container but i'm gonna ask dave and glenn if they have more you know any more insight on that than i do so i i don't know if anyone does but i was just trying to quickly research so giga science uh the journal vega science is kind of one of the leaders here i think um in in also providing like toys for data sets so with with uh with the gig of science when you when you submit your your your your your paper

you also have to submit your data and uh and make it available and i believe i believe that gets a doy as well separate from from the article but they're you know they're associated so i'd like to see that type of um registry for for containers as well because to greg's point and like what i mentioned before it's it's one thing to have github have your code and a link to your container um but you will you know will your will your github account be around forever right well you know you always have access to it uh if you move from you know from place

to place or or what happens if uh you know you want to take it uh the software and make a company out of it and shut some of it down and and keep some of it open so i i think you know for that split second in time that has to be you know fossilized in time so that people can always go back to that point in time and not have to use whatever comes you know upstream of that so um so a singularity hub uh kind of makes sense you could have a separate hub but then i think more journals should start requ you

know requiring essentially uh dois for for containers um and maybe we have to get you know a build process that's approved you know for for journals as well like you could like i could put up a container there that starts bitmining all your computers so um right so and and and you know i may i make that joke but you know i just realized you know the the the number of like illegitimate journals right that are out there now asking you know requiring requesting papers i don't know if any of you guys have seen some of this stuff but like some people have published just

complete nonsense in these you know online journals that you know you know um make themselves sound respectable and uh it's you know so if someone like falls for that right and just you know puts their container up there um or you know they start putting up a bunch of you know malware filled containers and then someone someone you know not realizing that's not a really you know legitimate journal downloads that then they're you know then they're infected right or then they're part of the you know the bit mining and so you know we might start needing somewhat of a process that's certified or certifiable i

guess when we build containers before we we put them up um before they're posted and maybe and maybe that's something the nih or nsf could be a part of i you know i don't know but obviously it has to be a global kind of agreement so i don't know if you know the sanger and the road get together and g4a uh the global alliance for genomic health um they're they're big into api development you know maybe it's something that you know they could jump ahead on and start uh thinking about i know dave we kind of talked about this yesterday a little bit having some

sort of process behind it and even kind of the beauty behind that taner is being able to do cryptographic signings so that you can ensure what's in there if you want to share some thoughts on that yeah so my two points um so i've got a couple of points i'd like to make based on so so you sparked a couple of thoughts in my head uh when you were talking there glenn so one of them is um you know going back to mystic knight's question is there gonna be uh you know do we envision any um standard for a doi uh you know kind of

system well the tools are all out there that's kind of like the the in fact there's there's multiple there's multiple different ways already to do this there's lots and lots of tools that are out there to ensure that you know you can you can use shaw sums you can get you can you know sign and verify your containers you can do things to make sure that you're getting bit for bit reproductions of the original container that the author intended but i think what we really need is um you know some more community agreement on what kinds of tools we're going to use and how we're

going to implement those and that's what's kind of missing right now is that there's there's lots of tools but it's kind of like the wild west as far as how people decide to implement them or whether they decide to implement them or so on so we need we need kind of like you were saying glenn we need um you know to get together and uh you know try to come up with some standards um around this tooling if this is gonna succeed if we're gonna have this kind of uh infrastructure in place and then my second point that you kind of like you know sparked

in my mind and this is i've said this before i've said this before in some pretty public forums so people have probably heard me talking about this in the past um but i'm going to get up on my soapbox for for a second i think most people understand this now it used to be that very few people understood this but you know if you're look if you're interested in reproducibility and you're interested in you know making sure you get exactly the same uh code base in your container every time you cannot rely on a def file or a doctor file you really have to have

that container you have to have those bits and um you know this was i really got surprised one night i mean you know i have an antidote that i that i often share you know one night i was doing some um some testing on a tensorflow container and i was building the container and testing it and building the container and testing it and as i was doing this iteration all sudden the container build stopped working and i couldn't understand why and it was because on docker hub that tag had been removed that i was using and so the internet is subject to change and it's

changing all the time and your your definition file is you know the same definition file that you created five minutes ago might not work again so if if you really are interested in reproducibility you have to have the container save the container don't save the the file that created the container i think what you're saying dave is always use latest no matter what no no so here's actually a good question so what about work during covet or in covet how did that play out what any any thoughts on that glenn yeah so uh my my prior uh employer we we sequenced like 70 of

uh the coveted tests uh in in arizona and uh we were set up for our pipeline was you know set up very well but it could it could have been improved but because i saw all over the internet um people posting containers specifically for you know viral genomics uh and then they would tweak it and say oh it's you know but specifically for covid right so it would it would probably contain reference you know coveted reference genomes or something like that right so um and some projects you know uh became more prominent than others arctic was one project that um was putting out lots of

pipelines um for covid sequencing uh an analysis and they uh were publishing them to uh so the workflow orchestrator they were using was next flow and next flow i was a huge fan of because years ago when next flow started they they had they abducted singularity uh like a year after docker right like they supported docker in like 2013 and then singularity in like 2014 or 13 and 14 something like that really or i might have anyway a long time ago and um and they support both still very very well and so the arctic network was actually uh the labs that were publishing a lot

of those pipelines uh put all that code in containers just because it's obviously reproducibility was a big deal during a pandemic right when you've got people using pipelines created halfway around the world to do viral you know genome analysis so we can do you know kind of the genomic epidemiology so containers there you know there wasn't really any question that that you know that's what that's what people had to do and um the only code i saw that wasn't in a container was you know like from a publication you know someone who had just decided figured out a new mutation or something like that and

they had their code they figured it out but people who were spreading pipelines for other people to use uh all the code was in containers very nice dave i don't know if you have anything to add to that as well about covid um not really i think that glenn uh really really captured the essence of that uh that question and answer quite well all right so i've heard i've heard of various research organizations doing things national labs and whatnot doing research on it as well and i've also heard anecdotally that a lot of that was happening through singularity and apptaner but it's really cool to

hear glenn's specific experience in in exactly working through that i mean one of the you know one of the things too um that is so great about container technology is that it really um it's it's enabling for researchers and so um you know at the nih a lot of the researchers uh build or use containers um and do so without you know the explicit uh help of any administrators and so because of that you know i'm sure that there was a lot of enablement and that was happening kind of behind the scenes uh you know that that administrators didn't even have to deal with because

users were able to to use the containers on their own you know yeah one more thing that you kind of remind me about it too so with with with uh with viral genomics too the the uh the the genomes are so small compared to human uh or other mammalians or even lots of plants and things a lot of these things can actually be done on a laptop you know if you're just gonna do one you know one analysis maybe not my laptop because it's a couple years old but um uh so with containers you could actually you know go pull a you know viral recon

container or you know something from from arctic and and run a workflow on on your on your laptop and so that allowed you know people who didn't have large hpc clusters to actually participate in um you know kind of the global effort yeah that's fantastic we did have one question that just popped in yeah could you comment on the pros and cons of container vs packages on conda environments throw this up to whoever would like to take it anyone yeah um i mean a pro a definite pro is that you can replace the condom environment with the container um so yeah conda has become really

um it's it's become pretty unwieldy uh so it's gotten to the point um where you know so it used to be at the nih we have we there's a there's a big huge uh like conda environment that has lots of standard tools installed in it and uh just you know going through the the dependency resolution step it gets it's gotten to the point where we actually need a cluster to run the dependency resolution step because the memory is you know requirements are such that they will crash your note they're just too much so um yeah so it's become completely unwieldy and so yeah i mean

now um to be able to take those packages out of you know this big uber condo environment and just stick them in containers instead uh has been you know really kind of a huge win um for uh you know for us and hpc i think thanks yeah uh well i i second dave's comments i i felt the pain of trying to you know the yaml environment yaml files a conda and and you know build everything and you know it takes hours i'm like you know what delete and just you know just uh not try to update it right just create a whole new one um

i saw i i was huge uh huge fan of conda um and working working talking with peter back in the day uh like the continuum days and actually and thought days before continuum and it was a huge leap for python and being able to be as a python developer but then i started getting worried when it became all of a sudden like a package manager for anything like you could just you could install anything with conda now like you could stall r you could solve like uh unix utilities i'm like oh this is gonna just be like the sprawl all over everywhere and then just

kind of get out of control um and that's kind of how i felt it happened and so we with our users we've we've had this battle with yes you could have this environment yes you have a list of things to to do but like making an incremental update to that takes hours uh to do becomes less unwieldy and we were trying to promote look if you just put a single single app in a container your pipeline will work if you need a development environment we tried to limit it like you said to keep the dependencies down because like you know conda again can pull in

everything and uh and like you know let's just limit it to like you know you know numpy scipy and the python packages you need and stop pulling in r and other things that that really just make it just you know a real mess to deal with i mean one thing too i kind of feel bad because you know i just sort of trash condo and said that just replace it with containers is the way to go um one thing i will add to that is you know i mean conda is a decent package manager when it's used for what it's meant to be used for

i think that one of the problems um is that we kind of you know and i think a lot of hpc sites probably did this tried to make like a big uber you know python environment based on conda and that maybe is not really what it was used for kind of within containers is actually great uh you know there's there are there have been many many many applications that i've installed where the developers um have you know in either instructions or they have you know the files to to package all their dependencies within conda up on github and if you know if i just want

to grab that and stick that in a nice clean container environment uh with mini conda or something and then use that to install everything into the container that a lot of times works really really great and you know then within the container just you know figuring out where the con environment is and putting that on the path or even just you know sourcing uh sourcing condo and kind of activating it within the environment of the container i mean those things usually work just fine and so it's usually pretty quick to use conda to install something inside of the container but you know if you do

that then you sort of modularize it so that you don't have this big you know condo environment that's got a zillion different dependencies they're all conflicting with each other and you know how are you gonna resolve them all so i think the two working together can work together quite nicely if you do it maybe more a little bit more in the way that conduit is intended i'm excited i'm gonna look at i'm gonna respond kind of in a very more general perspective one of the really big benefits of containers i mean being able to isolate environments being able to move things around is incredibly important

but but the packaging format that we've just created we basically just took packaging up a layer instead of packaging certain certain applications or services or files within an operating system we're now actually using the package manager to help create that custom operating system then we're packaging the operating system and then moving that whole operating system around so for me containers is actually i mean again it's a cool run time there's a lot of cool things you can do with having a separate isolated environment but it's also a packaging system and um that packaging system is is incredibly flexible because it is the user space of

the operating system so it has every single thing you need and it's portable between whatever linux distributions you want as long as your architecture is binary compatible you you got an entire application set in a package in a matter of speaking um so yeah yeah i think you just pre-empted jonathan's question too i think if we can pop that one up so if you see any issues with reproducibility when running a container on a different kernel version so i mean you just talked about having a container with different packages in it what about kernels yes yes um not only kernels but hardware so the idea

of saying that a container is reproducible is not exactly true depending on what what perspective of reproducibility you're talking about if you're talking about reproducibility of outcome it's not guaranteed kernel bugs driver bugs firmware bugs firmware changes we saw something not too long ago where a gpu firmware bug i'm not going to talk about with gpu but a gpu firmware bug was actually causing wrong answers so these things do happen now when we talk about reproducibility though from a container perspective what we're talking about is reproducibility of the software stack 100 guaranteed reproducibility of that software stack of user space software stack so some attention

still has to be given to the underlying resource if you want to guarantee the entire the reproducibility of the outcome generally speaking though most of us consider reproducibility of the software stack good enough most of us aren't thinking about kernel bugs driver bugs firmware bugs hardware bugs we're just assuming that all of that's going to work properly and in the kernel specifically the kernel has an incredibly stable uh backwards compatible api and abi that it's passing to the user space portion of the operating system so generally speaking it's something we can strongly count on but again yeah things happen and you know depending on what

direction you're going usually if you're going backwards so if you've got a newer kernel and you want to run an older container you can actually run really far back like that api is so stable uh you know zane you know the story i i could bring up from uh from berkeley we you know we actually had a system that was like 17 years old and it the hard drive died we took the hard drive out we made a container out of it and that software continued to run in that 17 year old operating system so fantastic yeah but you can't go the other way if

you've got an old kernel and i mean the obvious the extreme case is very easy to all right the 17 year old kernel is not going to support namespace as much as anything else but if you even have a kernel that's you know just let's say two or three major revision major operating system revisions late um like you know uh if you wanted to run on centos 6 a new let's say rocky 8 or ubuntu or a a you know fedora uh user space it's not going to work uh glib c is actually going to check and you're going to get a glib c error

so the direction of compatibility needs to be considered in that but generally speaking knock on wood usually we just say everything just kind of works so i have a quick question so i've seen that glib sierra a lot of people have seen the glibc error when you've got you know some new uh version of the operating system you try to run it within your container and your kernel's old um so i've seen that i know in theory you could uh you know run a container on a different kernel and end up actually without like a syntax error but actually you know just a different answer

um and i know in theory also with different hardware like a gpu i've never actually seen that has anybody ever ever actually seen that i mean it's it's it's obvious if you get the g libs here it's great because it's just like okay well i can't you know it's not going to run on this on this host but um you know it's a little scary if you if you have a math problem and you come up with a slightly different answer you know but i've never actually seen i know that's possible in theory but has anybody ever actually seen that occur i've never seen it

in person i just know of people who know people that have seen it and um uh an example is again you know the gpu issues where people were getting wrong numbers uh that came from another national lab which i believe was the first ones who who saw it so i just knew people who knew people uh who have seen it but that you're exactly right that's the one that scares me right right when it errors out and you know it's just not gonna work that's actually a good thing when it when the error doesn't happen and it just silently gives you the wrong outcome that's

actually scary it's actually also very difficult to debug i've seen that happen a number of times in software uh with various math libraries and whatnot that were just for some reason just you know not given the right right answer um or optimized differently or you know loops were unrolled in a way that for optimization that that yielded a wrong uh and not completely wrong just slightly wrong error that's almost as bad as you know you get the call from a scientist who says my application's running five percent slower than it did yesterday what's going on um now debugging that one is is tricky too but

you're exactly right i much prefer a error to a wrong answer right absolutely and i think so john i don't think you can pop this one up jonathan or john made it a interesting statement it would be nice for the language on reproducibility to make the distinction between developer reproducibility and scientific reproducibility i think it's a good statement i think we all agree 100 yeah we should almost use different words right you're talking about like reproducing an experiment versus reproducing a software stack true so we're kind of getting close on time and one of the things i really wanted to be able to have dave

and glenn dive into real quick before we wrap up is the common real world use cases that you guys see in life sciences or that you have seen and been a part of kind of just tell us what you what you were a part of what you saw and it would help me a lot because i like seeing real world examples and helps me sure i can jump in um so uh at the nih um there were different domains of science that picked up containers and are picking up containers at different speeds but um there's there's kind of like and i've already differentiated that and

kind of talked a little bit about that there's you know two different kind of flavors of end users the producers and the consumers and what it all boils down to is that containers are just really enabling uh researchers to you know get their stuff done it gives them more power and it gives them more convenience and so we're seeing a lot of you know that's the standard kind of use case is that um we're seeing a lot of researchers just using containers to either have stuff pre-packaged or to be able to use um you know like uh apt or yum or whatever to install things

that they wouldn't otherwise be able to install in their hpc environment uh we're also seeing you know a lot of people are using containers because they're using you know workflow languages uh glenn already talked a little bit about next flow but also things like whittle and cromwell and things like that that are using containers that um and then you know even um there are uh well you know i won't dive into that too much but that's that's people are picking up on that more and are starting to use containers more uh for like orchestration you know within like workflows and things uh even within sciences

and so that's from the user perspective but then also from the administrative perspective uh you know so at the nih it's kind of a little bit of a unique hpc environment because people talk about the long tail of hpc and nih is like entirely a long tail right i mean it's there are a thousand applications and growing that are installed on the cluster which are maintained by staff which is just a staggering number um so this the staff members have really picked up on using containers and it's now become the default and the vast majority of new applications that are installed on the system are

just containerized and we put together um i think there's there's a few different ways to do this there's and there are different levels of automation uh but we put together a pretty you know easy way for staff to just install a container write a little wrapper script in fact in many cases just copy a boilerplate wrapper script into the same directory and then make some links to all the binaries and things that they want to expose inside the container and then after that's all done it can be put under the the module system just like any other application and the end result is that staff

very easily install these these applications in containerized ways and users come along and use them as though they're installed directly on the system and never even know what a container is and so that's a that's a really big use case that we're seeing as well that's interesting i'm just going to touch on that before um before glenn uh jumps in uh when we were first developing a singularities image format and we basically decided that we're going to make it a a loop-based file system an actual file system and squash all the contents into that file system and it was a single file i i forget

how the idea entered into our heads but at some point somebody said it's a single file can't we just execute that and we actually talked about how to actually run and make that single file file system uh executable and uh and when we first figured out how to do it we realized oh man it was actually much easier than we thought it was going to be as we're using a crude hack of the shell to do it but with that being said you can literally build a singularity or an apptainer container define a run script for that container put it in your path and then

that becomes your application the fact david you kind of just talk through that real high level but the magic that that actually creates and what it does is actually super super super cool so just wanted to throw that in there yeah i mean that's always a fun use case just because python virtual environments can get kind of ugly so being able to do that it's magic yeah i i second uh dave's experience at the nih with our with our cluster you know we were struggling with modules uh and then trying to bring containers into it and i thought it was this ugly hack but then

i thought i was kind of kind of like like Greg was saying was really brilliant it's like well why don't we just you know write you know when they load the module they just load a container instead of instead of like the binary that we used to build for them and i'm like that's really just this ugly hack why can't we i'm like oh yeah the inertia of using modules we can't get people just to change their their scripts to use containers and and so so that was that's actually how we're kind of you know sliding in these containers a lot uh onto our cluster

the the other thing that um we we thought containers were going to be really kind of required for um so there's there's folks that that that was working with it that wanted to basically be the the future of hvc was going to be cloud-native and now that didn't necessarily mean you're going to run into public cloud but you basically needed to be able to you know as one of my colleagues would say run my stuff anywhere anytime right and so the only way you can really do that with is is containers and so when uh we were experimenting with a lot of our pipelines trying

to run them in the cloud um you know we thought that containers were were the only way we were going to be able to reproduce anything but it was the only way we were going to be able to be kind of like mult because everyone wanted to be multi-cloud and cloud agnostic and just you know throwing out all the you know let's be vendor neutral terms um i stopped using the word cloud and started using the word serverless because uh because i just got tired of you know what cloud was me because it meant it could be vmware it can mean gcp it could mean

you know um lots of other things so i so i think containers are are are kind of going to be the key to cloud native computing and we've we've predicted this and we've started seeing this so then the newer uh hires at tgen the the faculty members kind of expect like so where's your aws account like how come i can't just spin up this you know my instance there yeah how do i get this and so we we kind of predicted that you know as users are using containers and being more cloud native um because they could be the long tail of hpc and not

running on big hpc clusters they just expect to be able to spin this up um that that's going to be you know that the researchers are going to eventually just start demanding it when they go to be recruited by an institute right like so what's your how am i going to be able to compute here there's a traditional hpc or am i going to everything i have is out in the cloud my petabyte of data is out there how am i going to keep doing my analysis that i got hired for i'm not downloading it here right so how do you help me you know

get my stuff out there um under whatever organization's account right um and if this organization doesn't have an account with this cloud provider how do i make it happen so i think that's something that we've just started to see the tip of the iceberg on but we we expect it to happen just because of container usage condo usage and more what i like to call just you know container or serverless kind of base computing the only thing i'd add on top of that by the way i think that was 100 accurate the only thing i'd add is i think containers are really uh the necessary

next step for any form of more modern high performance computing as we're looking forward whether you want to just run in the cloud you want to be able to move your workflows effectively and whatnot instead of rebuilding your whole software stack up in the cloud it's just so much easier and more convenient just to containerize that software stack and now be able to run it portably anywhere absolutely i think it also kind of as i'm hearing glenn talk it reminds me very much of what Fuzzball was intended for so i started thinking fuzzball immediately so thank you guys we are at the end of

our time and i really appreciate you guys joining today very informative and anybody out there if you have any questions please reach out to us give us a like and subscribe follow us every week and we are looking forward to talking to you again thanks for joining guys it's fun thanks thanks