Apptainer: Latest Enhancements, GA Release & More

And a big example of that is containers. Uh containerization didn't or didn't really have a place in HPC before singularity. Um it was kind of something that was more used in enterprise, but there wasn't really a good implementation of it in high performance computing architecture. Um and so as mentioned this need was recognized uh because containers are a fantastic piece of technology for high performance computing um because of the portability, reproducibility, um security and other convenience uh aspects of them that um you know make them very useful for a lot of different things in that sphere.

Um and so yeah uh essentially Singularity was built to um fill that container gap in HPC um and provide a meaningful way to allow containers to be built and deployed across high performance computing architecture um you know with all of those modern benefits like uh or with all those still um ability to use MPI and stuff like that. Um so in essence singularity was born out of a need to bring containers out of enterprise and into HPC as well. Um and so uh it was built it was deployed at many different sites over a few years. Uh it kind of started to gain a very large user base um across a lot of different places.

Uh there was over I believe the number out there is 25,000 installs of Singularity at different sites around the world. Um and so essentially uh Singularity continued to develop until uh about last midl last year or so. Um Singularity is a very mature technology. It's been development for a number of years. It has a number of different features that as I've discussed are very useful. Um and so there was kind of a question that was put out to the Singularity community at that point. Um they kind of dealt with uh you know what was next? Where should Singularity go?

And this question revolved around moving Singularity into the Linux foundation uh and allowing it to kind of um be in that sphere in a similar uh way that other projects like uh Kubernetes and such have ended up um you know under the purview of different uh you know organizations like the cloudnative computing federation. Um the question went out as to whether or not Apptainer wanted to be moved into the Linux foundation. Um the Apptainer community unanimously approved this.

um about 50 or so community members were sent the question you know should Apptainer join the Linux foundation um the response from the singularity community was unanimous from those 50 people um that yes this was a good idea um and so kind of inroads were started to be made there uh as a part of that process you know it was demonstrated also that there was strong support from industry users of singularity as well so places like Intel and AMD um and people also that um wrote in support of this move into the Linux Foundation. Um and so with all that uh you know set and both the community and kind of the um you know maybe more enterprise or industry side of the Singularity community in approval of it um things went forward.

Uh Singularity ended up joining the Linux Foundation. uh and because of essentially just uh you know legalities around uh surrounding trademarks um and another version of Singularity that's maintained by a third party, a commercial version of Singularity that's maintained by a third party. Um the name Singularity couldn't continue to be used. Uh and so the project had to be renamed to something. Um Apptainer was a suggestion that came up and it ended up winning out among the community. And so now we are here um with the initial release of Apptainer um separate from the uh previous um you know singularity uh development kind of its own little uh uh what's the word I'm looking for?

It's like the next step of it essentially. It's the open source singularity has been like development on it has ceased and it's now moving under the purview of Optainer and so that is how we're here with Optainer. Thanks Boris. I appreciate that. Robert, I'll throw this one over to you. I know most of this was built for HPC and I know that that's kind of where everything's been focused, but what what things are changing or what has been done now that's in the Linux Foundation to kind of drive this to the enterprise as well? Yeah.

So in a in a lot of cases what we're seeing is a need for the immutability, need for cryptographic signing of applications, the need for software supply chain security, a need for uh the running uh traditional HPC type of workloads inside of uh enterprise AI, machine learning and and other uh workflows are really driving the need for containerization of those applications. So, we're seeing a a big growth in that capability and this is a a format that folks are appreciating. And then as moving this into the Linux Foundation, has that helped uh with adoption? Is it is it helping grow capabilities in the enterprise and across HPC?

What is that what does that look like and what is that doing today? Yeah, Zayn, that that was one of the biggest reasons to move it there was to get tighter with CloudNative Compute Foundation, Open HPC, um the entire Linux Foundation's uh family of projects. Um it was an extremely important step I believe uh to expand the reach of what what it can do. Yeah.

just to jump in there um you know you look at some of the other projects that are currently under the purview of like the cloudnative computing federation or the open containers initiative um and you see things that have huge effect on um you know the enterprise and increasingly HPC um but we see that you know these organizations are already heavily involved with technologies um you know like Kubernetes containerd run C um all kinds of different um Helm also is another good example um this whole kind of cloud cloud computing as it relates to containers um you know is really organizing around these organizations uh and

as you know HPC increasingly moves into the cloud um having something like Apptainer with you know the model that makes it good for HPC things like um for example it's uh integration over isolation approach whereas you know with docker something like that you'll find that the natural kind of way the container runtime wants to run is in an isolated way so that the containers are as isolated from the host they're running on as possible. Um, Apptainer is far more focused on integration because there are oftentimes a lot of kind of specialized hardware and software pieces that are a part of a high performance computing cluster. Um, and so we definitely want those to still be able to be used within Apptainer.

Um so it has very robust support for um you know GPUs, MPI um you know these things that we need to make available inside of containers so they can be effectively used and scaled in HPC. Um it also uh you know has that single runtime that essentially builds its containers as single flat files that you can move between systems very easily. Um you can then take and cryptographically sign those containers. you can encrypt them as well, the underlying file system that they're made up from. Um, so it has so Apptainer is obviously it's something that's hugely focused on HPC. Um, and so as HPC begins to move into the cloud, um, we see more and more use of, you know, public cloud um, or even private cloud for people wanting to run HPC.

Um we definitely want to be coupling Apptainer as tightly as possible to these um organizations that are driving that innovation and driving that development of um you know as I started by mentioning here um you know those vital technologies out there like Kubernetes and that kind of thing. So it's definitely a very good thing for Apptainer being um you know essentially the premier uh container runtime in HPC um to be a part of those organizations and to kind of be able to um you know make sure that Apptainer is included in making sure things work with it and you know such like that. So yeah there's a lot of cross foundation that can go on there.

Um there's also you know as a part of the Linux foundation like open HPC um which is another big project that uh traditionally is kind of focused around um you know the open source like you know if you go and look on there they'll talk about Warewulf and rocky Linux and stuff like that at open HPC um so you know that's something we're very excited to be closer to as well um being you know Apptainer and HPC as I've said a few different times um yeah we're definitely very excited for to also move closer to you know high performance computing focused projects. projects that are part of the Linux Foundation like that.

Great. Thanks, Worst. So, Mike, I appreciate you jumping in and being the first one to post that you're listening and where you're from. Uh, Greg obviously joining too. So, we have a good question from Stephano and he's wanted to kind of compare and contrast. What are the the pros and cons of Abainer compared to Docker for an HPC workload? Um, I think it's an important question and something we we probably should distinct. Absolutely. Um, so as I've kind of touched on a little bit, but I'll go into a little bit deeper. um essentially the security model and kind of the running model that Docker uses in order to run its containers doesn't work well with a multi-tenant HPC system.

So when we think of a high per when we think of you know for example an enterprise server we're maybe thinking of something like a web server where if you don't count you know thousands of people that might be accessing your website the total number of people that are actually going to interact with that server is probably very low. It's basically just you and you know the rest of your system administrators. So, you know, half dozen, dozen people more depending upon your organization. You get what I'm saying? Um, just lost my train of thought. Oh, yeah. A multi-tenent HPC system. Sorry. Yes. So, when we're um when we're in uh an HPC system, we're not in an environment where we can count on just having a select number of very access controlled users um you know, being on that system at any one given time.

A traditional HPC system could have um you know hundreds of accounts for different users on it to be able to log into it. Um typically what we see is uh sites will set up a login node which is essentially um a node that is just for users to be able to log into things on. Um this is done for uh you know certain reasons in HPC avoiding um things being run on a head node stuff like that. Um but essentially when you have uh hundreds of people that are on this um node or that need to be able to use these resources um Docker's model of needing to be privileged in order to interact with it falls apart.

Um when you have hundreds of different people, you can't possibly give uh hundreds all those people the level of access um with regard to security that Docker requires in order to be able to run it. Um and so Docker in general just does not work very well in um you know a traditional high performance computing cluster environment. Um it also doesn't particularly scale over nodes uh as easy as Apptainer can um with its you know integration for NPI and stuff like that. Um and so those are a couple of big things. Uh as I kind of touched on a big focus of Apptainer is integration over isolation.

So whereas in Docker you would have a container that's trying to isolate from the host um and kind of present you know its own file system its own interface um with Apptainer when I go and get into one of you know an Apptainer container on a um compute node or HBC architecture I'm going to be my same user as I was outside of that container as I am inside the container. And so you know I mean you can immediately see how that'd be useful. I can spin up custom environments that have you know like we've touched on AI or ML tooling in them. um and then be able to essentially apply that environment to what I'm doing on that HPC cluster.

Um maybe if I'm developing on a node or something like that, I can interact with the files that I own on that cluster directly from inside of an AppTainer. Um the security model makes it very easy to integrate and do HPC work. Um, another big thing about Apptainer that I want to make sure I mention is it can lead to much greater efficiency when you're deploying high performance computing workloads because of certain aspects of how it does file uh input and output. Um, if you can imagine trying to run a MPI job across you know a thousand nodes at once. Maybe we're running molecular dynamics simulation or something like that.

Um, or you know maybe something that's using Python doing a lot of small file IO imports um that kind of stuff. Um, if you were just doing this on a normal file system, the operating system would be having to, you know, seek and, you know, look through that file system, traverse directories and stuff to find those files. if you're using an Apptainer uh instead the internal file system that all of the different components that are represented in that container um you know like I said the AI or ML tooling um you know anything system stuff um that's all represented inside of a squash uh a

compressed squash fs file essentially um and so when you're writing or sorry not when you're writing I'll get to writing in a second when you're reading from this squash fs and doing this um you know small file import stuff like that the speed of that will be much faster than if you were just um pulling it from a standard directory because instead of doing directory traversals and stuff like that, um the squash fs uh you know compressed file system underneath it essentially allows you to just read offsets into that squash fs. Um so it becomes much faster. The target of your IO becomes a single file instead of the entire directory structure.

Um and you can get that same thing if you're writing a lot of data by adding an overlay to a container. Um an overlay in Apptainer is an X3 uh image basically um you can add onto a container to add a writable layer to it. And if you target that writable layer with um you know outputs of small file IO um because you're once again basically just writing offset writing data at offsets in an image file. Um it has that same type of speed than if you were trying to traverse all over a directory and write stuff. Um so Apptainer has a lot of different benefits over docker in an HBC environment.

Um docker is really meant for running services um doing that kind of thing. Um you know this is kind of where the concept of like microservices one container for one small task came out of um um essentially the difference boils down to the fact that uh you know Apptainer has a fundamentally different model than docker with how it works. Um like I also kind of touched on uh like I also kind of touched on um it also features essentially a single runtime whereas in Docker you're going to be and this touches on kind of the security model of it as I discussed needing um people interacting with Docker to be privileged.

Um Docker's Damon based system um you know requires you to manage you know starting and stopping that with system ctl or whatever that whole kind of thing. Um in Apptainer the container runtime is essentially a single um you know executable platform that you just feed in these um container images that Apptainer produces called um CIFS singularity image formats. Um it essentially Yeah. Yeah. That's great for us. I appreciate it. Robert, I know one of the other things Yeah. No, you're good. Go ahead. Um sorry, I did want to finish my thought there. Um if I can remember Oh, yeah. the run times. Sorry. Um, there was a knock at my door a couple minutes ago and I'm not sure what it was and it kind of threw me off a little bit.

Um, essentially with how the runtime goes, um, like I said in Docker, it's a dam based system. You have to manage it with system ctl. In Apptainer, it's essentially a single um, it's essentially a single execution runtime that operates over these single flat files. In Docker, you're going to be managing images that are all managed through like, you know, you can imagine something like Docker image ls and you'll be able to see the Docker images that you've built on your system. Um, you know, it's something that's inherently tied to Docker. Um, with Apptainer and these CIFs, they can essentially just be treated like any other executable file.

Um, especially if you use some of the other features that Apptainer has that literally allow you to use them like an executable and specify what they do when you run them that way. Um you can essentially treat Apptainers as an executable and they can be moved you know between systems um or between you know for example research groups or whatever uh with that kind of with the ease of basically just moving a file around. And then because of the security features that we've discussed um you know you can also cryptographically sign and encrypt those containers in transit so that you can have uh you know guarantees that uh you know the container is what it is and it hasn't been tampered with at all.

So so essentially a lot of differences Apptainer is very different than docker. So, thank you for So, on the security side, Robert, I know you like to talk about this quite a bit is even though I mean it's great that you can encrypt the file. So, in transit you'll have it, but you've also like to talk about the fact that you can actually execute that thing and it stays encrypted. So, at runtime you can verify it before you execute it. You can execute it, run it encrypted is am I am I off? No. And you know with you know the software supply chain uh being such a crucial aspect to everything now you know having multiple signatures to identify uh that container similar to what Forest was just describing as far as sharing but also now as far as placing that onto resources.

Right? So when you go place those onto an endpoint or a resource and you can then run it in an encrypted state, it never uh decrypts basically out of memory. It gives you the capacity to uh deliver um a solution that uh can fulfill an entire life cycle of say an AI workflow um all the way from training to inferencing to delivering those algorithms to an edge device to now perform. So it just gives people a way to systematically maintain that software supply chain throughout their entire uh solution. So and so you can imagine you know building on that how you know Apptainer could be integrated into you know for example your CI/CD practices um to do automatic verification and encryption and security and stuff like that.

Um there's a lot of different ways that you can look at those security abilities um to as Robert talks about um use those features to ensure Apptainer security um you know across the software supply chain. So very useful features there that um are useful in both more human focused um but also more kind of automated settings as well. Thanks for so Misha has a question and it's it's talking about vendor specific uh container hubs. So like Nvidia has theirs, AMD has theirs, and the Docker obviously has theirs. Is there any plan to get them to use Apptainer instead of Docker? I I mean I I think I know where the answer is going to go, but I I'll let you answer that for us.

Um I I doubt I I can't really speak to any plans. Um but I can also tell you that that's not particularly necessary. Um one big thing that Apptainer is also built with is Open Containers Initiative or OCI um compatibility. So the standards that they're putting out for things like Docker, Apptainer is fully um yeah, Apptainer is fully compatible with um whereas in Docker, just to kind of explain uh you know, how exactly that happens. Um you know, when you're pulling a container off of the NGC or something like that, we might do a Docker pull and then provide the uh you know, pull tag. Um with Apptainer, it's the same thing.

apptainer pull and then provide you know the pull tag and the you know the URL to the container. Um whereas in docker there's a concept of layers. You know for example if you're building a docker file every run copy or other command I can't remember right off the top of my head uh introduces a new layer into the container. Um and that you know has other things that for example increases like your um attack surface with the number of containers or number of uh you know layers you have in an image. Um so that's the way those work. Um but if we're using Apptainer and we do that apptainer pull, there's no concept of layers in Apptainer.

So essentially all of those layers will be compressed down into that single squash f or compressed squash fs that makes up the core file system of a um of an Apptainer container aka the se. Um so in essence uh there isn't a particular need for them to specifically build it with Apptainer. um that'd be great especially if they're using um uh especially if this is you know HPC focused technology and I know that for example if you go to the NGC and look around I believe that they give you instructions on how to use um both docker and they'll call it singularity but um Apptainer

to deploy those containers as well I've seen I believe I've seen like tutorials over there um it's just included in their instructions um so there is already a kind of a concept of support for some of them out Um but essentially the most important part of this is uh Apptainer has full support for OCI images and pulling those from a container registry is essentially just an operation of taking all the layers that make up that Docker container, compressing them down that squash FS and then delivering you the um Apptainer. So it works just fine with Docker and uh containers built and deployed on those registries. Excellent.

That's where I figured this was going to go. I just wanted to to let you guys talk about it. So I mean again Misha great question we appreciate it. Um absolutely proliferation of docker there's it was built that way for a reason with singularity in Apptainer to make sure that you could just pull from those so I know that Apptainer 10 I believe one has been released now right and I I think 1.0.1 0.1 was also released yesterday as well, the next one after. Excellent. Can you can you tell us some of the features that are in that? Yeah. Um I have a demo here in a little bit as well of one of those um kind of the one of the um bigger ones that came out of it.

Um so just to kind of go through a little bit of our change log here. Um what I'll demo here in a bit is instance checkpointing using uh DMTCP. That's distributed multi-threaded checkpointing. um that is kind of an HPC focused technology that allows you to checkpoint the state of an application while it's running. And you know why would you want to do this? If you're running something in HPC um it's entirely possible that this one single um you know run of an execute uh executable could last days or even potentially weeks. And so if we're sitting there and we have a simulation that's running for a very very long period of time, it would not be good to have that, you know, something go wrong with that.

You know, I don't know, there's an error in the um code that becomes apparent after a long period of time. Um, you know, cosmic ray hits a bit and flips it and causes the whole thing to crash. Um there's, you know, then suddenly you've just lost those days or weeks of computational time that you've been waiting on. Um, so DMTCP allows you to do checkpointing of applications and essentially this allows you to save off the state of an application while it's running and then be able to restore the application to that state at a later period. Um, and so this allow so we've implemented this on container instances in Apptainer.

um a container instance in Apptainer is to a um it's what you would let me get the uh equivalency here. A container in Apptainer is essentially equivalent to what would be called an image in Docker. While an instance in Apptainer, a container instance in Apptainer is essentially equivalent to what would be called a Docker container like a um this is the concept of you know the oneoff immutable base copy of an image versus you know um one off copies of that image made to do something specific.

like you have a you know specific service container and uh you know you're spinning up copies of that to run a service um you can now use DMTCP to checkpoint those container instances and their state in Apptainer so you can start up an instance um uh essentially attach a checkpoint to it um checkpoint that application as it's running in that instance and then be able to restart that instance at any point from that um checkpoint and have the state of the uh application inside of it be the So that's very useful. That's a great um HPC focused uh kind of part of it that'll be very useful for checkpointing um workloads running out of um Apptainer containers on high performance computing architecture.

Um just to run through a few other things here um that are a little bit smaller but are still you know kind of interesting. Um there's an option to set up a writable temps when you're working with a container and this will essentially the container is immutable. It's a readonly file system. So while you can interact with your host file system inside of it because of how Apptainer's security model works um you cannot actually edit the container itself. The container is immutable. This is you know the basic concept of a container. Um you can use a writable temps in Apptainer that essentially gives you a temporary file system in RAM um to be able to write things to and that's useful for certain use cases.

Uh nowadays, one of the new features is that when you're doing a build of an Apptainer container, you can take the test section, which is a section that allows you to specify commands that'll be run after the container is built to test it for functionality. Um so this could be, you know, checking for the, you know, version string of different applications or even something more complex. um you can now use writable temp fs with this build option to generate a writable space in RAM for the actual build process to use with that test um functionality so that you can use a writable um temporary file system with your tests as well on building a container or once a container build is done.

So that's very useful for some use cases. Yeah. Um there's kind of a shorthand flag that implies a few different options that people typically want to use with OCI or Docker containers. Um so that is kind of just an interesting shorthand there that makes that easier. Um there was some improvement around the way that GPU libraries are brought into a container. Um, as I've mentioned, one of the big things about Apptainer is that it wants is that it um has support for GPUs. Um, and being able to utilize a uh being able to utilize uh you know, give essentially what's inside the container a GPU to work with.

Um, normally the libraries that this needs to have bound into the container from the host. um in order to make this work are discovered via a configuration file that comes with Apptainer um or they're discovered via one of uh Nvidia's command line tools I believe called Nvidia Docker. Um nowadays there's some new tooling that's come out of Nvidia um kind of surrounding managing uh you know Nvidia based containers and uh you know essentially GPU based containers. Um, so there's some new support in Apptainer to be able to use their NVIDIA container CLI instead of this um, pre-made list or this Nvidia Docker tool. Um, if you specify that you want to use that container CLI tool.

Um, so that's kind of a new way to work with uh, GPUs and um, you know, with making them or having support for them in containers as far as you know their libraries and that kind of thing. So that's very useful uh for you know deploying AI, ML, anything that uses a GPU um in a high performance computing environment. Um there's some differences surrounding how croups are done. Um it now supports the second version of croups and that kind of hierarchical um setup as opposed to just the first version of croups. Um so that was interesting and um yeah in general there are a lot of other kind of smaller changes that came out with it.

Those are some of the bigger uh new features that have come out with it. Um there were some changes to how for example some of the internal security options that Apptainer provides work. Um, so for example, you can do things like um set flags on whether or not you want um you know unencrypted CIFs to even be able to be run within the App Chainer runtime. Um and so yeah, there's like I said there's also other things that gave better support for Docker and OCI images. Um then yeah in general lots of um kind of internal changes that work on um kind of how libraries are found across you know not just GPUs as well but other aspects of um binding those into the container.

Um so in general a lot of different changes um a lot of these changes that came out in the initial release um like I said we had the initial release a couple of weeks ago um we just had the next release yesterday. Um, so a lot of the changes that were in the initial release were focused on providing backwards compatibility for Singularity installs, um, which if you're curious about, maybe you haven't made the transition yet to Apptainer, uh, we do have a previous webinar that went over that, uh, migration and how that works and, uh, what that kind of looks like. Um, so feel free to refer to that if you're looking for kind of a tutorial on what that migration looks like.

Um but in general the first release was focused around um kind of the different uh you know things that need to be done in order to provide that backwards compatibility. Um improving support with Docker and OCI images, improving the ways that libraries are found and presented to the container. Um and like I said added a few new features like most significantly the DMTCP checkpointing and I believe that's what you're going to show us, right? Absolutely. Let's see it. So, um, if you give me just a moment to pull up my VM here and share my screen. All right, just up that font a little bit more.

Okay, there we go. Should be able to see that fine. Nice. All right, cool. So, I'm here on a Rocky 8.5 box. Um, I've got Apptainer installed and I've also got DMTCP installed. Um, we'll make sure to provide a link to the pull request that kind of show uh on Apptainer's repo that kind of shows the um basic tutorial I'm about to go through here of how DMTCP works. Um, it kind of has some of the information about how you should install DMTCP. Um there's you basically want to you know install it from source but there's also an option that you want to make sure you use that um uh I believe enables it uh to use static C libraries or something along those lines.

I would have to um go and look at the thing again. But anyway, we'll make sure that that link is out there that tells you how you should build DMTCP and like the options that you should make sure it's built with in order to provide um support in this manner and stuff. Um and then I basically just installed Apptainer. My apologies. Um, I basically just installed Apptainer as uh an RPM from the repo there. Uh, so we'll go ahead and get started on this uh demo here. I've got my commands here just to make sure that I don't lose them. Um, so the first thing that we'll do is apptainer checkpoint create and we'll provide a name of that checkpoint.

So we'll just call this example checkpoint checkpoint. Go ahead and create that. Um this creates a file in uh container checkpoint. You can see it kind of goes down this. There's nothing in there right now because we haven't um actually done anything with this checkpoint yet. But you can see that's where um that data is kind of put at. So, we haven't done anything with this checkpoint, but let's go ahead and do something. Um, I have a simple HTTP server uh container recipe that I have built here um that we're going to use with this demo. Uh you can see that we uh this is pretty simple.

Um you can also get this from the um pull request that we'll link. Uh if you have any problems with which and for whatever reason DMTCP not liking the function that um which occasionally is implemented as um this is a workaround. Uh basically just setting um an alias there inside of this specific folder. Um so little bit of an odd deal there, but um if you run into any problems with that, that is a workaround. Um we'll go ahead and do now uh with the uh CIF that I've built from this definition. We'll go ahead and start a container instance of it. Um as I mentioned, we have the base CIF that is our container in this case or that uh yeah is essentially our container in this case.

And then we're going to start an instance of that container that's essentially a one-off copy of that. Um that is like I said comparable to a Docker container or yeah, a Docker container running a service. um based on a Docker image. So I'll go ahead and do let's see apptainer instance start and I'll provide the DMTCP launch flag to let this know that we want DMTCP to be um set up inside of this container to be used as well. Um we'll provide the checkpoint that we want to link to it which is the one that we just added or the one we just created. Example check point.

Um and then we'll go ahead and provide the path to the container SIF that we want to make this um container from or we want to make this instance from. So we do server.sif because it's just here in the same home directory that I'm in. Uh and then we'll go ahead and provide the name that we want this instance to have. And then um you'll notice down here we have a start script. Um this is what this container is going to run when an instance of it is started. Um, so in this case, you can see that we have Python 3 and we're just going to run this code right here basically.

And then we're also going to pick up an option from the start of the instance. So in this case, that option will come from right here. So we'll provide the port that we want this to connect to um here on our local host there on the command line. See that we have the instance started successfully. Um, we should at this point be able to curl localhost 8888 and get a response out of it. um pretty simple one but still a response that so you can see we get a zero printed out of this. Um this like I said is a very basic example um but you can see how if we can checkpoint applications like this it would be very useful across HPC.

Um so this is currently serving us at zero when we um do that curl command. Um we'll send a post so that we can sorry I've got two monitors here. um we'll send a post so that we can change that to just a one. And then we'll go ahead and curl that again. And you'll see that that's now changed. We're getting a one out of it. Um at this point, we've changed something about the way that this instance is running. So we can actually checkpoint it. And so we'll do um apptainer checkpoint instance server. And so um as I mentioned uh the name of this instance is server.

So we're just referring to um that instance here and telling it to checkpoint it with the checkpoint that we set up which is example checkpoint. So we'll go ahead and do apptainer checkpoint instance server. You can see we get info telling us what checkpoint we're using. Um and then we'll go ahead and stop the server. So normally um this would essentially wipe away the state of the server. um this where's the name there it is. Normally this would essentially wipe away the state of the server and if we restart this off that same image um per the concept uh it'll basically create a new co new and unedited copy of that base um CIF container.

Um but because uh we're using DMTCP checkpointing, we'll be able to restart this container instance here in um a second once uh I put the command in and everything. Um here in just a second, we'll be able to restart that instance based on that checkpoint. We'll be able to see that immediately once that data is loaded. um we're able to actually see that one being printed out again as opposed to uh the zero that it was before even though we're creating this container uh instance from the same base container that we were using before. So we would expect it to be zero um if we weren't using checkpointing.

So I'll go ahead and um put the next command that we need in here. we do apptainer instance start dmtcp restart which in this case tells it that we're looking to restart a previous checkpoint uh with this instance that we're spinning up um we'll then go ahead and provide the same name of the checkpoint that we used before and actually really quickly I'll show you you can see now that there's data inside of that um apptainer checkpoint DMTCP example checkpoint um so that's how you can kind of have some visibility into um what the checkpoint looks Like for example, you can see the uh the restart script there and stuff.

Um you can see like I said, for example, the restart script that's about to um restart this uh checkpointed instance basically. Um so go ahead and provide I'll go ahead and copy this. So go ahead and provide apptainer instance start DMTCP restart example checkpoint and then we'll call this restarted. Well, actually, hold on. Do server. And then we'll do um we'll just name this something else. We'll call it restarted server. And then we provide the same port. And when we do this, yes, that's correct. When we do this, this instance will restart, you'll see that instance starts successfully. Uh and then here in just a second, once DMTCP loads all that memory and stuff, that application, um we should be able to curl this again.

and get the um one that we previously saved out of this before. So we'll go ahead and do this and you can see we get that one out of there. Um and we've essentially checkpointed this instance. Um so yeah uh that is the DMTCP checkpointing. Um just to once again reiterate what I did there. Um I created a Apptainer container instance of this server. uh contain Apptainer that I have here. Um I changed some data in it. I checkpointed that with that Apptainer um what is it? apptainer checkpoint instance server um I checkpointed it with that. I shut that server down.

So when we restart a if we were to um restart another another server off of that um without doing this checkpointing we would get uh it would curl zero in that case because we're making copy of what's already there um without the modifications that we used curl to post to it. Um, and so once uh those changes were made and it was checkpointed, we were able to restart that instance then um and have its state essentially or the state of the application inside of it essentially be saved and be able to be redeployed with that instance. Um so you can see how that would be useful for a wide variety of different use cases um especially things that are long running and are using this instance framework.

Um so yeah that is DMTCP checkpointing in Apptainer. Thanks Forest that's great it's very interesting and cool to see actually in progress. Very neat. So as we come to the end of this Robert what is CIQ's role in Apptainer? Yeah, we're uh definitely stand behind it as uh as one of the members of the Linux Foundation uh and have multiple uh folks that actually uh contribute to the uh solution um as others and we do have a full commercial offering around helping people utilize it and get the most out of the solution. Um, so, uh, yeah, I mean, CIQ is definitely standing behind it in in all ways.

Um, additionally, I what I I'd also say is we're actively growing our team extremely fast. Um, hiring on folks that can, you know, take solutions like this and help customers execute. Um, so this is something that, uh, CIQ is committed to doing. Excellent. I know Robert if you want to touch on kind of the support model that we have and the way that we approach that. Yeah, definitely. So, you know, you know, our core ethos is to support people and empower people. Um, so our our model is based on that. So, it's based on the the number of people that need support and help uh in your organization.

uh we do have other ways to uh structure that that um might be more beneficial for you. Um but that is the best way we've seen yet is to really work with the individuals that are using it um in order to make them successful which then makes their organization successful. We found that to be the most direct way to do it. And you know we're in this to help people and empower people. So at the end of the day, we we thought our our support and structure and our services model should represent that. Yeah, it's definitely something that I haven't seen in the industry before. It's very interesting and it seems to be getting a lot of a lot of interest instead of having to count everything and keep track of everything.

So it's a great model. Thanks, Robert. So Patrick, I know Patrick Roberts posted in there that he had done some that they do smoke testing, EDA smoke testing after a build. So appreciate that comment in there, Patrick. Um, so we are at the end of time, guys. If you don't have any more questions, like Robert said, we're growing very fast. There's a lot of openings out there on our website. Go check them out. I think they just got posted to links uh for the jobs that are out there. Looking forward to hearing from you guys. Again, don't forget, like, subscribe so that you can stay in touch with us.

We keep track of you. I appreciate the time today, guys. We'll hang out for another minute. See if we have any other questions come in. Maybe. I like the new background forest. Thank you. Very nice. Robert got a haircut. All right. Well, we appreciate the time today, folks. Uh, join us again probably next week for the next topic. Thank you very much. Talk later, guys. Thank you.