Container Education Series: Cloud Day with Apptainer

This webinar will demonstrate Apptainer with commonly used cloud containers and artifact registries. Many major clouds have support for storing Apptainer in their registry solutions through OCI Artifacts and OCI Registries As Storage (ORAS), allowing you to easily use Apptainer to interact with your SIFs in the cloud. Join us to learn more!

Webinar Synopsis:

• Forrest Burt's Background in HPC

• Dave Godlove's Experience with CIQ and Apptainer

• Brian Phan's Workflow Experience in HPC

• Rose Stein's Sales Work With CIQ

• Why is Forrest Burt so Excited By Containerization?

• Why is Cloud Apptainer so Important?

• How to Get Involved With Apptainer

• Containers are a Great Way to Learn Linux

• What Does Open Source Mean for Apptainer?

• GCP Container Registry Interface Live Demo

• The History of Docker Support in Apptainer

• Storing SIF Files in the Cloud

• How to Obtain Authorization Tokens in GCP

• Does Apptainer Have Any Container File Size Limits?

• Amazon AWS Apptainer Live Demo

• What Happens if You Don't Clear Your Cache Before Pulling From a Different Registry?

• Does Every Registry Follow the OCI Standard?

• What is the Origin Story of ORAS?

• Is It Possible to Automate Pushing and Pulling From Cache?

• What is Brian Phan Working on at CIQ Right Now?

• Generating Access Tokens for Apptainer With GitLab

• Why Does Pushing SIF Files To Registries Matter?

• Using Mirrors to Verify Authentication Security

• Using ORAS to Bypass Mirrors For Software Supply Chain Security

• How to Perform Fingerprint Verification With Docker Hub

• How New is Uploading SIF Files to the Cloud?

• Does Apptainer Work With Oracle Cloud?

• What is the Most Exciting Thing About

Speakers:

• Zane Hamilton, Senior Vice President - Sales, CIQ

  • Linkedin

• Rose Stein, Solutions Engineer, CIQ

  • Linkedin

• Forrest Burt, High Performance Systems Computing Engineer, CIQ

  • Linkedin

• Brian Phan, Solutions Architect, CIQ

  • Linkedin

• Dave Godlove, Solutions Architect, CIQ

  • Linkedin

---

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Hello, Rose. How are you?

Rose Stein:

I'm good. I was like, do I have time to get my Rocky earrings on?

Zane Hamilton:

Just enough time. Just enough time.

Rose Stein:

Just one. So, I'll just point this way the whole time.

Zane Hamilton:

There you go. We'll take it. Welcome everyone to another CIQ webinar. We appreciate you joining. This week we're talking about cloud, right Rose.

Rose Stein:

We are talking about cloud, specifically around Apptainer. So this is very exciting, and I think we're going to have to do a little explanation of what exactly this means. But, I'm very excited. I think Forrest is going to join us. There he is. And Brain and Dave!

Zane Hamilton:

The whole crew. Perfect. Well, welcome guys. I'm going to go around and do introductions real quick. I think you've all been on here enough that people probably know who you are, but for those who are new, we appreciate you joining. Let's introduce ourselves. Forrest.

Forrest Burt:

Whoops. There we go. Hit the mute button.

Zane Hamilton:

Wrong button.

Forrest Burt's Background in HPC [6:03]

Forrest Burt:

Good morning everyone. My name is Forrest Burt. I'm an HPC systems engineer here at CIQ. I've been with CIQ for about two years. I come out of the academic HPC sphere where I worked with academic HPC and the National Labs-sphere of that type of thing. So excited to be here on the webinar today and excited to be talking about Apptainer.

Zane Hamilton:

Great. Thank you Forrest. Dave, welcome back. Good to see you

Dave Godlove's Experience with CIQ and Apptainer [6:26]

Dave Godlove:

Hey, good to see you too, Zane. Good to see you, Rose, everybody. I did not have time to put my Rocky earrings in. I apologize. I'm Dave Godlove. I've got a long history as a research scientist and used to work at Biowulf at the NIH. I've been around the Apptainer community for quite some time. I do participate in a lot of these webinars. And I'm a solutions architect at CIQ.

Zane Hamilton:

Thank you, Dave. Brian.

Brian Phan's Workflow Experience in HPC [7:03]

Brian Phan:

Hey everyone. Brian Phan here. I'm a solutions architect here at CIQ. My background is in HPC administration and architecture, and I have workflow experience across various verticals of science, from automotive to aerospace to genomics. Happy to be back on this webinar and excited to see the demos that Dave and Forrest have prepared.

Zane Hamilton:

Thank you Brian. I see Brian, you didn't get your Rocky earrings on either, so I guess we just missed out.

Rose Stein:

They're in the mail, guys. They're in the mail. Alright. We'll get you a little clip on ones. They'll be cute.

Zane Hamilton:

So, Rose, it's been a while since you've introduced yourself. Why don't you introduce yourself as well?

Rose Stein's Sales Work With CIQ [7:48]

Rose Stein:

Hey, everybody. My name is Rose Stein. I work at CIQ. Zane is actually my boss. He does manage people. And so, we help in making sure that people who are wanting our services are able to get the services. So, we're like the in between, right? They can call it sales. Some people really jive with that word, and other people don't really, but someone's gotta do it, right? Somebody has to take the customer and bring them into the company and make sure they're getting all the incredible work that Forrest and Brian and Dave are doing. So, we're in between people there, and it's very exciting. The other day we had a customer well, a potential customer at first, now they are a customer. And we were having a conversation with them about what it is that we provide and what it is that we do. And the guy was just like silent for a couple minutes and then he said, you just solved all my problems.

Zane Hamilton:

Those are good conversations.

Rose Stein:

It was such a good conversation. So this is actually one of those things when, Forrest, I hope that you are willing, I am putting you on the spot. I didn't tell you this ahead of time, but I would love for you just to share your excitement of a story that you shared with me around, I think it was 2019, you said it was when you first discovered containers and what you were thinking about them. Oh, I think you're on mute. Can't hear you.

Zane Hamilton:

Can't hear you. Oh, you broke his mic.

Rose Stein:

Dang.

Zane Hamilton:

Now you're on mute.

Forrest Burt:

How about now?

Rose Stein:

There we are.

Why is Forrest Burt so Excited By Containerization? [9:40]

Forrest Burt:

Okay, perfect. That was odd. Rose, if I'm thinking of the same story. The idea of containerization and using that as like a system. This is obviously an idea that has been massively expanded on and architected and was being thought of in places and other places at the same time. But I remember shortly after getting into containers one of the biggest things I started to think about with them was the ability to do something similar to what we've expanded on at CIQ with like the Fuzzball side of things. But just like in my very new to HPC, very new to Linux type way, I just remember getting so excited about what people could do with containers and how it improved module file systems and improved what I was doing with building software.

My users were able to utilize containers. Like I said, instead of module files, I was able to build and deploy software a lot easier. I was suddenly just like, wow, Singularity, which is what it was at the time. But Apptainer, this is incredible, to be able to actually work with the system at this level. And then to be able to do away with some of these outdated paradigms that were very annoying not to deal with as a SysAdmin. So, I just remember sketching out at the time this little, CM shared containers of a different type of architecture which was, like I said, pretty crude for the time. But, it's very cool to have seen at CIQ, especially with the release of Fuzzball, those types of ideas that other people were thinking of at the same time. And the containers are the way that whole sphere is going. So, I've been very pleased to see, like I said, other people were thinking of that and that we've been able to expand on those types of concepts so far at CIQ with Fuzzball Orchestrate and our different container based paradigms like that. With all the work that we do around Apptainer, with what we've done with Warewulf. It's very cool just to continue to see containers, especially the Apptainer sphere playing such a huge role in the future of HPC.

Zane Hamilton:

Thanks for the story Forrest. So, I feel like we bring this up every time, but it's interesting to me and I want to make sure that everybody that is watching this, especially for the first time that's new to Apptainer and Singularity. I'm going to ask you this question, Dave, can you point out, when we talk about containers, a lot of people think of Docker. Can you just high level, real quick, tell us the differences between Apptainer and Docker and why?

What is the Difference Between Apptainer and Docker? [12:18]

Dave Godlove:

I'll try to be quick about it, like you said. So the history, and I talk about this a lot, the history of Apptainer really is that people started to come to HPC, Sys Admins and staff scientists like me and like Greg and like other folks around 2015 or so, and started saying, "Hey, I need you to install Docker on our HPC systems." And so we found that for security purposes at the time, Docker was not up to snuff to be installed on an HPC system. It just doesn't have a security model that works well with a multi-tenant system, basically. So, if you have multiple users, and some of them don't have root access, then it doesn't really work to install Dockers. So Greg thought, well I would like to design a container platform that works with multi-tenant systems.

And so while I'm at it, why don't I just go ahead and make it as HPC friendly as possible? Cause I'm targeting HPC. So that's one of the big differences right off the bat, is that the security model's different, the architecture is different at a really fundamental level, and it was targeted specifically to HPC users. So, it's got a lot of HPC goodness associated with it. And then the other big difference, which is something that we're going to be talking about, I think a lot today, is the format, the container format. So, it is my understanding, and I've repeated this, whether it's correct or not several times, but I think that is one of the big reasons. So of course Apptainer used to be called Singularity, and one of the big reasons that Greg named it Singularity was because of the single file format the containers come in.

So instead of having a whole bunch of tarballs that come in and have to be stitched together as layers into a single coherent file system at runtime, you've got a single image file that's wrapped up in something that we called a SIF, a Singularity image format file. And that is how your container comes in and you use it and you can move it around and do things like that. So that's a really, really big difference. And it's something we'll be talking about today, I think as we talk about cloud support.

Why is Cloud Apptainer so Important? [14:43]

Zane Hamilton:

So I know the topic today is cloud, and we're talking about Singularity. How does that play with the cloud? Why is that important?

Forrest Burt:

So when it comes to Apptainer in the cloud, being able to utilize the common registry solutions and stuff like that are provided by the different clouds out there gives you the ability to, if you've already got a bunch of architecture, we found in our investigation for this webinar that container works on most of the major clouds. And so being able to utilize these cloud container registries alongside your existing architecture or to supplement if you're not up in the cloud a ton, to be able to bring container registries to your lab or whatever, if you haven't already had them. Being able to utilize all of these cloud registries for Apptainer just makes it really easy to push containers, pull containers in the same way that you would imagine with the Docker based sphere of things. And, like I said, it really gives you a lot of ability to move around between different clouds, between different architectures and redistribute those Apptainers that you might have out there very easily.

Rose Stein:

So I think that we're just about ready to see it, right. Talking about it is nice and beautiful and wonderful, but we want to see it. But just before that, I know that people get really excited about containers, just like you did back in 2019 Forrest when you first found containers, you were like, whoa, cool! The story that Forrest told me, he was like, "...and I got my whiteboard and I was there!" I imagined him with two, like a pen in each hand, doing things like this and just being amazing, right? And figuring out all the things that you can do with containers. And some people might not already be in the mix. So, is the Slack channel, the Apptainer Slack channel, the best way to begin to get involved and jump in and ask questions and things like that?

How to Get Involved With Apptainer [16:50]

Forrest Burt:

That's a great place to hop in. Okay. If anyone else has any other suggestions, definitely feel free to throw them out there. I think the Apptainer website Apptainer.org has a listing of all of the different community spaces and things like that are associated with Apptainer. But, at the moment, a lot of the activity is centered around Slack. So feel free to join us.

Rose Stein:

Awesome.

Zane Hamilton:

I will say it's very well documented and there are a lot of people that contributed to that documentation, and we really appreciate it. But if you look at it pretty close, Dave's written a lot of it.

Rose Stein:

Dave's the man, go find him in there.

Containers are a Great Way to Learn Linux [17:30]

Dave Godlove:

Well, I was just going to jump in too and say that another thing that I wanted to highlight for people who are new to containers, this came up. So, a couple of us were just at the Rocky Mountain Advanced Computing Consortium or RMAC meeting, and this came up several times when we were talking to people. So, if you're new to containers, if you're not really, maybe you're newish to Linux, or maybe you're not very newish, but you've not used it extensively beyond just SSH and SCP and moving files around, stuff like that. If you haven't used Linux extensively, you might be a little intimidated. Do you get into containers because of that? And I want to say that you should really feel the total opposite way.

Containers are a great way to learn Linux. And this is one of the cool things about containers that new folks who are just new to the area find, is that a lot of times when you're in a Linux system and you're messing around, there's like stuff that you're scared to mess with or touch because you become root and then you start doing stuff and maybe I'm going to mess something up. But if you're in a container, delete files, mess stuff up, try to install things in weird ways, do experiments and break your containers. And this is a great way to just learn Linux and to get your feet under you and start to get a feel for how things work and what works and what doesn't and so on. So I think that... Don't be intimidated if you're new to Linux, because that's actually a great time to start learning about containers.

What Does Open Source Mean for Apptainer? [19:13]

Rose Stein:

I love that. Thanks for sharing that. And just to be clear too, Apptainer is its own thing and it's totally open source. So Dave, obviously you work at CIQ, but you also spend your some off time helping and supporting the open source community, which is Apptainer. So when he says, go and grab it and play with it, go do that, right? It's all open source, it's available for you to do. And CIQ as a company, we provide an opportunity for people to have professional support on top of just grabbing the open source files and playing with them. If they want something actually built in a specific way or any of that stuff. So, sometimes people ask about the clarity of what that means. So thank you for that. And I think we're ready. And who's first?

Forrest Burt:

All right, everyone. So, like I said, this is Cloud Day with Apptainer. We are just going to be going over how to use Apptainer with the various different cloud container registries and stuff like that are out there. Just to situate everyone a little bit more, a registry in this case is a place where you can put a container and store it up in the cloud and then be able to pull that back out with your credentials for that registry. Registry is essentially the de facto term for container storage. They are more or less synonymous in the cloud. You might see that we also talk about artifact registries, which are sometimes a little bit different. An artifact registry is sometimes meant to hold other artifacts than just containers like Helton charts, things like that.

So, I'll make the distinction when it's there, but just so everyone knows, a registry is more or less just like a container storage type thing. So most major clouds out there in different places offer some way to do a container registry. They offer some type of solution around that. What we're going to be going through today, like I said, is showing how this interfaces with a few of the major clouds. So we're going to go over GCP, AWS, Azure, GitLab. Dave is doing DockerHub to show how SIFs are supported there these days. And then we're going to do OCI, so the Oracle Cloud Infrastructure. And then at the very end we'll do a very quick GCP demo on some resources out there on our GCP CIQ optimized Rocky Linux that we have up there. My apologies.

Zane Hamilton:

I thought you'd refer to the Dave Cloud for a minute. You threw Dave out there.

Forrest Burt:

Did I? The Dave Cloud?

Zane Hamilton:

No, just kidding.

Forrest Burt:

My apologies if I did. There's a lot of clouds for a moment here. There's a cloud storm going on at my window. That's fantastic. So, let me share what I've got here. Like I said, we're going to go over GCP, AWS, Azure, GitLab, Docker Hub, OCI, and then a quick GP demo at the end. So I'll go ahead and share my screen and we'll go ahead and start with the GCP artifact registry. So just to show you what we're looking at here, I have pushed into this registry. Can everyone see my screen?

Zane Hamilton:

We can.

Forrest Burt:

Fantastic.

Zane Hamilton:

Okay. At least I can.

GCP Container Registry Interface Live Demo [22:32]

Forrest Burt:

Fantastic. Okay. So what we're looking at here is the GCP Container Registry Interface. So this is the GCP Artifact Registry Interface. So this is an artifact registry that we've created on GCP. It's called containers. I have, if I go back here, you can see the full list of them, but I don't exactly want to do that. So there's a whole bunch of other containers that are also stored in this registry. Docker containers and Apptainer SIFs together. So you can see I've got this container demo container that I've gone ahead and pushed here. If we go and look at this we can see that this should be an ORAS image that gets pushed up there, but sometimes the tagging on it gets a little bit odd. But this is an ORAS type image that we have pushed up here.

What is OCI and ORAS? [23:20]

Dave Godlove:

We might have to time out at some point, maybe not right now, sorry. But at some point we might have to talk a little bit about what things like OCI and ORAS mean.

Forrest Burt:

Dave, if we want to jump into that right now, that'd probably be a good time if we want to really quickly explain OCI and ORAS and that type of thing, because a lot of these registries that we're using are based on that concept. Just to preface the discussion a little bit, we have OCI type images, and then we have ORAS, which is a way of using these container SIFs with OCI registries. But then I'll let you explain that so I don't confuse things.

Dave Godlove:

Sorry to interrupt. I should have jumped in sooner and said something. So all right. So, a few acronyms to be aware of. We've already talked about SIF Singularity image format. SIF is an image, it's basically a file system image saved with some metadata in one file. OCI means the Open Container Initiative. And way, way back in the day around 2015 or so, somewhere in there, the company Docker decided to start a consortium called the Open Container Initiative to set up a bunch of standards for how containers should be stored and run and all that kind of stuff. And they basically started the standard based on Docker. And so that's what the Open Container initiative means. OCI standards cover the container format, the container runtime, and now most recently registries as well. So there's a standard for how you store containers online, and we're going to be talking a lot about OCI registries today. Oh, Rose has got her hand up.

Rose Stein:

So, this explains why more times than not, you're messing around with Apptainer and you see Docker, right? And you see, it's a Docker pool or go get from Docker, Docker Hub, or Docker something or other. And people are like, wait, if I'm using Apptainer, why does it say Docker? And this is why.

The History of Docker Support in Apptainer [25:37]

Dave Godlove:

Right. So a long time ago, shortly after Apptainer became a thing in its previous incarnation, Singularity, one of the community members, and I'm going to give her a shout out because she really helped out quite a lot. Vanessa Socket said, wouldn't it be cool if we could download OCI containers directly from places like Docker Hub and run them in Apptainer then Singularity. And so she submitted a bunch of Python code that allowed us to do just that. And so that was hugely influential and a big bang and really made the container take off. Because now instead of building all your own containers, you could just go get them. So fast forward. And so then I covered OCI a little bit and what that means.

The last acronym I wanted to cover was ORAS. So fast forward a little bit, a bunch of people in the container community got together and said, wouldn't it be cool if we could use an OCI registry as storage - ORAS, and so wouldn't it be cool if we could just push random data, any kind of a data blob up to an OCI registry. And some folks actually from Microsoft talked to some people in the Singularity community and said, wouldn't that be cool if Singularity, and now Apptainer, supports that as a protocol that you can use to store SIF containers. So that hopefully loops it all together. So now, it used to be that people would build their containers using Docker files, push them to someplace like Docker Hub, where they could get at them easily, and then they would pull them with Apptainer and transfer them from layers into a SIF at that point in time. Now, with ORAS, it's possible, as long as the registry supports it, with I assume all the registries do now, it's possible to just push your container directly up to an OCI registry as a SIF and take advantage of all the great things that SIF offers, which hopefully we'll be talking a little bit about later on. Okay. Sorry, level set terminology.

Zane Hamilton:

It's perfect. Thank you Dave.

Storing SIF Files in the Cloud [27:57]

Forrest Burt:

Thank you very much Dave. Exactly. So, what we're going to be seeing here is leveraging all that tech that Dave just discussed in order to be able to store these SIFs in these cloud registries. So to switch back to my screen sharing here, we can examine this image. For whatever reason, when we look at this image in the artifact registry, it shows up as a Docker format. So I don't know if they're wrapping this as something like that on the other side, but this is a SIF that we've stored up here in the GCP artifact registry. You'll notice so we can get a pull tag to it with this, this, and this. So I'm going to go ahead and share a terminal really quickly and I'll show you all how we can pull down the tag that we have here for this. You can see that we've tagged this stable, so we can imagine this is maybe like a stable release version of our application or something like that. So we'll go ahead. I'm managing the fact that I have to remotely log into like five different container registries here and sometimes the credentials get a little mundged. So, bear with me if I have to drop off, not drop off, but drop off screen sharing for a second to set up a credential in the background. I think we should be good, but we'll see how it goes here.

Zane Hamilton:

Live demos Forrest…

Forrest Burt:

Always. Let's see here. Let's see if I can make that a bit bigger.

Rose Stein:

There you go. A little more. Nice. Thank you.

How to Obtain Authorization Tokens in GCP [29:39]

Forrest Burt:

Okay, there we go. Awesome. Okay, so we're here in the terminal. In order to go ahead and interact with this GCP registry I have got to do a Apptainer remote login -u oauth2accesstoken. And we're going to paste in the name of this registry. So let me grab that really quickly, pull the tag and then I'll back a little bit.

But when I do this, I'm going to, in another screen, just do a quick GCloud off print access token in order to get a temporary access token into this registry. I believe my authentication should already be set up but I'll just do this just to show how the login works on the command line. So when we do obtain a remote login, we have to provide -u, which gives us a username, and then we have to provide the registry here at the end. So this is the URL to it. You'll notice that we're using the ORAS spec. That means that we want Apptainer to treat this as a registry that we're going to put SIFs into. So we'll go ahead and do this, get our password token request, and I'll paste in my GCloud auth print access token. I'll get that token stored there. If I want to go ahead and pull that container.

I can go ahead and use this right here. So just referring directly to the container pulls the URL that we got from the top of that. So just that CIQ/SA/containers, and then the name of the container that we put into the registry as with the tag. So we can go ahead and run this and it's all the same container that I've put all around these. So if I forget to do something like a container cache clean between them and it says using cached ORAS image, that's just because it is detecting that it's got the same image coming down from that registry. But I will try to remember to obtain her cache.

Clean each one of these so we can see a clean pole. So we'll go ahead and do this and we'll see if this will start to download an ORAS image. This is pretty small, just like a hundred megabytes or so. So we should see this finished pretty fast. But you see, we get that indication that we're pulling down an ORAS image. We get that right there. This image is just a really simple Python type image. It's just what I built with this container.def here. So we should be able to go ahead and do a container shell demo of this and then we can do our python3.

So that's the basic of the container that we're actually, or the the basic idea of the container that we're pulling down here. I'm going to go ahead and remove that each time so it's not clobbering itself. What happens is we want to push a container up to GCP. We can do that just as easily. Just go over here, paste this in, and we're just doing Apptainer push and then the name of the container. And we're going to push that to the latest tag. So say we have a new version of our application that we want to push out for testing we can tag that latest and go ahead and push and upload complete. So that finishes pretty fast. And when we go back to this page over here, we should now be able to see that we have a new tag on that image.

Does Apptainer Have Any Container File Size Limits? [33:02]

Zane Hamilton:

While you're pulling that up for us, I know some of our customers have some fairly large container images and I'm sure that some of the container registries have limits on size, but Apptainer doesn't care about the size of the container. Correct?

Forrest Burt:

Can you say it one more time, Zane.

Zane Hamilton:

So I know we have some customers that have some fairly large containers, like several gigs. There's no limitation or no size limitation on Apptainer. It doesn't care how big.

Forrest Burt:

Nope, absolutely not. You can build essentially an arbitrarily large container out to the limits of the performance of your file system and the storage space that you have. There's strategy there, you probably don't want to build a mono container that's hundreds of gigabytes in size, because that becomes very unwieldy to move around versus the concept of containers, which is to have in a lot of cases a single app just kept in a container. But it can do any size. When you build an container, if you're like building from an OCI image, it takes all the layers that you put into a Docker container, squashes them together into one file system with MakeSquashFS, and then you just have this essentially SquashFS image there that represents the rest of the container. So I've gone ahead and pushed this unaware that pushing the same container is actually just going to retag it with the same tag. So you can see that as I've gone ahead and pushed that up I've got now the latest tag applied there. But if there was a material difference between those two containers, we would see like for example.

For example, another container that we have here that's also a SIF. We have a number of different versions here as things have been pulled, and then we just maintain a stable version of it. That's the latest one. So that's the Google Artifact Registry. You can see we can store SIFs there, we can move things around. We'll go ahead and switch to the next cloud that we're going to look at, which is the AWS ECR. So this is the Elastic Container Registry that AWS provides so we'll go into here. So some of these places do the registry set up just a little bit differently. But I'll explain what we're looking at here. So we're in our Amazon Elastic Container Registry. I've got our Apptainer demo container here. And you can see we've got the stable image tag applied there, just like we did in the previous one. You're going to see that we basically have that same thing across all of these clouds. We have in Azure, a stable one in Oracle we have a stable one. And then in GitLab we also have a stable one. But here in the ECR you can see we have this artifact type other which is just indicating that it's not exactly a Docker container. It's something different. In this case it's a SIF. So you can see that we've got one waiting there. I'll go back to the command line.

Amazon AWS Apptainer Live Demo [36:02]

And we will go ahead and do an Azure deal. Where's that? There we go. Or sorry, not Azure. I may have misspoken an AWS deal. So in order to get in here I'm going to just go ahead and do this right here. So just AWS ECR get login for the region. And then I'm just piping that into Apptainer. I need this right here.

Oh, no, there, it's, I'm just going to copy it. I also need, of course, the actual URL that we're going to. So there's that right there. So, AWS ECR gets the login password region et cetera. That basically just gets you like a token that you can use to log into the ECR. We're going to pipe that through to Apptainer and just use the password standard in the option there. The username AWS you can find the username that you're supposed to be using generally in like the pulling instructions and tagging instructions for the registry. So a lot of times that'll tell you there's like a standardized username like you saw with GCP. It was oauth2accesstoken. With AWS it's AWS with Azure it's a bunch of zeros. So you'll see that generally they have something like that. So that's why in this case we have that username. And then, like I said, I don't want to reveal this token, so I'm just going to use this password coming in from standard in we'll go ahead and do this. And we once again got the token stored confirming that we have a good login.

I can go ahead and like I did before pulling down this stable release. I've got there. So I go ahead and pull down the stable release, copy this.

There we go. So I should be able to pull down the stable from this. I'm going to see if we get downloaded ORAS images. I'm probably, let's see, we'll wait a moment. There we go. We can go back and go into this and one's again so that you can see, we can pull that down from the AWS ECR there. We can also push a new tag. Like I said before this might just cause a duplication because I didn't realize that it's going to just do that tag on there. But if we do this Apptainer push container SIF like this, we can go ahead and container up, which completes pretty fast. Once again, I can, I think it's completing pretty fast because on the technical level of the hashes it's the exact same container that I'm pushing up for latest that I'm pushing up for stable in this case. So you would see a little bit of some separation there, like I showed with GCP. And you'd also expect a little bit of upload time. But in this case it's just recognizing that it's already there. So if we go back to the ECR, we'll be able to see that we've got upon refresh, this image is now tagged, latest and stable. So we can make changes to it there. So that's the AWS ECR. Let's see here.

What Happens if You Don't Clear Your Cache Before Pulling From a Different Registry? [39:40]

Brian Phan:

Oh, first I have a quick question for you. So if you didn't clear your cache before pulling from a different registry, would it just use the container that's cached then?

Forrest Burt:

I believe so. Because I think when it pulls that down from the registry, it's going to take the hash of what it finds at the registry. It's going to take the hash of what's in the cache. In this case, I don't think the registries would be doing anything to alter it at all. So I think that you're going to get that, but this next time I won't clear it and we'll see what 

happens. I'm pretty sure it should just cache. So that is the AWS ECR. I will move over to my next one, which is Azure. So here in Azure we can create container registries. We have our Azure Apptainer demo one. There's a lot of information inside of this. We're looking at repositories, which are the actual containers that have been pushed up to this. So you can see we have once again, a stable tag here. I can go ahead and pull this down once we switch over to the command line. All right, so here we are.

The token in this case comes out with a little bit of extraneous information, so I have to copy it from another window. So gimme just a moment. Copy that. So in order to get into this registry, we are going to do this right here. So you can see, Apptainer remote login username, a bunch of zeros, and then this azureapptainerdemo.azurecr.io. We can go ahead and copy the token. And we have our token successful message, meaning we've logged in. Once in here, we can go ahead and pull this down. And so then when I go ahead and do this, see if we can pull this down. And it does use the cache SIF image, so there we go. So, once again, as usual, that's the same image that we've had there. And then if we want to go ahead and push, I wish I had a different container right handy to push up to this so that we can see what it looks like when it's not just pushing the same thing.

Dave Godlove:

If you build a new container from the container that you have, it won't be the same container because it'll have a timestamp saved into the SquashFS file system that is in the SIF.

Forrest Burt:

So like from or like what's the bootstrap method on that?

Dave Godlove:

So you could just type the command completely, the command line, just build, so the Apptainer builds the name of the new container, and then the name of the old container, actually the full path to the old container.

Forrest Burt:

Cool. All right, let's give this a try. So we want to go ahead and push this latest version that we've just created of our application. We can go ahead and do Apptainer push, we'll do Apptainer over container -2.sif. And then once we go ahead and do this, this will actually take a second to upload because it's not just using the cached one but we'll see it appear there as I believe a new image in Azure in the same way that I showed there with GCP, how we have that different tag from here on out, I'll just use this container2 one, so we can see that. I apologize for not just doing that from the start. It's always something that gets you at the demo. So we might be waiting a second. This is a 119 megabyte image.

Does Every Registry Follow the OCI Standard? [43:53]

Dave Godlove:

And while we're waiting for that one of the things that you're highlighting, which I think is really cool, is that each, I mean, it's not cool that this is the case, but it's cool that you're highlighting it. Each one of these registries is a little bit different. It's worth pointing out too that even as far as like the way that each one of the registries implements OCI. So OCI is a standard, but it doesn't mean that everybody has to follow that standard, right? So each registry puts their own little flavor a little bit on OCI and might not implement all the different standards that are associated with that, or might do so to differing degrees. And then when you get into ORAS, things are a little bit different too as far as what's implemented and what's supported and what's not.

What is the Origin Story of ORAS? [44:47]

Forrest Burt:

Dave, correct me if I'm wrong, but isn't the somewhat comical origin story of ORAS that people were doing, like Docker Hub as a database type thing, stuffing arbitrary data into the layers of containers and then storing them out on registries and that type of thing?

Dave Godlove:

Could be. And I think that you, So I don't know. I don't know exactly. I wasn't around for the very start of ORAS. I was only around when folks started to come to us and ask us to make ORAS a first class client within Singularity. But I do know that for a while ORAS was not supported, explicitly not supported by some registries because they didn't want people just using it as a backup and putting all their pictures of their cats up there and stuff.

Zane Hamilton:

Gotta back stuff up somewhere, Dave, come on.

Forrest Burt:

So I'm pretty sure, just to demonstrate really quickly, I think my credentials are cached. But we'll find out here in probably about 20 seconds if this pings back unauthorized or if this actually uploads you'll notice that I've just gone back to the AWS ECR and I'm just uploading that container2 that we just had. So we'll give that a minute or so. And I think we're going to start to be up on time, so we're going to move it along a little bit more after this one, I think.

Zane Hamilton:

Oh, we're waiting for that. Are you guys, oh, of course it finishes right as I start talking.

Forrest Burt:

Sorry. Zane here we go. We're back into it. What were you saying, real quick?

Is It Possible to Automate Pushing and Pulling From Cache? [46:22]

Zane Hamilton:

Are you guys seeing people automate this type of build and push and pull to get it in cache? Is that something people are automating or are you seeing people do this real time like you're doing it now?

Forrest Burt:

Oh, absolutely. People are automating this. You can do the same type of CICD goodness that you can do with Docker entirely with Apptainer. So, this is entirely a process that you could have automatically building, pushing, testing, deploying. You could build those same types of pipelines Apptainer in this case. It's convenient to manually upload it here.

Dave Godlove:

I was going to say, Brian, go.

What is Brian Phan Working on at CIQ Right Now? [46:55]

Brian Phan:

Yes, this has been something I have been working on, but I'm still trying to figure it out. Like at CIQ we work with a bunch of different scientific software. So getting all of these different tools to build at scale and in a cost efficient way is a little bit challenging. But look forward to a future webinar where we can cat demo some of this type of stuff.

Zane Hamilton:

Oh, very nice, Brian. Thank you.

Generating Access Tokens for Apptainer With GitLab [47:23]

Forrest Burt:

Awesome. So I went ahead and, let's see, I did pull, I did push. Okay, good. We'll go back to my menu here really quickly and I will just show you that here in Azure.

We have our latest and stable containers now separated because we rebuilt that container. And if we go to the ECR, we also see that we have two different tags in there now. So you can see how different images will indeed engage different tags, just as you would expect. The next one that we're going to look over is GitLab. GitLab is a really common DevOps type platform or is this window out? There we go. You can see that I've got my GitLab containers repo here. It's got a container registry inside of it. I have my Apptainer demo container here with my stable tag. If I want to go ahead and jump into this. This one right here, isn't it? Yep. There we go.

so for this one right here I'm doing just the same remote login. I'm also pasting an access token that I got in from here. When I say I'm pasting an access token in this case, I go generate that from the GitLab website. You go generate a personal access token with the read registry capabilities, you need the write registry capability in order to push up to it. But we'll go ahead and do this. We'll find my note with the token, paste that in, and we've got a successful login. I can go ahead and Apptainer pull that, there we go. Cache.sif image again that we had out of that. And then I can go ahead and of course, if I would like to, something up to this. So there we go. Oh, whoops. Dave, while we're waiting for this one to upload, that's the GitLab part of it. Do you want to jump into the Docker Hub part of it?

Why Does Pushing SIF Files To Registries Matter? [49:40]

Dave Godlove:

Sure, absolutely. So I wanted to highlight too. So you might be thinking to yourself okay, it's great that you can push native SIF files up to these registries, but why would you do that? Why would you care? So I want to highlight some of the advantages or some of the things that SIF gives you, especially when you're working with Apptainer instead of working with something like Docker or a PodMan or whatever, and then show how you get those advantages of working with a SIF if you are able to push the SIF directly and use it directly throughout the life cycle of the container. So last night in putting this together, I went ahead and I created a repo up on the good old Docker Hub called Rocky Linux.

Docker Hub Apptainer Live Demo [50:37]

And I've been meaning to do this for a long time and create a bunch of different Rocky containers. And so it was good that I was able to do this. There already is a repo called Rocky Linux up there, which is the official repo, but those are all OCI images. And so these are all SIF images. Okay, so I tagged four different images. Two of them are the same. So 9.2 and 9.0 are basically the same. 8.8 and 8.0 are the same. But let me go ahead and show you really quickly how I created these. All right, so the 9.2 I created here with this Rocky Linux 9.2 def file. So if you look here I'm going to start talking about software supply chain and knowing where your software comes from and verifying that it's exactly the same software that you think it is when you get it.

Using Mirrors to Verify Authentication Security [51:36]

So in this case, normally when you start and build a container from a source, you usually build it from another container. What you can see here is I'm actually building from the upstream mirrors. So I'm skipping the whole grab a container from Docker Hub or Harbor or Quay.io or whatever step entirely. And I'm just building directly from the mirrors that you would normally build your operating system from, right? So that's cool because it means that you, there's, there's less people that you have to trust. Basically, when you create this container, you can be more sure of the contents of it. Okay, so I built another one here called Rocky Linux 9.1.sif. And I haven't pushed that up to the registry yet. So let me go ahead. I want to push that up to the registry, but once it goes up there, I'm going to want to download it again later.

And when I download it again later, I want to make sure that I actually get the same container that I pushed up there. Okay? So first off, let me just show you. So this 9.1.def this one I had to use a slightly different URL because that's in the vault. Now, that's not in the public repos. So when you're building from the upstream mirrors, it's a little bit more difficult. But what I'm going to do to make sure that this is the same image. When I pull it later, let me just go ahead and demonstrate. So if I do Apptainer pull ORAS. So if I pull the 9.2 image that I've already created as an ORAS it's going to give me the cached one, but that's fine in the interest of time, if I've got that one here, I can verify.

And this works because I've signed this image. And not only can I verify this image, and I know that the fingerprint is supposed to be this fingerprint, by the way, I actually have this fingerprint also on the repo. So if you wanted to verify this image, you could do so too. And not only can I verify this image, but anybody can verify that this is the image that I created because the public key material is up in a public place and the Apptainer will automatically download it for you when you try to verify. So that's something that you guys can do on your end time, is try to pull these images and verify them. If I try to verify the one that I just created for 9.1, though, that's not going to work. Cause I haven't signed it. So let me go ahead and sign it. And I have to enter my pass.

Zane Hamilton:

I saw it.

Dave Godlove:

Oh, okay. My bad. That'd actually be really bad if you saw that because this is like my production key that I use for all my important stuff. So this repo, I actually think that this is like something that people might use. So I'm going to go ahead and push now. You can see that I already tried to do that once. I'm going to go ahead and push this to Rocky 9.1.

Using ORAS to Bypass Mirrors For Software Supply Chain Security [55:17]

And as soon as that gets done here, we should be able to refresh. And now we've got the 9.1 tag. So that's pretty cool. And so not only can you verify just to complete this. So there's two more things I want to talk about. Okay? So not only can you verify the image just at the command line like this, but another thing that you can do, let's say I wanted to build from this image, and once again, we're talking software supply chain. So I want to make sure that what ends up in the image that I build is exactly what I think is in the image, and it's not going to be tampered with or messed around by some malicious entity. Well, what you can do when you now build. So it's cumbersome to build from mirrors all the time.

How to Perform Fingerprint Verification With Docker Hub [56:12]

So I can take advantage of the convenience of building directly from Docker Hub, but I can use the ORAS protocol to build directly from a SIF file. And oh, by the way, when it comes in, I can check that the fingerprint is that fingerprint that I expected it to be so I can perform a verification step right here at build time. And then if this fingerprint is not the same, then this'll bomb out and it won't work. Does that make sense? I'm just going to talk about it and not show it right now because I think we're a little over time. And just one more advantage that working with SIF gives you over working with Docker Hub or OCI images when you're in Apptainer. So let's say I wanted to download the famous LOLcow container because I wanted to know how it was originally built.

Well, if I did that, you'd think I could use something like Apptainer inspect def file LoLcow, but if I do that, that's supposed to give me the definition file that was used to build this. But since I pulled this from Docker Hub, the actual definition is just that I pulled it from Docker Hub, it created this from layers that were up on Docker Hub. So that was the build. And so that doesn't give me any information at all. Instead though, if I do a pull, I have this same image saved as a SIF with the tag SIF here. So if I pull that same image as a SIF file, now I can do my inspection.

And it's actually going to gimme the definition file that was originally used. So you just get more information and you've also got that software supply chain aspect that I was talking about. So just like to zoom out and summarize what I'm trying to say there when you're working with Apptainer. So, there's advantages to building as OCI containers and using those, like if you're going to be using both Docker or PodMan or something like that and Apptainer, that's probably the route you want to go. But if you're working on Apptainer now that OCI registries support Apptainer, you get a lot of benefits by just actually working with SIF files and using them as is. And some of those benefits like software supply chain and knowing where your software's coming from that's not tampered with are actually really important security features.

Forrest Burt:

Thank you Dave. That was fascinating. Very much

Rose Stein:

It was, and it's even better too because when you're talking about it, you can tell how excited you are about it, that like this didn't always exist, right?

How New is Uploading SIF Files to the Cloud? [59:20]

Dave Godlove:

And as recently as just like six or eight months ago, Docker Hub in particular would not let you upload SIF files. So it's only been within the last year that Docker Hub even allows you to use the ORAS protocol to upload SIF files. So it's pretty cool. It's new stuff.

Rose Stein:

It's really cool. Well, thank you to all the people that were involved in making that happen. And it seems like it's so exciting what's happening in the world of containers and yet as we're looking at what we have and where we're going and all the things that we can do with them, it's even more exciting. And so I definitely want you guys to stay tuned as the weeks go by because we're going to continue to share with you and show you and demo for you and encourage you to play around with this software and show you how it can improve your compute resources and your abilities and also the way in which you, if there's admins watching how you can encourage the people that you are, like you're supporting the systems, but like all the people that are then using the system Right. It makes it a lot easier for them to be able to access those resources too, without you having to do everything for them. Right. Which is really cool. So I know we are closing out here. Are there any final thoughts that you wanted to add here? We'll start with Brian.

Brian Phan:

No final thoughts. Apptainer is great.

Rose Stein:

That was the perfect final thought. Apptainer is great. Awesome. Forrest, do you have a final thought?

Does Apptainer Work With Oracle Cloud? [1:00:58]

Forrest Burt:

If I may have a final screen share just like 30 seconds? So I know we're up on time, so I won't dive too deep into the rest of this stuff, but I do just want to point out really quickly if we're back over here on GitLab, we have that latest and stable container there. This whole type of thing. And of course it's going to want me to log in. This whole thing also works up on the Oracle cloud. You can use the Oracle Cloud Infrastructure Container Registry there to store these types of things as well. You're only going to see this stable one in there, but you can see here that much like everything else we add we've got that stable thing there. You can log into one of these with just one of the standard auth tokens that you can make through OCI. And this one was the one with the most complicated login. But we are up on time, so I think, I suppose we will skip.

Dave Godlove:

I soaked up all the time there talking about the advantages of SIF.

What is the Most Exciting Thing About [1:02:06]

Rose Stein:

No, it's totally awesome. Forrest. Just like because we were talking about time and how these things didn't always exist. What is the most exciting thing about being able to have containers up in these different cloud environments?

Forrest Burt:

Just what we're seeing here, the massive extensibility of being able to take the architecture that you've built and the systems and things that you've built that you've put tons of effort into. And that instead of having to move around and rebuild these different clouds or change the technology that you already have, et cetera, into a different format. Apptainer works for all of the same use cases that we would expect to see out of a cloud container solution. So there's nothing that we're essentially limited by here. So it's very cool to see Apptainer working in the cloud. And that basically concludes Cloud Day . We can also, I was going to pull down an image onto a GCP enabled Rocky Linux instance just to, so that you can pull and run out of that type of thing. But I'll leave that as an exercise to the reader to deploy that type of AI and get that architecture running.

Dave Godlove:

So everybody should know this. So Cloud Day was Forrest's idea and Forrest basically put all this together. Maybe we could put together a blog post or something since we already have the commands and stuff and like a companion to this so that we get all that information out there.

Forrest Burt:

Fantastic idea. Keep an eye out for that everyone. That'll be coming out soon. Thank you all for Attending Cloud Day.

Rose Stein:

Yay. Yes, thank you everybody, make sure that you like and subscribe so that you can get notifications of all the different things that we are pushing out to you. And we'll be back here at the same time. Same place CIQ in the house. I do want to give you a moment. Zane, dude, you want to pop in and say hey, I feel like I've taken over.

Zane Hamilton:

No, I appreciate it. I got pulled into like three different things all at the same time, so.

Rose Stein:

It's exciting. Never a dull moment, right? Never a dull moment here. Awesome.

Zane Hamilton:

I appreciate you guys joining again. It's always fun to see you. I feel like I don't get to spend enough time with you guys anymore, so I love it when you're on. Thank you very much.

Rose Stein:

Awesome. All right, we'll see you next week. Thanks for joining.

Dave Godlove:

Later.