Apptainer: Securing Containers with Signing
In this webinar, we will be demonstrating how to trust containers with signing using Apptainer (formerly Singularity). We will also be discussing key generation and exchange, SIF, Apptainer Execution Control List, and building, signing, and publishing a container from a CI pipeline.
Webinar Synopsis:
Speakers:
-
Zane Hamilton, Vice President of Sales Engineering, CIQ
-
Gregory Kurtzer, Founder of Rocky Linux, Singularity/Apptainer, Warewulf, CentOS, and CEO of CIQ
-
Forrest Burt, High Performance Computing Systems Engineer, CIQ
-
Jonathan Anderson, HPC Systems Engineer, Sr., CIQ
Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.
Full Webinar Transcript:
Zane Hamilton:
Hello, and welcome to another CIQ webinar. We appreciate you joining us. Good morning. Good afternoon. Good evening. Wherever you are. Today we are going to talk about securing and signing containers in Apptainer. We are joined by Jonathon Anderson, Forrest Burt, and Gregory Kurtzer.
Zane Hamilton:
Thank you for joining us today. Jonathon, since this is your first time on the podcast, could you introduce yourself and tell us what you do at CIQ?
Jonathon Anderson:
I've been all over the kind of academic HPC space: I've been in the national lab system and the university space. I've been an HPC Sysadmin for most of my career working on some of the top 500 systems, but also some systems here and there at different scales and I am really glad to bring that experience to CIQ and start solving problems here for us and our customers.
Signing a Container [00:01:27]
Zane Hamilton:
Jonathon, What does it mean to sign a container? Why would someone do that?
Jonathon Anderson:
Sure. So Apptainer signatures are based around open PGP. The most common way of seeing this is by signing emails. This way you can verify that the email is coming from the person you expect. It works on this idea of a web of trust where you have an asymmetric key. So you have a public and a private component. Through some out of band mechanism you exchange your public keys with other people that you want to correspond with. You can submit assertions of trust into some kind of key server. There are many different kinds. We will be dealing with the most standard. For Apptainer it's pgp.org.
You can verify a “trusted person” by having their key. You can also trust other keys that are verified. They will likely be the right author/organization of a container. There are other ways to get keys. A good example is downloading a PGP public key from an HTTPS protected website from a company. From that point, you can verify that the containers that you've pulled down are valid and from the point of their publication.
Different Keys [00:03:33]
Zane Hamilton:
So having you talk about keys and keys from here, keys from there. So obviously a lot of different keys could be involved. How do you deal with a bunch of different keys? Is there somewhere that they live? How do you get them? Where do you put them?
Jonathon Anderson:
Yeah, so generally speaking, there's some kind of database on your local system in, what's generally called a key ring. But then the public key material can be uploaded to any random server because it is public. You can use those key servers to pull down other public keys. On your local system you'll have a key ring of public keys of other people and your own that you trust with various degrees of certainty. And then a private key ring that stores the secret material for keys that you own. You can sign with this. Another example is I have the PGP keychain app installed for Mac OS. Apptainer also has a key ring that it will use by default, both for verification and signature.
When and Why Containers? [00:04:38]
Zane Hamilton:
Could you tell me what the purpose of containers is? What are containers used for?
Jonathon Anderson:
So from my perspective, the point of containers in general is, or one of the biggest points is portability and the ability to publish a workflow and then other people benefit from that workflow. A typical way that people work is you develop a container and then you push it up into a container registry or an artifact registry of some kind. Those may be private registries, and then maybe the signatures aren't as important, because you have to authenticate to the registry, but lots of these registries are public. With standard web stuff, the general internet community has decided to protect that through TLS signatures, X.509 signatures on HTTPS websites, where you can verify the identity of the server that's serving that material to you.
PGP signatures on a container are a way to do that in this other space. For example, you download a container from a container registry that you trust. Now that container is just a file. After a while, who knows where it came from? One of the benefits of the Apptainer mechanism for signatures is with SIF container formatted files is that the certificate is bundled with a single file. So there's no separate signature file that you have to manage or keep track of. It's just, if you have a container and it is signed, you can verify that it is signed by the entity you expect.
Zane Hamilton:
It's interesting that it's all together in one file. This way I don’t have to look around for a bunch of different things. I hope that you'll show us how we can see that.
Gregory Kurtzer:
When containers were starting to gain in popularity and we were starting Singularity, we started to integrate with Docker Hub registries. This was before OCI existed. We were basically pulling containers down from that interface. The questions started coming up over and over again, which is right. We don't just go and install random RPMs on our system that are not cryptographically signed, that we don't trust. Right. We may be pulling packages off of various mirrors for updates, but they all contain a signature. And each one of those signatures can be verified by the material, the public key material that we have on a particular system.
So it doesn't matter what mirror the update came from. We can validate the authenticity of that package through the asymmetrical key method. What are we doing with containers? In many cases, we're downloading containers of unknown origin. We don't have clear visibility on who has touched them, the provenance of that data, and the material within those containers. And and then worse, we're actually executing those on giant supercomputers. Even worse than that, we're even doing that as root. it's, yeah. We got a major security vector going on that we have to be thinking about.
How do we guarantee the trust of any of those containers and all of the material within those containers so that they can be leveraged? We feel like we can guarantee who it came from, where it came from, and that it has not been tampered with. These are really good reasons why I think everybody should be thinking about cryptographic trust and validation of anything that you're going to be running or leveraging on your computer. This is not just Linux, if anyone has a Mac and you download an application and it is not cryptographically verified by apple as going through the trusted mechanism, you have to jump through some hoops just to get it to work. This is a much simpler interface and it works extraordinarily well. It integrates with standard mechanisms that we're currently using within the industry for dealing with key material. Everything from standard PKI to standard HKP protocols. That is some background information.
Cryptographic signing a Container [00:09:56]
Zane Hamilton:
This is a completely different topic from encryption, right? In this case we are talking about cryptographic signing?
Jonathon Anderson:
That's correct. Apptainer does support encryption of containers at rest and during execution although it's handled through a completely different mechanism. Eventually we’ll do a demonstration, but for today we are going to keep it simple and talk about PGP signing.
Zane Hamilton:
I wanted to make sure that we clarify that before we get started. Could you show us the generation in exchange?
Generation in Exchange [00:10:40]
Jonathon Anderson:
I've got a terminal open with a very simple container. This is a “hello, world” container that I threw together real quick. It's based on the Alpine base container. It's really small, easy to upload, and download. I am starting the demo. The first thing I'll do is build this container. It’s important to note that because Apptainer supports this fakeroot mechanism I can do this as a regular user. I'll end up with a SIF file on the other end of this in a few moments.
Here we have a basic SIF file. It's built out of parts (we'll learn a little bit more about what that means in a moment). We can see that it's got this SquashFS System. That's the majority of our file system and then a couple other things here that are part of the container. We can't verify it. So if I say, Apptainer verifies hello-world.sif it has no signature so there's nothing to verify. Within Apptainer there are other ways to bring keys. We can generate a new key right here within Apptainer and use that to sign our container. Type in Apptainer key newpair. Filling in the metadata can take a moment. The issue here is that I’m running on a small VM. There is not much CPU power available.
Zane Hamilton:
Seems very similar to creating SSH keys.
Jonathon Anderson:
As mentioned before, Apptainer has a key ring in it. We can see the keys that are in our Apptainer key ring with this key list command. There are two. Key number zero here is my real PGP key. We'll use that in a moment. Key number one is the one that we just generated. It's our demo key. The public key has a fingerprint that we can use to identify it with key servers and each other. Sometimes you use that fingerprint to reference this key to Apptainer. Sometimes use the index. It's pretty clear from context, which one is appropriate.
From here with our new key, we can sign our new SIF file. It asks us which private key to use. It's asking us to specify by index. We can tell it to use key number one (our demo key) and because we're using the private key of our key pair to sign, we have to enter our passphrase. This is all about trust so we should have a passphrase that protects all private key material. We want to protect that private key so that when people receive a container that has a signature on it from this key they trust us. They will also expect us to maintain security of our private key or our passphrase around that private key. That way the private key is never sitting decrypted on disk in case there's a security incident on the local workstation that we're working on.
Now that we have signed and verified our container, we can see that the verification happened locally. We're doing this with the local key in our local key ring. We see which key has signed it. It gives us a fingerprint here and then shows us the way you can sign individual portions of a container. Here we have our entire container verified with that key. One more thing, if we list our container again, we'll see that we've grown another object. This here is the signature itself. It exists outside of this group one and is signing all of group one. There is a copy paste area here. It's signing the group one set of objects in this container. The next thing we can do is sign it again. This time we'll use the zero with identity which is a real one that I maintain and imported in out of band from the GPT keychain app.
Now we've signed it again. If we list our container, we see we have a second signature here. They're both in my local key ring again. We have our first signature with the demo that we had before and our second signature from the second key.
Is a Public Key Enough? [00:16:58]
Zane Hamilton:
If I had one of those public keys, could I execute it? Or do I need both keys?
Jonathon Anderson:
Container execution is a different question. By default Apptainer will allow you to execute containers regardless of whether or not you've been able to verify the signature. You can define a policy in the configuration for requiring the containers that you execute have one or more signatures. That's part of the execution control list (ECL). I Once you are in there you can say containers in this path must have one of these list signatures, it must have this list of signatures, or it may not have any of this list of signatures. Control execution that way right now in the verification logic or the way that the Apptainer verifies containers. It just verifies the signatures one at a time. And so something we'll see in a moment is if I didn't have this key in my local key ring and it wasn't available through a server, then verification would fail. It wouldn't go on to verify this. Even if we have the second signature it would say that the verification failed. I think that is an opportunity for change in the interface. There's nothing about the cryptographic signature that necessitates that. That is a specific centricity of the way that Apptainer handles multiple signatures right now. In theory, you should be able to verify that a container has been signed by any of these keys regardless of their order or how many there are.
Jonathon Anderson:
Now we have everything we need to pass this container onto someone else. The first thing I'll do is push this container up to a key server that we're using internally. We are going to use this one for the purposes of our demo. This effectively uploads this container up. I need to log in first to place the container up on a server. Forrest will then be able to access it and pull the container down.
Once that happens, we're going to demonstrate two mechanisms for key exchange. One is passing around your PR public key out of band. I'm actually going to download the public key off of this system and pass it to Forrest over Slack. It's a file and I'll just hand it to him by dragging it over there. Then we're going to push my regular public key up to a key server. This key has been pushed there before. It's an item put and processed. We can do it repeatedly with the same result. If I added email addresses to this key, it would update the record on the key server. Let's do that now.
We can export another key. Here, I'm specifying an ascii armored key. That's only to make it a little bit more palatable if we were copying and pasting the key around in plain text. We're going to export a key into a file. I'm going to export key number one (the demo key that we generated earlier) and that gives me this demo key.pub file. You won't be able to see this right now, but I'm just going to copy that off this server and onto my local system. Then I will drag it over to Forrest.
Forrest Burt:
Cool.
Zane Hamilton:
Super secure Slack communication for private cases..
Jonathon Anderson:
Yeah.
Zane Hamilton:
Private case.
Jonathon Anderson:
No, it's a, it's a pre-negotiated trusted communications mechanism.
Jonathon Anderson:
And so, with that, I'm, I'm basically done at this point. So I'm going to copy this over because I'm one more time, because I may have gotten it confused with a version of this key that we were doing before in test. But for now I have a copy of that key and should be able to take it from here.
Gregory Kurtzer:
You should be able to share the whole container over Slack too.
Jonathon Anderson:
In theory, this container for sure would be fine that way. Because it's not very big, but containers can get large and perhaps I wouldn't want to download and drag a multi gigabyte container over to Forrest over Slack. I think they might not like it.
Zane Hamilton:
That too. Not if you like your Slack me messages from prior to 10 minutes ago.
Gregory Kurtzer:
Yeah. All of a sudden my Slack bill is going to go up a lot. You guys.
I’ve actually seen people sharing containers over everything from Google drive to Dropbox, to FTP. I mean, of course I have been SCPing them around as well. Having the whole container in a single file makes it really easy to deal with. That was always one of the goals going back to the early Singularity days.We named Singularity after a single file. Singularity was always focused on the single file methodology and ideology. And to the point that was raised earlier regarding the cryptographic validation, what's really nice about this is again, we're handling this very similar to how RPM would handle this. We are handling cryptographic validation in a very similar way to RPM. There's no external services that are necessary to validate against everything contained within the actual image file. It is very easy to deal with.
Zane Hamilton:
I wish we could get a show of hands of how many people are using FTP.
Forrest Burt:
I interacted with one recently for some genomics data that I was pulling down. I know they are still out there.
Jonathon Anderson:
A while ago I had to build a multi-server FTP service for someone.
Gregory Kurtzer:
I believe there are numerous labs still using FTP as the interface to their archive storage system.
Zane Hamilton:
All right. Forrest.
Forrest Burt:
Cool. I just want to note, first off, before I jump into this, that when Jonathan sent me that key snippet through Slack, I got a message on Slack saying that I received a PGP snippet. It is awesome that it specifically recognized that it was a PGP public key snippet. and that's awesome. Not as much in the notifications, so apparently they've planned for it on their end. It seems that Slack has planned for this. But yeah, so you can see that. We'll jump in here to my component of this. I am going to copy the container URL that I'm going to be using here.
Jonathon Anderson:
Oh yeah. It it's the same one from before of course, but yeah,
Two Mechanisms for Key Exchange [00:24:30]
Forrest Burt:
I am in an empty directory. You can see that I've just made this so we can have a place to pull that container down into and pull that. The key I'm going to download into downloads. This is to have a place that we can do this demo from. First, I'm going to go ahead and get myself a copy of that container. So I'll go ahead and do a container pull and then provide that same URL that Jonathan just pushed that container to. You'll see, we get this message saying that we're downloading the image. I go ahead and do an ls. You'll see that it's there. That was very quick. You can see that I now have that same container that Jonathon built on his machine. It is now here on my machine.
If I do apptainer SIF list you can see that we have the same components inside of this that we had before. Just that definition filed, this was built from, a couple of JSON bits here that include some metadata that was built into this file when it was created. S The environment, the labels, and anything like that we then have that squashFS file system that contains the container file system proper. Then those two signatures that Jonathan signed this with. You can see this container comes down on my side with those signatures. If I go ahead and do Apptainer or verify, however, actually I think that might be Apptainer or Apptainer or verify it is. If I do Apptainer verify and then I provide this, you'll see that this fails to verify as Jonathan had mentioned. Even though one of those keys is out there on a public key server because I have imported that local key that he created here into my own key ring.
The result that we get is saying that this fails to validate because it can only validate one of those keys from that internet location. I still need to import that other one locally. I'll go ahead and download that key that Jonathan sent me which is right here. I can do ls-latr~/Downloads | grep demo-key. You can see the key there. I am going to move it here.
We have the key and the container file here. I'll go ahead and actually import this key into the key ring so that the Apptainer can recognize it. So we'll go ahead and do Apptainer key import and then just provide the path to that key file. You'll see that we get the message here that the key with that fingerprint has been successfully added to the public key ring. The first thing I need to do is do the Apptainer key list and you can see the two keys that Apptainer knows about here. One of these is basically just a test key that I put in here for something else long lost a lifetime ago. And you can see the other one here was just created and imported right now.
So if I go ahead and do this Apptainer or verify on this container again, what we'll see is that we should, instead this time get a message confirming that all of the keys on it can be verified. Because once again, I've imported that local key into my key ring here on my laptop. And then the other key that this is assigned with is sitting at that open PGP key store and the Apptainer will reach out and verify it from there. So we'll go ahead and run this, and you'll see, we get these two outputs here indicating that we have a local key being verified. We have a remote key being verified. So this is the one I imported. This is the one that came from open PGP, and you can see that we have a listing of the objects that we verified here. So once again, we can see those same objects that we had in there prior to this. And we can now tell that those are verified. So I just want to go ahead and show, I can run this container and you'll see that I get the quick little “hello of world” message out of it. But yeah, that is container signing.
Zane Hamilton:
So outstanding. Thank you.
Forrest Burt:
No problem.
SIF Format [00:29:00]
Zane Hamilton:
One of the other things I wanted you to talk about, Jonathan, is the format of a SIF. I know whenever you do the Apptainer list, it kinda shows you the different pieces of what's inside that file. Can you talk us through what that is a little bit, and then talk about the groupings and kind of how we can use that?
Forrest Burt:
Yeah, I can run through that really quick. I think Dr. Jonathan, but yeah, I'll run through that real quick. Let me stop sharing and share this other screen that I have here. So this right here is kind of the standard image that we use to express exactly what is inside of a SIF and exactly what that internal format looks like. You can kind of see how we represent the ability to both evolve a container after it's created in a verifiable way, as well as the ability to cryptographically sign the entire container itself. So as I kind of touched on there, when we looked at that Sy list, we had four different components in there. We had a Def file, two JSON files, and then that squashFS file system. Inside of a SIF you can see this image describes the structure of it.
So I'll kind of map what we saw there on the command line, to what we see in this image: the global header and the descriptors. We didn't really have a ton of visibility there. Those are kind of more, I believe, metadata that you would have to go into a X dump or something like that, probably to take a closer look into but for these other components here we can see recipe definition and labels and environment. Those map to the three different components that we saw outside of that file system when we did CPH list. So that one that was mentioned was the Def file. This is actually the recipe that the container itself was built from. The labels and the environment are bits of metadata that can be added during the container, their metadata that can be added during the container build.
I think Jonathon briefly showed what the container definition was built from. And I believe he put his name as the author in there as a label. So labels are just metadata that comes with the container that kind of describes certain user customizable aspects. The environmental variable set up that is present with that container. For example that would be that's going to be like the path, the LD library path, custom environment variables. Things such as those will all be contained within that portion of the CIF. The immutable runtime container image is the actual file system that contains everything the container is meant to contain. SquashFS is an immutable compressed file system.
So you can't really alter that, meaningfully after the container is built. And then obviously signing gives you another layer of verification that nothing has changed within that container. But that immutable runtime container image, when you go ahead and build one of these containers, it pulls down the base image that the container is going to be built off of. And then within the post section, it goes ahead and runs any of the commands that you put in there. So DNF installs compiling software doing downloads of different things. Anything like that is run and put into the container file system. And then when the container build is done the last step is that it takes that file system and basically makes squashFS with it. You get this compressed file system image, basically that represents everything.
The container was meant to contain on a software application level or the file system level. When we looked at this at the start there wasn't a signature block there. That's because there's not a generic signature that goes with every container that is only shown and present. If that's only shown in the present the signature is actually attached to the container. Up here at the top where it says “cryptographically signed” we can see what was represented when we verified the key. We had all four components: Def file, two JSON components, and squashFS. All four of those components were verified as a part of that verification operation because they are signed with that key.
And so that's where you can kind of see this concept of how the signature allows for the cryptographic signing of everything else within the container. While it's not the focus of this demo I want to explain what we mean by evolvable. Within Apptainer you can add a X, three formatted overlay image to a built container. This will add a writeable layer to that container. These are typically immutable. The squashFS isn't meaningfully modifiable because it's a compressed file system. For example, if you have some type of high performance computing workload represented within a container and you want to have outputs written to a place that they're also packed with that container.
You can add a writeable overlay, whether that be as an X3 image or a directory on your file system that the container can use for writing. This reminds me of some of the optimizations that Apptainer provides surrounding how it stores data. In a typical use case you are running some type of high performance computing system with thousands of processes starting up. All of those are doing small file IO or Python imports. Your file system can get bogged down with doing all of those different small metadata operations. But instead if you're running out of a container and all those metadata operations across your shared file system on your compute cluster, if instead, all those metadata operations are just directed to the squashFS inside of the container.
And it basically just becomes reading offsets into that squashFS. These read operations happen much faster. For the same reason, if you're using an X3 image overlay the writing happens a lot faster because instead of metadata operations on the actual file system, it becomes just reading offsets into an overlay image. So that's a bit of a long-winded explanation, but that's what's inside of a SIF. That's what can be inside of a SIF as well, if you make certain modifications to it. but you can see how the format allows for, as you know, we've touched on in various capabilities. a lot of very interesting features that are good for HPC. and yeah, it kind of contributes to why we do things with retainer and not other containerization platforms within the context of a high performance computing environment.
AI Training into a Container [00:36:00]
Zane Hamilton:
That's fantastic, Forrest. And it brings up an interesting point. I know Greg, you talked about this yesterday and being able to have a real world example of being able to do AI training into a container like that. Could you kind of tell us that story about what that really means having the ability to have that rideable space inside of the container?
Gregory Kurtzer:
Sure. Is that for me or Forrest?
Zane Hamilton:
I asked you, but I don't care. Forrest wants to tell it, Jonathan, anybody.
Forrest Burt:
I think what we're going over here is kind of a use case. We discussed yesterday the ability to selectively sign different components inside of a container. You may have noticed that when we were doing all those verifications, lists, and stuff, there was in addition to a component ID for each for that Def file, those two JSON files and that squash FS, there was also a group ID that was associated. You'll notice that when we were looking at those signatures, it shows that we've signed the objects in group one with that key. This leads to an interesting use case where if you have evolving data that you want to move with a container. You have a machine learning use case that you need to have new data constantly being fed into developing your data set.
You want to be able to easily pass all that code and all that data around to people in your research group as that data still develops. Those data sets kind of take shape into something that you could do in order to provide verification. Sign all of those group one objects inside of that container. Sign the actual base image and the definition. Then add an overlay into a different group inside of that container that is then unsigned. It remains mutable as a result. You can have the base code and the base container say your model code is stored immutable, being verified, and signed by people in your research group. You can actively still modify that overlay that contains the images and the data that your model is supposed to run over. You can still easily add and remove from it. Whatever you need as your whole code and project takes shape in a way that keeps the original stuff verified and provides editing access.
Gregory Kurtzer:
I just want to mention that my artistic abilities in Google slides have come a long way since then. That image brings up memories for us. It's an older image. I created it for a supercomputing presentation. I know YK committed it to the repo. The image is based also on an ELF binary image. It lends itself to a lot of functionality in terms of being able to extend and do different types of things with that image. Another important thing to mention is the image itself is executable. If you have a Singularity file container and you've specified a run script you can actually execute that container like a binary. You can put it in your path and it will execute the stack within the container. This is similar to any other program. So it's, it is super cool. A lot of functionality with regards to that. And it's quite extensible. Jonathan, I think you came off mute, but I didn't stop blabbing.
Jonathon Anderson:
No, I didn't know how far you were going to go into that. I, that was one of the first really aha surprise moments for me was seeing precisely what you just said, because of the use of the, an extension of the L format, you can just dot slash your container and it runs, it's pretty cool.
Difference Between a Local and Remote Key [00:40:00]
Zane Hamilton:
That is very cool. So I think we had a couple of questions that popped up. I think Justin was the first one. So why is there a difference between a local and a remote key?
Jonathon Anderson:
In general this is just the reality of the exercise. It is just reporting whether the key is stored locally or was pulled from a key server. Apptainer by default is configured to use an open PGP general public key server. Remote keys are coming from there. You may want to know where the key is when you are verifying. You can pull that key down from the key server, it's Apptainer key pull and then the fingerprint. It will be copied into your local key ring. Then it would be reported as a local key. In general you could imagine it's public key material. There shouldn't be a barrier to uploading that key to any key server. Why does it matter?
Even in this scenario I passed our demo key to Forrest out of band because I don't want to throw random public keys with an email address that are never to be used again into the key server. We have that flexibility. It could be privacy related and you don't want that key up on a public key server. It could be for convenience. You just want to be testing something to work just with the local keys. You write to the public key servers and the keys never go away. They can be revoked. They can be marked as active, but they are never deleted. It's still an external service that you're dependent on. If you have a production level service that depends on the ability to verify a key, you probably want to pull that key locally. You will want to verify it with your local copy rather than expecting to always be able to go out to the internet and pull that key down off the remote service.
So it's just the specifics of where the key happens to be located right now and then reporting that to you. It might be important in some circumstances.
Gregory Kurtzer:
Just to follow up on the last point that you just mentioned. It is important to know not all levels of trust are equal. For example, the key that you sent to Forrest could have been not just sending it over Slack, but you could have also been doing a verbal verification of that fingerprint with Forrest . Typically keys that are locally installed are ones that people consider most trusted. These are in my trusted key ring and that has a greater sense of security and trust than one that I'm downloading from a service on demand over the internet. If somebody wants to have that level of trust what needs to happen is Forrest would call up Jonathan and say, “I found your key on this public key service I'm pulling it.”
“I just wanted to verify your fingerprint and make sure this is the right key.” Verbally go over the last six characters or however many characters you want to make sure that you trust that key. When you import that into your local key ring you know for a fact that that key came from Jonathan and now anything signed with that private key can be trusted with that public key. St really just demonstrates that as well as Jonathan was saying to let you know how we are verifying any particular container. I think the colors were green for local and yellow for remote. Green is good..Remote you will want to validate by hand.
And then of course, if that's something you're using a lot, you can import it. One other facet I'd just throw out there was, we're talking about local versus remote keys. We already mentioned execution control lists and ECLs are a really important way to guarantee that the containers you're using and the containers that a system administrator on a particular host allows. If you're implementing an execution control list for a particular container set of container fingerprints, you will want to make sure that those fingerprints are actually local. Not that there's an easy hack here, but it is theoretically possible that a malicious user can redirect that request. It goes to their fingerprint or a PKI type service, then get somebody to run some malicious code. There's a lot of under under work that needs to happen for an exploit of that magnitude, but it is in theory possible. So if you're going to go as far as to execute a control list, you probably also want to make sure that you're properly managing those keys and validating those keys locally in key rings.
Jonathon Anderson:
And in fact, the execution control list feature doesn't even interact with remote keys at all. For those keys to be verified, they have to not only be in your local key ring, but Apptainer. It was something we didn't demonstrate. Apptainer has a key ring for the user, but it also has a key ring globally for public keys. ECLs only validate based on the keys that are in that global key ring. It's not enough for you to have it. It's not enough for it to be present. The systems administrator that configured ECL in the first place has to have imported those keys as well.
Gregory Kurtzer:
I completely forgot that. You're right though. I remember it now. Thank you.
SIF Encryption vs. SIF Signing [00:46:11]
Zane Hamilton:
Very nice. So we have more questions coming in. One of them has been sitting there for a little bit. When would you use SIF encryption compared to SIF signing? Are the two features complimentary or do they support each other? Like why would you do one versus the other? Or not both?
Jonathon Anderson:
So signing is to verify the integrity of the container to make sure it has not been modified since it was published. To establish some degree of trust through the key material that you have from the past. It doesn't obscure the contents of the container from the person who receives it and from taking it apart to see what is in the container. If you need to do that and if the data that's stored in the container itself is secret, that's where you would use encryption. You should be able to sign containers and encrypt them at the same time. Some software environments do those together by using the same keys to sign and encrypt at the same time. Apptainer seems to use a different mechanism. It is RSA keys for signing rather than the PGP mechanism. That's something I personally need to look more into. I don't know, Greg, or, or Forrest, if you have more specific experience talking about them.
Gregory Kurtzer:
There is a neat feature that you can do to encrypt. For example, we went through this process and Forrest obtained Jonathan's public key. If Jonathan were to share a container to Forrest again, we can validate becauseForrest knows for a fact that this is trusted. Forrest can now create a container using the reverse process and encrypt it using Jonathan's public key. Jonathan would be the only person in the world. Well, whoever has access to Jonathan's private material, private key material and encryption or decryption key. Jonathan would be the only person who would be able to actually open up that container and look at it’s contents. You can now think of different use cases where you can verify and trust the integrity of not only the contents, but who actually signed. You can layer signatures to manage providence of containers and manage it as it goes through a DevSecOpscycle.
The author signs it, the security team signs it, or maybe the production team signs it. You require all of those signatures for a container to run on a particular infrastructure. We're not actually encrypting the data within that container. We can do encryption of that data plus the signing. You can imagine the level of security and trust that you're building in that. Forrest could be not only encrypting the container that he's sending back over to Jonathan, so only Jonathan can open it, but Forrest can also sign that container and hand his public key over to Jonathan. Now Jonathan knows it came from Forrest. He knows it hasn't been messed with and Jonathan is the only person who can actually open up that container, evaluate, and leverage those contents.
Apptainer and Open Source Key Servers [0049:44]
Zane Hamilton:
That is very cool. Very cool. All right, George, we got a few questions from you. I think the first one was what open source key server can Apptainer integrate with?
Gregory Kurtzer:
You want me to take it since I'm already talking unmuted?
Forrest Burt:
Absolutely. Yeah. Going on my head.
Gregory Kurtzer:
So Apptainer is leveraging standard protocols. HKP is the protocol that we're using to move the keys around. It is a standardized protocol. This means any key server or PKI infrastructure that leverages HKP as that protocol will automatically work with Apptainer. We would be able to immediately start pulling and pushing keys. This is why we're using a very common public key service to do this. It's very easy to integrate this with any kind of key server.
CI Pipeline and Building and Signing Containers [00:50:53]
Zane Hamilton:
So that was the first one. Next one is it possible to build and sign containers from a CI pipeline?
Gregory Kurtzer:
Another great one? I'll take it just because I'm on a roll at this point, but I do fully expect Jonathan, Forrest, and Zane to add if I mess anything up or miss anything. So yes, absolutely. obviously there's some integration that you would have to do into your CI pipeline with regards to secrets. Unless your private key is not encrypted.I don't recommend this. Especially putting it up into a CI pipeline infrastructure. It'll probably have some secret associated with that and you're going to have to leverage your CI pipeline infrastructure to properly manage that secret and the key material. It is possible if done properly. In a DevSecOps pipeline you'd have a GIT operation that causes a CI to go off and build your container.
Then you have multiple pieces of this pipeline or multiple processes in this pipeline to do things like not only build, but now actually run various security compliance validations against it. You can do container scanning against it. You can do all these different pieces. You can be signing them every step of the way so that the resulting container is absolutely guaranteed to be trusted. You can host that once it's been signed, you could literally host that anywhere. People are actually hosting this within registries that are associated with a CI pipeline. Various CI infrastructures are offering OCI based registry support. Apptainer works directly with these various registries that are supporting ORAS. ORAS is an OCI registry as storage.
If I remember correctly, what ORAS stands for, it allows us to store and persist the Singularity blob directly. And that was the mechanism that we actually demonstrated just a moment ago with Jonathan sharing that container to Forrest. There are a lot of ways you can integrate this into a CI pipeline. This is definitely the trend that we're seeing from most organizations in terms of building their containers and integrating those containers with HPC. They do that in an automated fashion so that their containers are always up to date. When they integrate them into the HPC system they can have either automated ways of doing that. If we're talking Fuzzball, it will automatically pull from a particular tag.
If you're building from a CI pipeline, as something like latest, tag is latest, or the tag is production, you can always ensure that you're pulling that version of that container in an automated way. The infrastructure manages all of that. In an HPC system it's one extra stepIt's actually CIMD. It gives you that ability to automatically be building those containers. Signing and trusting those containers. Then sharing appropriately with your infrastructure.
Allowing Containers to Trust Signatures [00:54:38]
Zane Hamilton:
Very nice. I think we just had another question that popped in. Is it possible to lock down execution to only allow containers that have trusted signatures to run? I think Jonathan, you touched on this a little bit earlier, but I would rather you answer it again.
Jonathon Anderson:
Yeah. So this is the execution control list function. It's just a config file that exists globally for the system. It's not a per user setting. Within the general etc Apptainer directory structure, there's an ECL. “something” file that allows you to specify one to many blocks of configuration for containers under a certain path. You can give either a list of signatures or a list of fingerprints for keys that must have been signed or may not be signed. So you can block containers with certain signatures. You can require that all of the signatures in a list are present on a container. You can also give a list of signatures, any of which would allow it to run. Then you can lock that down for only verifying containers that are in slash temp or only verify containers that are in certain directory structures and require that your containers be in certain places as well.
Zane Hamilton:
Very cool. Thank you. So we're actually running up against time here, guys. So I'm going to wrap up, but I'm going to leave it open to you guys. If you have any closing thoughts, you want to add, Greg, you have to go last. So I'm going to start with Forrest.
Forrest Burt:
No, nothing too much for me. We have that picture of Jonathan as the super computing mascot.
Zane Hamilton:
oh, nice.
Forrest Burt:
Throw that up there, throw up there, but I'm not sure if we want to, but I thought that was kind of funny. That does actually look like Jonathan to me.
Zane Hamilton:
Throw it up.
Forrest Burt:
Oh, sorry. I need to probably share this. My apologies. It's right here. If I can find where I put my streamy yard in among all these links.
Zane Hamilton:
Too many tabs, not enough time.
Forrest Burt:
I think that's
Jonathon Anderson:
Jonathan. So this is me many years ago.
Zane Hamilton:
That's awesome.
Jonathon Anderson:
There's a lot of story behind that, but I'll leave that for another day.
Zane Hamilton:
That's awesome.
Forrest Burt:
That's cool.
Zane Hamilton:
All right, Jonathan, do you have anything to close with here?
Jonathon Anderson:
No. First time on here, really happy to be with the team and glad to be finding ways to contribute and work with all of these Apptainer and more and more fuzzball. That's all. Very cool.
Zane Hamilton:
Thank you for joining. Thank you for putting the demos together too. Appreciate
Jonathon Anderson:
It. Absolutely.
Zane Hamilton:
Greg.
Gregory Kurtzer:
I like how you saved me for last. Is that because I talk so much? I'm just going to.
Jonathon Anderson:
Cause we're hiring.
Gregory Kurtzer:
So a couple points. Yes we are. We are hiring and we are very happy to have Jonathan and Forrest with us. Forrest has been with us for a little while, and the team is growing. So if you know, if this stuff interests you and you want to be part of it Definitely reach out to us, let us know. The other thing I'm going to make a comment on is the amount of times that we call it, called Apptainers singularity. And this has to do with the fact that it started life as a singularity. I created singularity back when I was at the department of energy at Lawrence Berkeley lab and joined an appointment to UC Berkeley. I have to make sure I give all credit where credit is due to my employers. And singularity came to be over there.
We prototyped it and we brought it out to the community. It's been a fantastic tool for a really long time. There's been corporate involvement with regards to Singularity. Moving forward we want to make sure that the project stays in the community. This is why Singularity has been moved into the Linux foundation. As part of that move, the Linux foundation asked if we could rename the project to make sure that they have the ability to protect the project, the name, and the branding of that project. Singularity is a term used all over the scientific community.
The community renamed it Apptainer. It was chosen because it is an application container system. It is hard for me to stop calling it Singularity because of my many years calling it that. It is taking on a new life as Apptainer. It has been an easy transition. There's another webinar that we did specifically on the move to the Linux Foundation. Having the Linux foundation run that project is a massive benefit to the community. That is all I wanted to say.
Zane Hamilton:
No, that's great. And part of the Linux foundation, it's actually getting more exposure and exposed to other things and allowing for more integration. So it's just growing faster and faster and having more people interested in it. So thank you for sharing that, Craig again, we're hiring. Check out the site. If you're interested in any of this stuff you can subscribe so you can stay in touch with us and if you ever need any help or you ever want to talk to us, just reach out on the site. We'd be happy to schedule a consultation with you and talk through your needs. What you like, what you don't like and just thank you for joining us and we will see you next week. Thank you. Bye
Gregory Kurtzer:
Everyone. Thanks, bye.