Apptainer and Public Cloud Container/Artifact Registry Offerings
As a follow-up to CIQ’s “Cloud Day with Apptainer” webinar, here’s more information about how to utilize Apptainers with the container/artifact registry offerings around the public cloud space. Apptainer uses the Singularity Image Format (SIF) for its containers, which are supported in many container/artifacts registries through the use of the “OCI Registries As Storage” (ORAS) container scheme, which allows for artifacts like Apptainer SIFs, Helm charts, and other commonly used computing infrastructure artifacts to be stored alongside OCI (or Docker, essentially) images in a single place.
Apptainer SIFs can be stored in the GCP Artifact Registry, the AWS Elastic Container Registry, the Azure Container Registry, on the Docker Hub itself, within the Oracle Cloud Infrastructure container registry, and in GitLab repository container registries. In this article, we will show some basic instructions on how to complete authentication to and pushing/pulling of SIFs from each of these places.
GCP
First, we’ll look at GCP. The GCP Artifact Registry can store SIFs, so make sure you have a test registry to work with.
To get the required credential to access the registry, you will either need to configure the GCP CLI so you can get a temporary credential with gcloud auth print-access-token, or you will need to have a service account set up with read access to your registry. Information from GCP on how to set these methods up can be found at https://cloud.google.com/sdk/docs/authorizing.
Once you have your credential, you'll execute:
apptainer remote login -u <credential-specific username>
oras://url.of.GCP.registry
This will then prompt you for the credential you created, so paste it in and you should be good to go to login. Once you've got the message indicating success, containers can be pushed and pulled from the registry just as expected with commands like:
apptainer push container.sif
oras://url.of.GCP.registry/apptainer_demo:latest
and
apptainer pull oras://url.of.GCP.registry/apptainer_demo:stable
The GCP registry keeps containers as overall entries in the registry, so you should be able to see it show up with the tag you pushed it with in the registry, and selecting a given container from the list will give you all the tags applied to it.
AWS
The AWS ECR can also store SIFs. In the ECR, each created registry points to a single container and its tagged versions, rather than a collection of different containers and all their collective tags, so you’ll need to create a different ECR registry for each different SIF you want to upload.
To log into the AWS ECR, you will first need the AWS CLI configured on your machine. Take a look at the instructions at this link to get that installed and configured with security credentials from your account: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html. Once you can login successfully from the AWS command line, create an ECR registry and view the push commands via the button in the top right of the ECR interface, which by default should give you the example pull command for Docker:
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin
526927152917.dkr.ecr.us-west-2.amazonaws.com
Modifying this slightly to do the same for Apptainer, we’ll run:
aws ecr get-login-password --region us-west-2 | apptainer remote login --username AWS --password-stdin oras://526927152917.dkr.ecr.us-west-2.amazonaws.com
This will generate an AWS ECR credential, and feed it into Apptainer through stdin for authentication. Once this is done, we can push and pull containers from the registry with:
apptainer push container.sif
oras://526927152917.dkr.ecr.us-west-2.amazonaws.com/apptainer_demo:latest
and
apptainer pull oras://526927152917.dkr.ecr.us-west-2.amazonaws.com/apptainer_demo:stable
Different tags will appear as different entries in a given container repository for that SIF.
Azure
Azure’s container registry can be used to store SIFs. This allows you to upload multiple containers to a single registry, with support for multiple tags on each container.
To log into an Azure container registry, you’ll need to install the Azure CLI with the instructions found here: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli. Once you’ve done that and logged into your user account with az login, you can log into your container registry with:
az acr login --name azureapptainerdemo.azurecr.io --expose-token
Which will expose the token onto the command line. Copy out the portion inside the quotes in the “accessToken” field, which we’ll then need to paste into the following to complete the login:
apptainer remote login --username 00000000-0000-0000-0000-000000000000 oras://yourcontainerregistry.azurecr.io
Once this is done, use the following commands to push/pull from the registry:
apptainer push ./container.sif
oras://yourcontainerregistry.azurecr.io/apptainer_demo:latest
and
apptainer pull
oras://yourcontainerregistry.azurecr.io/apptainer_demo:stable
Docker Hub/OCI Tools
As we discuss the commands to authenticate and push/pull SIF files to and from OCI registries, it might be good to take a step back and ask “why?”. For years, a popular model of Apptainer usage was to build your containers using Dockerfiles, push them to registries in OCI format, and convert them to SIF files when you are ready to run the containers in your HPC environment. If you are using both Apptainer and OCI tools (like Docker or Podman) to run your containers, this model is still useful. But with the advent of the ORAS protocol, it now makes sense to build your containers with Apptainer if you are going to run them exclusively with Apptainer, even if you plan to store them in an OCI registry.
There are several advantages to working with SIF images in Apptainer:
-
You can use a
signandverifyworkflow so that you (and others) can ensure that the SIF file downloaded from an OCI registry is a bit-for-bit reproduction of the original SIF. -
You can use the
fingerprintskeyword in a def file that builds from an ORAS image to ensure that the base image contains exactly what you think it should. -
You can use the
inspect --deffilecommand to see the definition file that was used to produce the SIF file. -
You can encrypt the image if it contains secrets and store it in a public OCI registry without fear of others accessing the data.
To push a SIF container to Docker Hub using ORAS, you first need to sign in. We recommend using a personal access token that you revoke after pushing your image, because the secret is stored in plain text in your ~/.apptainer directory. Once you have signed into Docker Hub you can create a personal access token here. After copying the token, the procedure to sign in looks like this. (Note that the token characters are not echoed to the terminal when you paste them at the prompt.)
```
$ apptainer remote login --username dh-user oras://docker.io
Password / Token:
INFO: Token stored in /home/user/.apptainer/remote.yaml
```
If you need to create a new repository for your SIF, you can do so (after logging in) here.
Now you are ready to push a SIF container to an existing Docker Hub repository. The command to do so follows. (Hint: It’s a good idea to create a key and sign the container first.)
```
$ apptainer push container.sif oras://docker.io/dh-user/container:tag
```
I’ll use a Rocky Linux image that I’ve pushed to Docker Hub to illustrate the various advantages of pushing native SIF images to an OCI registry.
First, when I pull the image, I can verify that this SIF file is a bit-for-bit reproduction of the original container I created (and you can too!).
```
$ apptainer pull oras://docker.io/godlovedc/rockylinux:9.2
INFO: Downloading oras image
$ apptainer verify rockylinux_9.2.sif
WARNING: No default remote in use, falling back to default keyserver: https://keys.openpgp.org
Verifying image: rockylinux_9.2.sif
[REMOTE] Signing entity: David Godlove (production key) <davidgodlove@gmail.com>
[REMOTE] Fingerprint: B7761495F83E6BF7686CA5F0C1A7D02200787921
Objects verified:
ID |GROUP |LINK |TYPE
------------------------------------------------
1 |1 |NONE |Def.FILE
2 |1 |NONE |JSON.Generic
3 |1 |NONE |FS
Container verified: rockylinux_9.2.sif
```
Because I know that the key fingerprint should be B7761495F83E6BF7686CA5F0C1A7D02200787921, I can trust that this image is exactly as I built it.
Second, if I wanted to use this as a base image for a new container, but I wanted to make sure that the base was clean and free of any unwanted changes, I could create a definition file like so:
```
Bootstrap: oras
From: docker.io/godlovedc/rockylinux:9.2
Fingerprints: B7761495F83E6BF7686CA5F0C1A7D02200787921
%post
echo “install stuff here”
```
Third, if I wanted to see how the Rocky Linux container was originally created, I can do so because the container was saved in SIF format.
```
$ apptainer inspect --deffile rockylinux_9.2.sif
BootStrap: yum
OSVersion: 9
MirrorURL: http://dl.rockylinux.org/pub/rocky/%{OSVERSION}/BaseOS/x86_64/os/
Include: dnf
%labels
Author: davidgodlove@gmail.com
%post
dnf -y update
dnf install -y epel-release file
%environment
LC_ALL=C
```
And finally, you can take advantage of the encrypted container workflow to protect your container contents as it is stored on Docker Hub and while it is in transit to and from your machine (and even while it is running).
Note: Remember to delete your personal access token on Docker Hub after you finish pushing your container. After doing so you will probably receive a 404 unauthorized error if you try to interact with Docker Hub. Unfortunately, you will need to delete ~/.apptainer/docker-config.json to clear your deleted token from the local Apptainer cache and use Docker Hub again.
Oracle
The Oracle Cloud infrastructure container registry can also store SIFs.
To log into the registry, go to your user account page and generate an access token. Your username for the registry login will be the text at the top of your account, so perhaps something like oracleidentitycloudservice/
After generating your token, run:
apptainer remote login --username <namespace>/<auth provider and email string> oras://sjc.ocir.io
So for example,
apptainer remote login --username deqy7khuydfw/oracleidentitycloudservice/test@ciq.co oras://sjc.ocir.io
For a namespace of deqy7khuydfw, a provider/email pair of oracleidentitycloudservice/test@ciq.com, and in the San Jose (sjc) region. After this, you can push/pull with:
apptainer push container.sif
oras://sjc.ocir.io/deqy7khuydfw/apptainer_demo:latest
and
apptainer pull oras://sjc.ocir.io/deqy7khuydfw/apptainer_demo:stable
GitLab
Finally, GitLab's container registry that they make available to repositories can also store SIFs. These will be listed in the repo’s registry alongside other containers that have been uploaded, with tags available under each container entry.
To log into the GitLab container registry, you should generate a GitLab personal access token for your account and that repo with the read/write registry permission depending upon your needs. Copy this credential to somewhere safe, as you’ll need to be able to paste it into the command:
apptainer remote login --username oauth2accesstoken
oras://registry.gitlab.com/rest/of/url/from/registry
Once you’re logged in, you can push and pull from a given repo’s registry with the commands:
apptainer push container.sif oras://registry.gitlab.com/test/gitlabcontainers/apptainer_demo:latest
and
apptainer pull oras://registry.gitlab.com/test/gitlabcontainers/apptainer_demo:stable
As can be seen, Apptainer SIFs can be easily used with many of the major popular cloud providers. Keep an eye out for further articles about modern use cases with Apptainer from CIQ, and if you have any questions, please reach out to us at info@ciq.com.
