What to Expect When Updating from SingularityCE to Apptainer
It has been awhile since Singularity became Apptainer. If you haven't updated Singularity to Apptainer, or if you accidentally switched to SingularityCE and now want to switch back, this post is for you!
How Singularity became Apptainer
In May 2021, a private company wanted to develop their own version of Singularity autonomously. It created a fork of Singularity, which it named SingularityCE. However, the program itself and the repository continued to be named "Singularity." This led to confusion.
In November 2021, the Singularity community officially moved the project into the Linux Foundation to guarantee that it will always be free and open source software (FOSS). You can find more information about this move here. To avoid confusion, the name Singularity was changed to Apptainer during the transition. Hence, the FOSS project that we’ve grown to know and love as Singularity is now Apptainer.
Besides the name, what else changed?
Initially, very little changed besides the name itself. The container format remains the same. Environment variables with the prefix SINGULARITY* were updated to APPTAINER*, (although both are accepted) and the user’s local configuration directory changed from $HOME/.singularity to $HOME/.apptainer. Changes were made to the built-in support for container libraries, keystores, and the remote builder depending on whether each API met the open source requirement set by the Linux Foundation.
Since the move to the Linux Foundation, the development of Apptainer and SingularityCE has diverged to some extent. However, for typical end-users, the two projects will still look and feel almost identical. The biggest change is that Apptainer now utilizes a non-privileged installation as the default.
We’ll discuss these differences in more detail below, but first we’ll talk about the process of updating.
How to update to Apptainer
The process of updating from Singularity or SingularityCE to Apptainer will vary depending on how you initially installed Singularity{CE} and how you plan to install Apptainer. As an example, let's consider a scenario where SingularityCE was installed using the package manager on an rpm-based system (Rocky Linux 9.1).
For this scenario, we assume that the administrator has already installed the extra packages for enterprise Linux (EPEL) repositories and has used dnf or yum to install SingularityCE.
$ dnf provides singularity
Last metadata expiration check: 0:05:14 ago on Thu 04 May 2023 09:31:00 AM MDT.
apptainer-1.1.7-1.el9.x86_64 : Application and environment virtualization formerly known as Singularity
Repo : epel
Matched from:
Filename : /usr/bin/singularity
singularity-ce-3.11.1-1.el9.x86_64 : Application and environment virtualization
Repo : @System
Matched from:
Filename : /usr/bin/singularity
singularity-ce-3.11.1-1.el9.x86_64 : Application and environment virtualization
Repo : epel
Matched from:
Filename : /usr/bin/singularity
Notice the dnf provides command shows that the apptainer package provides the singularity command. It does so through the magic of symlnks. This command also shows that singularity-ce has been installed (and is hence listed in the @system repo).
In this case, we can update pretty easily using dnf.
$ sudo dnf install apptainer
Last metadata expiration check: 0:07:41 ago on Thu 04 May 2023 10:03:47 AM MDT.
Error:
Problem: problem with installed package singularity-ce-3.11.1-1.el9.x86_64
- package singularity-ce-3.11.1-1.el9.x86_64 conflicts with sif-runtime provided by apptainer-1.1.7-1.el9.x86_64
- package apptainer-1.1.7-1.el9.x86_64 conflicts with sif-runtime provided by singularity-ce-3.11.1-1.el9.x86_64
- conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Whoops! An error! No problem though. In this case, the --allowerasing option is appropriate since we want to replace the SingularityCE installation with Apptainer.
$ sudo dnf install --allowerasing apptainer
Last metadata expiration check: 0:09:47 ago on Thu 04 May 2023 10:03:47 AM MDT.
Dependencies resolved.
===========================================================================================
Package Architecture Version Repository Size
===========================================================================================
Installing:
apptainer x86_64 1.1.7-1.el9 epel 23 M
Installing dependencies:
fakeroot x86_64 1.31-1.el9 epel 87 k
fakeroot-libs x86_64 1.31-1.el9 epel 39 k
fuse-overlayfs x86_64 1.9-1.el9 appstream 71 k
squashfuse x86_64 0.1.104-1.el9 epel 25 k
squashfuse-libs x86_64 0.1.104-1.el9 epel 26 k
Removing dependent packages:
singularity-ce x86_64 3.11.1-1.el9 @epel 133 M
Transaction Summary
===========================================================================================
Install 6 Packages
Remove 1 Package
Total download size: 24 M
Is this ok [y/N]: y
[...snip]
$ ls -l $(which singularity)
lrwxrwxrwx. 1 root root 9 Mar 29 06:37 /usr/bin/singularity -> apptainer
Note that the singularity command is now symlinked to apptainer.
If you installed SingularityCE from source, the process of updating to Apptainer is a bit more complicated. In this case, you will need to manually remove SingularityCE. If you used the --prefix flag with the mconfig command during installation, it will be easier since the entire application will be localized to a single directory. However, if you did not set a prefix directory, you will need to locate bits and pieces of SingularityCE that are spread out across /usr/local.
HINT: watching the standard output of the make install command can give you a good idea of where to look for all the bits and pieces.
Changes to be aware of after updating
Upon installation, Apptainer will not automatically transfer settings from singularity.conf to apptainer.conf. Therefore, it is recommended that you check for custom settings that need to be manually transferred. In addition, there are a few runtime changes that may impact you or your users.
But first… things that have not changed
Things that have not changed include pretty much everything. Apptainer’s look and feel continues to be almost identical from the old days when it was still called Singularity. Importantly, the Singularity Image File (SIF) format remains the same. So you can still use your existing containers and create new ones without any concerns about compatibility with Apptainer.
The default Apptainer installation is now unprivileged
In the past, Singularity needed a set UID (suid) binary to elevate privileges for certain operations during container setup. This is no longer necessary since Apptainer can use tools like the user namespace and squashfuse to perform all necessary operations without relying on suid privilege escalation.
There is still a version of Apptainer that uses the old suid workflow. If you want to build this version from source, you can pass the --with-suid option to mconfig. Alternatively, if you prefer to install it via rpm, look for the apptainer-suid package in EPEL. As above, you will need to add the --allowerasing flag when updating to apptainer-suid via dnf.
Changes in environment variables and config directory
The environment variables prefixed with SINGULARITY* have been replaced with APPTAINER* variables. Apptainer recognizes both variants, but the old SINGULARITY* variables will trigger a deprecation warning. If both SINGULARITY* and APPTAINER* variables are set for the same option, Apptainer will complain and use the APPTAINER* version.
When you first execute the apptainer or singularity command, a configuration directory is created in the user's home directory. Previously, this directory was named $HOME/.singularity, but now it is $HOME/.apptainer. If the $HOME/.singularity directory exists when you first run the apptainer command, Apptainer will try to move essential data from that directory to $HOME/.apptainer. For more information on these topics, please refer to the Apptainer documentation on Singularity compatibility.
Container library and keyserver configuration
During Apptainer's move to the Linux Foundation, developers were required to eliminate any code or configuration that referenced a private company. Previously, Singularity's default configuration files pointed to a private container library, which is a SIF native registry, when commands such as build, pull, and push were executed with the library:// URI. Unfortunately, this is the only native SIF registry that currently exists. Initially, there was a suggestion to remove the code managing SIF libraries, but it was ultimately retained because the library API is open source. To comply with the requirement not to reference a privately hosted library, Apptainer's default configuration now points to a placeholder library that does not exist.
For similar reasons, Apptainer now defaults to keys.opnpgp.org instead of the previous keystore. However, keys.opnpgp.org works differently requiring a new key be verified via email before allowing others to download and use it.
If you need to restore the default library and/or keystore in the local or global configuration files, you can find information in this section of the Apptainer documentation.
It is worth noting that the need for a SIF native container registry has been rendered largely obsolete by the widely used OCI Registry As Storage (ORAS) protocol.
The remote builder is deprecated
When Apptainer was moved to the Linux Foundation, default references to the remote builder were removed, similar to how the container library and keystore configuration were handled. However, unlike those services, the API for the remote builder is not open-source. Therefore, the code that supported the remote builder from within Apptainer was also eliminated, and it’s no longer possible to build containers using the --remote flag.
Again, it should be noted that this feature has become obsolete for most users since it's no longer necessary to use elevated privileges to build containers.
Closing thoughts
Apptainer continues the great tradition it started as Singularity, giving users the power to run what they want where they want. It’s exciting to see Apptainer evolve and mature, with the addition of new features such as unprivileged installation. If you have any queries or feedback about upgrading to Apptainer, please don't hesitate to contact us.
