Managing Containerized Services with Apptainer and Systemd
Apptainer is often compared to Docker as both a container runtime engine and container management environment. In general, each tool targets a different use case; but the two projects notably differ in their runtime strategy. In general, Docker is designed to host system services, and Apptainer is designed to containerize end-user applications (often in a batch-processing environment). Because of its service-oriented architecture, Docker containers are most commonly started in the background and run indefinitely. Apptainer has support for service containers, but Apptainer containers are most commonly started in the foreground for temporal, interactive use.
Apptainer’s architecture, however, is ultimately a more natural fit for the Linux ecosystem, as it comes with fewer embedded assumptions and is, ultimately, more flexible.
In the time since Docker’s architecture was first established, systemd has arisen as the de facto standard for service management in Linux. As such, we can easily use systemd to manage system services containerized with Apptainer. This technique should be compatible with most major Linux operating systems, including Rocky Linux, CentOS Stream, and Suse.
A minimal service container
First, here’s a an example definition file to use as a system service:
Bootstrap: docker From: alpine:latest
%runscript env; while true; do date; sleep 10; done
This is just a minimal Alpine Linux container that prints the current environment and then loops, printing the current date and time every 10 seconds. This suffices as an example system service that will run “forever” and that produces regular output, allowing us to confirm that the service is running properly.
I placed the example container at /usr/local/lib/apptainer/env-service.sif. (We’ll refer to that path later.)
$ apptainer build alpine-sleeper.sif alpine-sleeper.def INFO: Starting build... Getting image source signatures Copying blob f56be85fc22e skipped: already exists Copying config 4798f93a2c done Writing manifest to image destination Storing signatures 2023/04/03 16:25:13 info unpack layer: sha256: f56be85fc22e46face30e2c3de3f7fe7c15f8fd7c4e5add29d7f64b87abdaa09 INFO: Adding runscript INFO: Creating SIF file... INFO: Build complete: alpine-sleeper.sif
$ sudo mkdir -p /usr/local/lib/apptainer/ $ sudo mv alpine-sleeper.sif /usr/local/lib/apptainer/
systemd unit file
I then installed the following systemd unit file at /etc/systemd/system/apptainer@.service:
[Unit] Description=Apptainer container: %I After=network-online.target Wants=network-online.target
[Service] ExecStart=/usr/bin/apptainer run /usr/local/lib/apptainer/%i.sif EnvironmentFile=-/etc/sysconfig/apptainer/%i.env EnvironmentFile=-/etc/default/apptainer/%i.env
[Install] WantedBy=multi-user.target
This unit file is a template that can start an Apptainer container from an image stored at /usr/local/lib/apptainer/.
The unit file will also optionally read environment configuration files from either /etc/sysconfig/apptainer/ or /etc/default/apptainer/.
The first time you put the unit file in place you’ll need to tell systemd to reload its configuration to make it available. (This is otherwise automatic on boot.)
systemctl daemon-reload
Configuring and starting the service
To demonstrate the configurability of the service environment, I placed an environment file at /etc/sysconfig/apptainer/alpine- sleeper.env.
TEST_VARIABLE=nar_value
You can now enable the service to run automatically at boot and/or start it immediately, as with any systemd service:
$ sudo systemctl start apptainer@alpine-sleeper $ sudo systemctl status apptainer@alpine-sleeper apptainer@alpine-sleeper.service - Apptainer container: alpine/sleeper Loaded: loaded (/etc/systemd/system/apptainer@.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2023-04-04 11:46:31 MDT; 1s ago Main PID: 107209 (starter) Tasks: 9 (limit: 11364) Memory: 11.3M CGroup: /system.slice/system-apptainer.slice/apptainer@alpine- sleeper.service 107209 Apptainer runtime parent 107224 /bin/sh /.singularity.d/runscript 107249 sleep 10
Apr 04 11:46:31 rocky3 apptainer[107247]: LANG=en_US.UTF-8 Apr 04 11:46:31 rocky3 apptainer[107247]: PROMPT_COMMAND=PS1=" Apptainer> "; unset PROMPT_COMMAND Apr 04 11:46:31 rocky3 apptainer[107247]: APPTAINER_BIND= Apr 04 11:46:31 rocky3 apptainer[107247]: APPTAINER_CONTAINER=/usr/local /lib/apptainer/alpine-sleeper.sif Apr 04 11:46:31 rocky3 apptainer[107247]: SINGULARITY_BIND= Apr 04 11:46:31 rocky3 apptainer[107247]: SINGULARITY_CONTAINER=/usr /local/lib/apptainer/alpine-sleeper.sif Apr 04 11:46:31 rocky3 apptainer[107247]: APPTAINER_NAME=alpine-sleeper. sif Apr 04 11:46:31 rocky3 apptainer[107247]: PWD=/ Apr 04 11:46:31 rocky3 apptainer[107247]: SINGULARITY_NAME=alpine- sleeper.sif Apr 04 11:46:31 rocky3 apptainer[107248]: Tue Apr 4 11:46:31 MDT 2023
The string to the right of the @ populates the %i and %I variables inside the unit file template, allowing you to manage any such container with the single unit file, each with a discrete set of input variables.
Further, the specified environment variables are passed through to the process inside the container.
$ journalctl -u apptainer@alpine-sleeper | grep TEST_VARIABLE
Apr 04 11:48:18 rocky3 apptainer[107375]: TEST_VARIABLE=nar_value
Monitoring the service
Recent logs for the service are available in the systemctl status output, and you can retrieve full logs using journalctl. e.g.,
journalctl -u apptainer@env-service.service
Because systemd typically runs each service in its own cgroup automatically, we even get cgroup-based process isolation for free with this technique. See the above systemctl status output, which reports the number of tasks, the total CPU time, and even the amount of memory consumed by the service and, by extension, the container.
Finally, process management works as you expect. I can stop the service, and the container stops as well.
$ pgrep -lif apptainer 107574 starter
$ sudo systemctl stop apptainer@alpine-sleeper $ pgrep -lif apptainer || echo not running not running
And if the process dies, the service reflects this.
$ sudo systemctl start apptainer@alpine-sleeper $ sudo systemctl status apptainer@alpine-sleeper apptainer@alpine-sleeper.service - Apptainer container: alpine/sleeper Loaded: loaded (/etc/systemd/system/apptainer@.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2023-04-04 11:53:27 MDT; 10s ago Main PID: 107637 (starter) Tasks: 10 (limit: 11364) Memory: 11.8M CGroup: /system.slice/system-apptainer.slice/apptainer@alpine- sleeper.service 107637 Apptainer runtime parent 107651 /bin/sh /.singularity.d/runscript 107682 sleep 10
Apr 04 11:53:27 rocky3 apptainer[107674]: PROMPT_COMMAND=PS1=" Apptainer> "; unset PROMPT_COMMAND Apr 04 11:53:27 rocky3 apptainer[107674]: APPTAINER_BIND= Apr 04 11:53:27 rocky3 apptainer[107674]: APPTAINER_CONTAINER=/usr/local /lib/apptainer/alpine-sleeper.sif Apr 04 11:53:27 rocky3 apptainer[107674]: SINGULARITY_BIND= Apr 04 11:53:27 rocky3 apptainer[107674]: SINGULARITY_CONTAINER=/usr /local/lib/apptainer/alpine-sleeper.sif Apr 04 11:53:27 rocky3 apptainer[107674]: APPTAINER_NAME=alpine-sleeper. sif Apr 04 11:53:27 rocky3 apptainer[107674]: PWD=/ Apr 04 11:53:27 rocky3 apptainer[107674]: SINGULARITY_NAME=alpine- sleeper.sif Apr 04 11:53:27 rocky3 apptainer[107675]: Tue Apr 4 11:53:27 MDT 2023 Apr 04 11:53:37 rocky3 apptainer[107681]: Tue Apr 4 11:53:37 MDT 2023
$ sudo kill 107651 $ sudo systemctl status apptainer@alpine-sleeper apptainer@alpine-sleeper.service - Apptainer container: alpine/sleeper Loaded: loaded (/etc/systemd/system/apptainer@.service; disabled; vendor preset: disabled) Active: inactive (dead)
Apr 04 11:53:27 rocky3 apptainer[107674]: APPTAINER_CONTAINER=/usr/local /lib/apptainer/alpine-sleeper.sif Apr 04 11:53:27 rocky3 apptainer[107674]: SINGULARITY_BIND= Apr 04 11:53:27 rocky3 apptainer[107674]: SINGULARITY_CONTAINER=/usr /local/lib/apptainer/alpine-sleeper.sif Apr 04 11:53:27 rocky3 apptainer[107674]: APPTAINER_NAME=alpine-sleeper. sif Apr 04 11:53:27 rocky3 apptainer[107674]: PWD=/ Apr 04 11:53:27 rocky3 apptainer[107674]: SINGULARITY_NAME=alpine- sleeper.sif Apr 04 11:53:27 rocky3 apptainer[107675]: Tue Apr 4 11:53:27 MDT 2023 Apr 04 11:53:37 rocky3 apptainer[107681]: Tue Apr 4 11:53:37 MDT 2023 Apr 04 11:53:47 rocky3 apptainer[107687]: Tue Apr 4 11:53:47 MDT 2023 Apr 04 11:53:48 rocky3 systemd[1]: apptainer@alpine-sleeper.service: Succeeded.
You could even configure systemd to automatically restart the service, e.g., by specifying Restart=always.
Conclusion
Using systemd for Apptainer process management is different than Docker, but ultimately integrates with other traditional system services more seamlessly. Notably, Podman—a OCI-based Docker alternative included with Rocky Linux and many other Linux distributions—has first-class support for generating such systemd unit files as well.
You can read more about the configurability of systemd service unit files at https://www.freedesktop.org/software/systemd/man/systemd.service.html. Since this technique turns Apptainer containers into standard systemd services, you have access to all the service and process management capabilities of systemd, including declarative dependency management, process monitoring, and output capture.
