Introduction to Linux Containers for HPC
Containers have gained massive popularity in High Performance Computing (HPC) environments as users and administrators realize their essential value. To explain the ‘how’ and ‘why’ containers are such a good choice for research data management, CIQ’s Dave Godlove provided an introductory overview in a recent webinar for the Campus Research Computing Consortium (CaRCC).
You can watch the entire video here. Keep reading for the highlights!
What are Linux containers?
Linux containers allow you to package up applications along with their dependencies into lightweight, portable, and reproducible units. This gives you access to operating systems and software that you don't have installed locally. For example, you can instantly get a Python environment in a container even if Python isn't installed on your system. Containers provide a lot of flexibility for HPC users.
What is Apptainer?
Apptainer (called Singularity until it joined the Linux Foundation) is a Linux container platform purpose-built for High Performance Computing. It was created to provide a secure container solution tailored for multi-tenant HPC environments. Apptainer enables unprivileged users to safely run containers without escalated privileges, overcoming the limitations of earlier solutions like Docker. It also uses image files for portability and security features like cryptographic signing.
How can HPC users benefit from containers?
HPC users can leverage containers to easily access difficult-to-install HPC applications without going through complex manual installation steps. Containers also allow HPC users to take advantage of specialized hardware like GPUs and interconnects using customized software stacks. For example, software like AlphaFold expects users to utilize containers for complex installations.
How can HPC admins benefit from containers?
For HPC administrators, containers reduce the workload of installing and managing user applications. Admins can install apps in containers, then load them in environments like Lmod for users transparently, without users needing any container knowledge. Containers give admins an efficient way to deliver applications to users that can encapsulate all dependencies.
Container orchestration with Fuzzball
While traditional container orchestration tools like Kubernetes are great at what they do, they are specialized for services. In HPC, we need to orchestrate jobs. Fuzzball is an open source tool from CIQ, built for orchestrating HPC job workflows using containers. It provides a simple YAML interface to launch MPI and other jobs across clusters. This makes it easy to run multi-step computational pipelines on HPC infrastructure.
Learn more
Be sure to watch the entire video, which covers several other interesting topics, such as abstractions.
Containers and orchestrators like Apptainer and Fuzzball offer many benefits for HPC users and administrators. If you want to learn more about applying these technologies to make HPC easier and more efficient, check out the following resources:
