OpenFOAM Supports Apptainer v1.2.0 Template Definition File Feature
Introduction
We are happy to announce that OpenFOAM® (OpenCFD Ltd.) is the very first project that officially supports the brand-new Apptainer v1.2.0 feature “template definition file” just released on July 18, 2023. OpenFOAM officially supports Apptainer since OpenFOAM v2212 and since then we have closely worked together on the OpenFOAM container repository.
In this blog post, we will explain (1) Apptainer “template definition file" basics, (2) how to use OpenFOAM's official Apptainer template definition file to build your own brand-new OpenFOAM v2306 container, (3) how to create your own OpenFOAM development environment based on the official template, and (4) finally, we will show you how to share your own OpenFOAM development environment with your collaborators using Docker Hub. Apptainer helps drive researchers,' developers,' and engineers' collaboration by providing “Mobility of Compute.”
Apptainer Template Definition File Basics
The new “template definition file” feature introduces template variables to every section of the Apptainer definition file. Before we introduced these template variables, environment variables were used for similar purposes and still can be used, but it only works for limited sections of the Apptainer definition file (e.g., environment variables can not be used for bootstrap OS image names).
Template variables
With template variables, for example, we can switch bootstrap OS image versions like the following:
Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}
%arguments section
To support default values for template variables, an %arguments section is added to the Apptainer definition file. Let’s make a file called rockylinux-template.def with the following contents.
Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}
%arguments
OS_VER=8.8
With a default value, you can build this definition file as usual:
apptainer build rockylinux8.sif rockylinux-template.def
Now a Rocky Linux 8.8 container called rockylinux8.sif is built.
./rockylinux8.sif cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.8 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.8 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"
--build-arg option
The new option is added to apptainer build command, --build-arg, and --build-arg-file. When you want to bump up the OS version to 9.2, here is the time to use that new option --build-arg.
apptainer build --build-arg OS_VER=9.2 rockylinux9.sif rockylinux-template.def
This time, a Rocky Linux 9.2 container called rockylinux9.sif is built.
./rockylinux9.sif cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
--build-arg-file option
Create the file called args.list(or whatever name and extension):
cat << EOF > args.list
OS_VER=9.2
EOF
You can build the same Apptainer image as above with --build-arg-file option. The file-based approach is convenient when you want to keep the build command the same but change the build settings by just changing the arguments list (e.g., CI/CD pipelines) and/or when you want to share the build settings.
apptainer build --build-arg-file args.list rockylinux9.2.sif rockylinux-template.def
Again, a Rocky Linux 9.2 container called rockylinux9.2.sif is built. Let’s move on to the hands-on guide to OpenFOAM official Apptainer definition file 🙂.
Hands-on Guide to OpenFOAM Official Apptainer Definition File
The OpenFOAM official container repository is here https://develop.openfoam.com/packaging/containers .
Clone OpenFOAM containers repository
First, clone the container repository and change the directory to container/docker.
git clone https://develop.openfoam.com/packaging/containers.git
cd containers/docker
Apptainer Template Definition File for OpenFOAM
The following is the content of openfoam-run_rocky-template.def, the OpenFOAM Apptainer template definition file based on a Rocky Linux image. We can switch OS versions by using the OS_VER template variable, can change the OpenFOAM package using the PACKAGE template variable, and can change the OpenFOAM version by using the FOAM_VERSION template variable when we build images on the fly. Let’s see examples.
# ---------------------------------*-sh-*------------------------------------
# Copyright (C) 2021-2022 OpenCFD Ltd.
# SPDX-License-Identifier: (GPL-3.0+)
#
# Create openfoam '-run' image for Rocky Linux using copr repo.
#
# Example
# apptainer build openfoam2306.sif openfoam-run_rocky-template.def
# apptainer build --build-arg OS_VER=8 --build-arg FOAM_VERSION=2306 ...
#
# ---------------------------------------------------------------------------
Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}
%files
openfoam-files.rc/* /openfoam/
%arguments
OS_VER=latest
FOAM_VERSION=2306
PACKAGE=openfoam{{ FOAM_VERSION }}-devel
%post
dnf -y install wget rsync
dnf -y install 'dnf-command(config-manager)'
dnf -y install epel-release
crb enable
dnf -y install 'dnf-command(copr)'
dnf -y copr enable openfoam/openfoam
dnf -y install {{ PACKAGE }}
dnf -y clean all
%post
/bin/sh /openfoam/assets/post-install.sh -fix-perms -no-sudo
%runscript
exec /openfoam/run "$@"
%labels
Author OpenCFD Ltd.
Author CIQ Inc.
# ---------------------------------------------------------------------------
Build with Default Value (OS_VER=latest, FOAM_VERSION=2306)
This is the simplest example using default values specified in the %arguments section of the template definition file. rockylinux/rockylinux:latest points to the latest version of Rocky Linux 8.x, such as 8.8 here today.
apptainer build openfoam2306.sif openfoam-run_rocky-template.def
This command builds the image for an OpenFOAM v2306 development environment on Rocky Linux 8.8 without OpenFOAM tutorials.
Build OpenFOAM v2212 Developer Image on Rocky Linux 9.2 (OS_VER=9.2, FOAM_VERSION=2212)
This one builds the image for an OpenFOAM v2212 development environment on Rocky Linux 9.2 without OpenFOAM tutorials.
apptainer build --build-arg OS_VER=9.2 --build-arg FOAM_VERSION=2306 \
openfoam2212-dev.sif openfoam-run_rocky-template.def
Build OpenFOAM v2306 Default Image on Rocky Linux 8.8 (OS_VER=8.8, PACKAGE=openfoam2306-default)
This one builds the image for an OpenFOAM v2306 development environment on Rocky Linux 8.8 that includes OpenFOAM tutorials inside an image. Let’s use this image at the next step 🙂:
apptainer build --build-arg OS_VER=8.8 --build-arg PACKAGE=openfoam2306-default \
openfoam2306-default.sif openfoam-run_rocky-template.def
Before we move on to the next step, let’s deploy this OpenFOAM Apptainer image to your PATH.
If you don’t have an ~/bin directory, create the directory first. If you are working on Rocky Linux or equivalent derivatives, you can skip this step.
mkdir -p ~/bin
If you are working on Ubuntu or equivalent derivatives, make sure ~/bin is set to your PATH. If you are working on Rocky Linux or equivalent derivatives, you can skip this step.
export PATH=~/bin:$PATH
echo 'export PATH=~/bin:$PATH' >> ~/.bashrc
Move openfoam2306-default.sif to ~/bin .
mv openfoam2306-default.sif ~/bin
Now you are ready to go to the next step!
Run motorBike Example using OpenFOAM v2306 Default Apptainer Image
Make sure you are on your ~/ home directory. Create an openfoam2306 directory and change the directory there.
cd ~/
mkdir openfoam2306 && cd $_
Execute the container image you just created at the previous step:
openfoam2306-default.sif
This gives you an OpenFOAM shell that has all the necessary environment variables already set and ready to roll.
---------------------------------------------------------------------------
========= |
\\ / F ield | OpenFOAM in a container [from OpenCFD Ltd.]
\\ / O peration |
\\ / A nd | www.openfoam.com
\\/ M anipulation |
---------------------------------------------------------------------------
Release notes: https://www.openfoam.com/news/main-news/openfoam-v2306
Documentation: https://www.openfoam.com/documentation/
Issue Tracker: https://develop.openfoam.com/Development/openfoam/issues/
Local Help: more /openfoam/README
---------------------------------------------------------------------------
System : Rocky Linux 8.6 (Green Obsidian)
OpenFOAM : /usr/lib/openfoam/openfoam2306
Build : _fbf00d6b-20230626 OPENFOAM=2306 patch=0
Note
Different OpenFOAM components and modules may be present (or missing)
on any particular container installation.
Eg, source code, tutorials, in-situ visualization, paraview plugins,
external linear-solver interfaces etc.
---------------------------------------------------------------------------
apptainer-openfoam2306:~/openfoam2306
ysenda$
Check that the tutorials path exists on your container.
apptainer-openfoam2306:~/openfoam2306
ysenda$ echo $FOAM_TUTORIALS/
/usr/lib/openfoam/openfoam2306/tutorials/
Copy the OpenFOAM tutorials to your home directory. This home directory is the home directory on your host machine bind mounted to your container home directory. Apptainer is automatically bind mounted to your home directory on the container filesystem, so you can access your data under the home directory without explicitly mounting to the container like the other container solutions.
apptainer-openfoam2306:~/openfoam2306
ysenda$ cp -r $FOAM_TUTORIALS .
Change the directory to motorBike tutorial:
apptainer-openfoam2306:~/openfoam2306
ysenda$ cd tutorials/multiphase/interFoam/RAS/motorBike
If you are working on 64 cores system, you can change parallel run settings by doing the following. The default settings of number of parallel processes is 5. You can skip this step if you are good with default settings.
cat << EOF > system/decomposeParDict
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
note "mesh decomposition control dictionary";
object decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
numberOfSubdomains 64;
method scotch;
EOF
Run the tutorial using the Allrun script. This will run the motorBike example using MPI (default: 5 MPI parallel processes). The motorBike example is a bit heavy workload, so you can quit simulation by Ctrl + c .
apptainer-openfoam2306:~/openfoam2306/tutorials/multiphase/interFoam/RAS/motorBike
ysenda$ ./Allrun
Customize Definition File for Your OpenFOAM Development Project
Does your OpenFOAM development project need additional libraries? YES? No problem, we can customize the official definition file to fit your needs.
For example, for the “drlfoam project” (deep reinforcement learning with OpenFOAM) at https://github.com/OFDataCommittee/drlfoam, it seems like the project requires Python 3.8, PyTorch 1.12.1 (CPU only version), pandas, numpy, and PyTorch API for C++. Let’s add those dependencies to the definition file. See line 36 to line 51, as you may notice customizing the definition file is pretty straight-forward. You don’t need to learn tool specific grammar; it’s just the same as what you do on a Linux system.
We are not going to verify this definition file works for “drlfoam” in this article. This is intended to show you how easy it is to customize the definition file for your project.
# ---------------------------------*-sh-*------------------------------------
# Copyright (C) 2021-2022 OpenCFD Ltd.
# SPDX-License-Identifier: (GPL-3.0+)
#
# Create openfoam '-run' image for Rocky Linux using copr repo.
#
# Example
# apptainer build openfoam2306.sif openfoam-run_rocky-template.def
# apptainer build --build-arg OS_VER=8 --build-arg FOAM_VERSION=2306 ...
#
# ---------------------------------------------------------------------------
Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}
%files
openfoam-files.rc/* /openfoam/
%arguments
OS_VER=8.8
FOAM_VERSION=2206
PACKAGE=openfoam{{ FOAM_VERSION }}-devel
%post
dnf -y install wget rsync
dnf -y install 'dnf-command(config-manager)'
dnf -y install epel-release
crb enable
dnf -y install 'dnf-command(copr)'
dnf -y copr enable openfoam/openfoam
dnf -y install {{ PACKAGE }}
dnf -y clean all
%post
/bin/sh /openfoam/assets/post-install.sh -fix-perms -no-sudo
%post
dnf -y install unzip
dnf install -y python38 python38-pip
dnf clean all
cat << EOF > requirements.txt
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.12.1+cpu
pandas
numpy
EOF
pip3 install -r requirements.txt
rm requirements.txt
wget -q -O libtorch.zip https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.6.0%2Bcpu.zip
unzip libtorch.zip -d opt/
rm libtorch.zip
%runscript
exec /openfoam/run "$@"
%labels
Author OpenCFD Ltd.
Author CIQ Inc.
# ---------------------------------------------------------------------------
You can build the above example definition file with the following command:
apptainer build drlfoam.sif drlfoam-openfoam-run_rocky-template.def
Your own OpenFOAM Apptainer Image on Docker Hub
Reproducibility is important in many cases. When you work with collaborators, sharing your development environment with those collaborators is kind of pain point. The combination of Apptainer (formerly Singularity) and container registries can help with that part.
Once you build your own development environment as an Apptainer container, you can push that Apptainer container image to OCI registries (see more details here: https://ciq.com/blog/using-apptainers-with-public-cloud-container-artifact-registry-offerings/).
The best well-known OCI registry is Docker Hub. Docker Hub usually hosts Docker container images like the following:
But Docker Hub is also capable of hosting Apptainer container images because they supports ORAS protocol. We can push an Apptainer container image to Docker Hub using the apptainer command line tool that also supports ORAS protocol.
First, login to Docker Hub. Please replace USERNAME to fit your case. You will be asked for a Password/Token; please enter your token here.
apptainer remote login --username USERNAME oras://docker.io
Once you have signed into Docker Hub, you can create a personal access token here.
After successful login, push your image to Docker Hub.
apptainer push awesomefoam-dev.sif oras://docker.io/USERNAME/awesomefoam-dev:2306
If your repository is the public repository, your collaborators can pull this image without authentication.
apptainer pull oras://docker.io/USERNAME/awesomefoam-dev:2306
Now your collaborators can start developing the project using exactly the same environment as you do 🎉.
./awesomefoam-dev_2306.sif
---------------------------------------------------------------------------
========= |
\\ / F ield | OpenFOAM in a container [from OpenCFD Ltd.]
\\ / O peration |
\\ / A nd | www.openfoam.com
\\/ M anipulation |
---------------------------------------------------------------------------
Release notes: https://www.openfoam.com/news/main-news/openfoam-v2306
Documentation: https://www.openfoam.com/documentation/
Issue Tracker: https://develop.openfoam.com/Development/openfoam/issues/
Local Help: more /openfoam/README
---------------------------------------------------------------------------
System : Rocky Linux 8.6 (Green Obsidian)
OpenFOAM : /usr/lib/openfoam/openfoam2306
Build : _fbf00d6b-20230626 OPENFOAM=2306 patch=0
Note
Different OpenFOAM components and modules may be present (or missing)
on any particular container installation.
Eg, source code, tutorials, in-situ visualization, paraview plugins,
external linear-solver interfaces etc.
---------------------------------------------------------------------------
apptainer-openfoam2306:~/openfoam2306
xu$
Tada!
Acknowledgement
We would like to say thank you to those who participated in the feature request discussions.
-
Mark Olesen (OpenCFD Ltd.)
-
Xu Yang
-
Dave Dykstra
-
Dave Trudgian
-
Dave Godlove
-
Davis Catherman
-
Gregory M. Kurtzer
-
Jonathon Anderson
-
Yoshiaki Senda
Appendix
Available tags for Rocky Linux images
You can find tags here https://hub.docker.com/r/rockylinux/rockylinux/tags .
OpenFOAM packages
Currently available OpenFOAM packages on the Rocky Linux (EL8/EL9) environment are the following. The same packages are also available on Debian, Ubuntu, and openSUSE.
$ dnf search openfoam
Last metadata expiration check: 0:01:01 ago on Wed Jul 5 03:41:56 2023.
================================================ Name Exactly Matched: openfoam ================================================
openfoam.noarch : Free, Open Source, Computational Fluid Dynamics Package
=============================================== Name & Summary Matched: openfoam ===============================================
openfoam-default.x86_64 : OpenFOAM default installation bundle
openfoam-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam-selector.noarch : Tool to manage which OpenFOAM version to use
openfoam-selector.src : Tool to manage which OpenFOAM version to use
openfoam1912-common.noarch : OpenFOAM common files
openfoam1912-default.x86_64 : OpenFOAM default installation bundle
openfoam1912-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam1912-doc.noarch : OpenFOAM documentation
openfoam1912-tools.x86_64 : OpenFOAM-specific build tools
openfoam1912-tutorials.noarch : OpenFOAM tutorials
openfoam2006-common.noarch : OpenFOAM common files
openfoam2006-default.x86_64 : OpenFOAM default installation bundle
openfoam2006-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2006-doc.noarch : OpenFOAM documentation
openfoam2006-tools.x86_64 : OpenFOAM-specific build tools
openfoam2006-tutorials.noarch : OpenFOAM tutorials
openfoam2012-common.noarch : OpenFOAM common files
openfoam2012-default.x86_64 : OpenFOAM default installation bundle
openfoam2012-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2012-doc.noarch : OpenFOAM documentation
openfoam2012-tools.x86_64 : OpenFOAM-specific build tools
openfoam2012-tutorials.noarch : OpenFOAM tutorials
openfoam2106-common.noarch : OpenFOAM common files
openfoam2106-default.x86_64 : OpenFOAM default installation bundle
openfoam2106-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2106-doc.noarch : OpenFOAM documentation
openfoam2106-tools.x86_64 : OpenFOAM-specific build tools
openfoam2106-tutorials.noarch : OpenFOAM tutorials
openfoam2112-common.noarch : OpenFOAM common files
openfoam2112-default.x86_64 : OpenFOAM default installation bundle
openfoam2112-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2112-doc.noarch : OpenFOAM documentation
openfoam2112-tools.x86_64 : OpenFOAM-specific build tools
openfoam2112-tutorials.noarch : OpenFOAM tutorials
openfoam2206-common.noarch : OpenFOAM common files
openfoam2206-default.x86_64 : OpenFOAM default installation bundle
openfoam2206-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2206-doc.noarch : OpenFOAM documentation
openfoam2206-tools.x86_64 : OpenFOAM-specific build tools
openfoam2206-tutorials.noarch : OpenFOAM tutorials
openfoam2212-common.noarch : OpenFOAM common files
openfoam2212-default.x86_64 : OpenFOAM default installation bundle
openfoam2212-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2212-doc.noarch : OpenFOAM documentation
openfoam2212-tools.x86_64 : OpenFOAM-specific build tools
openfoam2212-tutorials.noarch : OpenFOAM tutorials
openfoam2306-common.noarch : OpenFOAM common files
openfoam2306-default.x86_64 : OpenFOAM default installation bundle
openfoam2306-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2306-doc.noarch : OpenFOAM documentation
openfoam2306-tools.x86_64 : OpenFOAM-specific build tools
openfoam2306-tutorials.noarch : OpenFOAM tutorials
==================================================== Name Matched: openfoam ====================================================
openfoam1912.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam1912.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2006.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2006.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2012.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2012.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2106.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2106.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2112.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2112.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2206.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2206.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2212.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2212.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2306.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2306.src : Free, Open Source, Computational Fluid Dynamics Package
