CIQ

OpenFOAM Supports Apptainer v1.2.0 Template Definition File Feature

OpenFOAM Supports Apptainer v1.2.0 Template Definition File Feature
Yoshiaki SendaJuly 25, 2023

Introduction

We are happy to announce that OpenFOAM® (OpenCFD Ltd.) is the very first project that officially supports the brand-new Apptainer v1.2.0 feature “template definition file” just released on July 18, 2023. OpenFOAM officially supports Apptainer since OpenFOAM v2212 and since then we have closely worked together on the OpenFOAM container repository.

Screenshot 2023 07 04 at 22 57 39 1024x779

In this blog post, we will explain (1) Apptainer “template definition file" basics, (2) how to use OpenFOAM's official Apptainer template definition file to build your own brand-new OpenFOAM v2306 container, (3) how to create your own OpenFOAM development environment based on the official template, and (4) finally, we will show you how to share your own OpenFOAM development environment with your collaborators using Docker Hub. Apptainer helps drive researchers,' developers,' and engineers' collaboration by providing “Mobility of Compute.”

Apptainer Template Definition File Basics

The new “template definition file” feature introduces template variables to every section of the Apptainer definition file. Before we introduced these template variables, environment variables were used for similar purposes and still can be used, but it only works for limited sections of the Apptainer definition file (e.g., environment variables can not be used for bootstrap OS image names).

Template variables

With template variables, for example, we can switch bootstrap OS image versions like the following:

Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}
%arguments section

To support default values for template variables, an %arguments section is added to the Apptainer definition file. Let’s make a file called rockylinux-template.def with the following contents.

Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}

%arguments
    OS_VER=8.8

With a default value, you can build this definition file as usual:

apptainer build rockylinux8.sif rockylinux-template.def

Now a Rocky Linux 8.8 container called rockylinux8.sif is built.

./rockylinux8.sif cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.8 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.8 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"
--build-arg option

The new option is added to apptainer build command, --build-arg, and --build-arg-file. When you want to bump up the OS version to 9.2, here is the time to use that new option --build-arg.

apptainer build --build-arg OS_VER=9.2 rockylinux9.sif rockylinux-template.def

This time, a Rocky Linux 9.2 container called rockylinux9.sif is built.

./rockylinux9.sif cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
--build-arg-file option

Create the file called args.list(or whatever name and extension):

cat << EOF > args.list
OS_VER=9.2
EOF

You can build the same Apptainer image as above with --build-arg-file option. The file-based approach is convenient when you want to keep the build command the same but change the build settings by just changing the arguments list (e.g., CI/CD pipelines) and/or when you want to share the build settings.

apptainer build --build-arg-file args.list rockylinux9.2.sif rockylinux-template.def

Again, a Rocky Linux 9.2 container called rockylinux9.2.sif is built. Let’s move on to the hands-on guide to OpenFOAM official Apptainer definition file 🙂.

Hands-on Guide to OpenFOAM Official Apptainer Definition File

The OpenFOAM official container repository is here https://develop.openfoam.com/packaging/containers .

Screenshot 2023 07 20 at 14 54 46

Clone OpenFOAM containers repository

First, clone the container repository and change the directory to container/docker.

git clone https://develop.openfoam.com/packaging/containers.git
cd containers/docker

Apptainer Template Definition File for OpenFOAM

The following is the content of openfoam-run_rocky-template.def, the OpenFOAM Apptainer template definition file based on a Rocky Linux image. We can switch OS versions by using the OS_VER template variable, can change the OpenFOAM package using the PACKAGE template variable, and can change the OpenFOAM version by using the FOAM_VERSION template variable when we build images on the fly. Let’s see examples.

# ---------------------------------*-sh-*------------------------------------
# Copyright (C) 2021-2022 OpenCFD Ltd.
# SPDX-License-Identifier: (GPL-3.0+)
#
# Create openfoam '-run' image for Rocky Linux using copr repo.
#
# Example
#     apptainer build openfoam2306.sif openfoam-run_rocky-template.def
#     apptainer build --build-arg OS_VER=8 --build-arg FOAM_VERSION=2306 ...
#
# ---------------------------------------------------------------------------
Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}

%files
    openfoam-files.rc/* /openfoam/

%arguments
    OS_VER=latest
    FOAM_VERSION=2306
    PACKAGE=openfoam{{ FOAM_VERSION }}-devel

%post
    dnf -y install wget rsync
    dnf -y install 'dnf-command(config-manager)'
    dnf -y install epel-release
    crb enable
    dnf -y install 'dnf-command(copr)'
    dnf -y copr enable openfoam/openfoam
    dnf -y install {{ PACKAGE }}
    dnf -y clean all

%post
    /bin/sh /openfoam/assets/post-install.sh -fix-perms -no-sudo

%runscript
    exec /openfoam/run "$@"

%labels
    Author OpenCFD Ltd.
    Author CIQ Inc.

# ---------------------------------------------------------------------------
Build with Default Value (OS_VER=latest, FOAM_VERSION=2306)

This is the simplest example using default values specified in the %arguments section of the template definition file. rockylinux/rockylinux:latest points to the latest version of Rocky Linux 8.x, such as 8.8 here today.

apptainer build openfoam2306.sif openfoam-run_rocky-template.def

This command builds the image for an OpenFOAM v2306 development environment on Rocky Linux 8.8 without OpenFOAM tutorials.

Build OpenFOAM v2212 Developer Image on Rocky Linux 9.2 (OS_VER=9.2, FOAM_VERSION=2212)

This one builds the image for an OpenFOAM v2212 development environment on Rocky Linux 9.2 without OpenFOAM tutorials.

apptainer build --build-arg OS_VER=9.2 --build-arg FOAM_VERSION=2306 \
  openfoam2212-dev.sif openfoam-run_rocky-template.def
Build OpenFOAM v2306 Default Image on Rocky Linux 8.8 (OS_VER=8.8, PACKAGE=openfoam2306-default)

This one builds the image for an OpenFOAM v2306 development environment on Rocky Linux 8.8 that includes OpenFOAM tutorials inside an image. Let’s use this image at the next step 🙂:

apptainer build --build-arg OS_VER=8.8 --build-arg PACKAGE=openfoam2306-default \
  openfoam2306-default.sif openfoam-run_rocky-template.def

Before we move on to the next step, let’s deploy this OpenFOAM Apptainer image to your PATH.

If you don’t have an ~/bin directory, create the directory first. If you are working on Rocky Linux or equivalent derivatives, you can skip this step.

mkdir -p ~/bin

If you are working on Ubuntu or equivalent derivatives, make sure ~/bin is set to your PATH. If you are working on Rocky Linux or equivalent derivatives, you can skip this step.

export PATH=~/bin:$PATH
echo 'export PATH=~/bin:$PATH' >> ~/.bashrc

Move openfoam2306-default.sif to ~/bin .

mv openfoam2306-default.sif ~/bin

Now you are ready to go to the next step!

Run motorBike Example using OpenFOAM v2306 Default Apptainer Image

Make sure you are on your ~/ home directory. Create an openfoam2306 directory and change the directory there.

cd ~/
mkdir openfoam2306 && cd $_

Execute the container image you just created at the previous step:

openfoam2306-default.sif

This gives you an OpenFOAM shell that has all the necessary environment variables already set and ready to roll.

---------------------------------------------------------------------------
  =========                 |
  \\      /  F ield         | OpenFOAM in a container [from OpenCFD Ltd.]
   \\    /   O peration     |
    \\  /    A nd           | www.openfoam.com
     \\/     M anipulation  |
---------------------------------------------------------------------------
 Release notes:  https://www.openfoam.com/news/main-news/openfoam-v2306
 Documentation:  https://www.openfoam.com/documentation/
 Issue Tracker:  https://develop.openfoam.com/Development/openfoam/issues/
 Local Help:     more /openfoam/README
---------------------------------------------------------------------------
System   :  Rocky Linux 8.6 (Green Obsidian)
OpenFOAM :  /usr/lib/openfoam/openfoam2306
Build    :  _fbf00d6b-20230626 OPENFOAM=2306 patch=0

Note
    Different OpenFOAM components and modules may be present (or missing)
    on any particular container installation.
    Eg, source code, tutorials, in-situ visualization, paraview plugins,
        external linear-solver interfaces etc.
---------------------------------------------------------------------------
apptainer-openfoam2306:~/openfoam2306
ysenda$

Check that the tutorials path exists on your container.

apptainer-openfoam2306:~/openfoam2306
ysenda$ echo $FOAM_TUTORIALS/
/usr/lib/openfoam/openfoam2306/tutorials/

Copy the OpenFOAM tutorials to your home directory. This home directory is the home directory on your host machine bind mounted to your container home directory. Apptainer is automatically bind mounted to your home directory on the container filesystem, so you can access your data under the home directory without explicitly mounting to the container like the other container solutions.

apptainer-openfoam2306:~/openfoam2306
ysenda$ cp -r $FOAM_TUTORIALS .

Change the directory to motorBike tutorial:

apptainer-openfoam2306:~/openfoam2306
ysenda$ cd tutorials/multiphase/interFoam/RAS/motorBike

If you are working on 64 cores system, you can change parallel run settings by doing the following. The default settings of number of parallel processes is 5. You can skip this step if you are good with default settings.

cat << EOF > system/decomposeParDict
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains  64;
method scotch;
EOF

Run the tutorial using the Allrun script. This will run the motorBike example using MPI (default: 5 MPI parallel processes). The motorBike example is a bit heavy workload, so you can quit simulation by Ctrl + c .

apptainer-openfoam2306:~/openfoam2306/tutorials/multiphase/interFoam/RAS/motorBike
ysenda$ ./Allrun

Customize Definition File for Your OpenFOAM Development Project

Does your OpenFOAM development project need additional libraries? YES? No problem, we can customize the official definition file to fit your needs.

For example, for the “drlfoam project” (deep reinforcement learning with OpenFOAM) at https://github.com/OFDataCommittee/drlfoam, it seems like the project requires Python 3.8, PyTorch 1.12.1 (CPU only version), pandas, numpy, and PyTorch API for C++. Let’s add those dependencies to the definition file. See line 36 to line 51, as you may notice customizing the definition file is pretty straight-forward. You don’t need to learn tool specific grammar; it’s just the same as what you do on a Linux system.

We are not going to verify this definition file works for “drlfoam” in this article. This is intended to show you how easy it is to customize the definition file for your project.

# ---------------------------------*-sh-*------------------------------------
# Copyright (C) 2021-2022 OpenCFD Ltd.
# SPDX-License-Identifier: (GPL-3.0+)
#
# Create openfoam '-run' image for Rocky Linux using copr repo.
#
# Example
#     apptainer build openfoam2306.sif openfoam-run_rocky-template.def
#     apptainer build --build-arg OS_VER=8 --build-arg FOAM_VERSION=2306 ...
#
# ---------------------------------------------------------------------------
Bootstrap: docker
From: rockylinux/rockylinux:{{ OS_VER }}

%files
    openfoam-files.rc/* /openfoam/

%arguments
    OS_VER=8.8
    FOAM_VERSION=2206
    PACKAGE=openfoam{{ FOAM_VERSION }}-devel

%post
    dnf -y install wget rsync
    dnf -y install 'dnf-command(config-manager)'
    dnf -y install epel-release
    crb enable
    dnf -y install 'dnf-command(copr)'
    dnf -y copr enable openfoam/openfoam
    dnf -y install {{ PACKAGE }}
    dnf -y clean all

%post
    /bin/sh /openfoam/assets/post-install.sh -fix-perms -no-sudo

%post
    dnf -y install unzip
    dnf install -y python38 python38-pip
    dnf clean all
    cat << EOF > requirements.txt
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.12.1+cpu
pandas
numpy
EOF
    pip3 install -r requirements.txt
    rm requirements.txt

    wget -q -O libtorch.zip https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.6.0%2Bcpu.zip
    unzip libtorch.zip -d opt/
    rm libtorch.zip

%runscript
    exec /openfoam/run "$@"

%labels
    Author OpenCFD Ltd.
    Author CIQ Inc.

# ---------------------------------------------------------------------------

You can build the above example definition file with the following command:

apptainer build drlfoam.sif drlfoam-openfoam-run_rocky-template.def

Your own OpenFOAM Apptainer Image on Docker Hub

Reproducibility is important in many cases. When you work with collaborators, sharing your development environment with those collaborators is kind of pain point. The combination of Apptainer (formerly Singularity) and container registries can help with that part.

Once you build your own development environment as an Apptainer container, you can push that Apptainer container image to OCI registries (see more details here: https://ciq.com/blog/using-apptainers-with-public-cloud-container-artifact-registry-offerings/).

The best well-known OCI registry is Docker Hub. Docker Hub usually hosts Docker container images like the following:

Screenshot 2023 07 05 at 11 47 53 1024x671

But Docker Hub is also capable of hosting Apptainer container images because they supports ORAS protocol. We can push an Apptainer container image to Docker Hub using the apptainer command line tool that also supports ORAS protocol.

First, login to Docker Hub. Please replace USERNAME to fit your case. You will be asked for a Password/Token; please enter your token here.

apptainer remote login --username USERNAME oras://docker.io

Once you have signed into Docker Hub, you can create a personal access token here.

After successful login, push your image to Docker Hub.

apptainer push awesomefoam-dev.sif  oras://docker.io/USERNAME/awesomefoam-dev:2306

If your repository is the public repository, your collaborators can pull this image without authentication.

apptainer pull oras://docker.io/USERNAME/awesomefoam-dev:2306

Now your collaborators can start developing the project using exactly the same environment as you do 🎉.

./awesomefoam-dev_2306.sif
---------------------------------------------------------------------------
  =========                 |
  \\      /  F ield         | OpenFOAM in a container [from OpenCFD Ltd.]
   \\    /   O peration     |
    \\  /    A nd           | www.openfoam.com
     \\/     M anipulation  |
---------------------------------------------------------------------------
 Release notes:  https://www.openfoam.com/news/main-news/openfoam-v2306
 Documentation:  https://www.openfoam.com/documentation/
 Issue Tracker:  https://develop.openfoam.com/Development/openfoam/issues/
 Local Help:     more /openfoam/README
---------------------------------------------------------------------------
System   :  Rocky Linux 8.6 (Green Obsidian)
OpenFOAM :  /usr/lib/openfoam/openfoam2306
Build    :  _fbf00d6b-20230626 OPENFOAM=2306 patch=0

Note
    Different OpenFOAM components and modules may be present (or missing)
    on any particular container installation.
    Eg, source code, tutorials, in-situ visualization, paraview plugins,
        external linear-solver interfaces etc.
---------------------------------------------------------------------------
apptainer-openfoam2306:~/openfoam2306
xu$

Tada!

Acknowledgement

We would like to say thank you to those who participated in the feature request discussions.

  • Mark Olesen (OpenCFD Ltd.)

  • Xu Yang

  • Dave Dykstra

  • Dave Trudgian

  • Dave Godlove

  • Davis Catherman

  • Gregory M. Kurtzer

  • Jonathon Anderson

  • Yoshiaki Senda

Appendix

Available tags for Rocky Linux images

You can find tags here https://hub.docker.com/r/rockylinux/rockylinux/tags .

Screenshot 2023 07 05 at 12 29 38 1024x876

OpenFOAM packages

Currently available OpenFOAM packages on the Rocky Linux (EL8/EL9) environment are the following. The same packages are also available on Debian, Ubuntu, and openSUSE.

$ dnf search openfoam
Last metadata expiration check: 0:01:01 ago on Wed Jul  5 03:41:56 2023.
================================================ Name Exactly Matched: openfoam ================================================
openfoam.noarch : Free, Open Source, Computational Fluid Dynamics Package
=============================================== Name & Summary Matched: openfoam ===============================================
openfoam-default.x86_64 : OpenFOAM default installation bundle
openfoam-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam-selector.noarch : Tool to manage which OpenFOAM version to use
openfoam-selector.src : Tool to manage which OpenFOAM version to use
openfoam1912-common.noarch : OpenFOAM common files
openfoam1912-default.x86_64 : OpenFOAM default installation bundle
openfoam1912-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam1912-doc.noarch : OpenFOAM documentation
openfoam1912-tools.x86_64 : OpenFOAM-specific build tools
openfoam1912-tutorials.noarch : OpenFOAM tutorials
openfoam2006-common.noarch : OpenFOAM common files
openfoam2006-default.x86_64 : OpenFOAM default installation bundle
openfoam2006-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2006-doc.noarch : OpenFOAM documentation
openfoam2006-tools.x86_64 : OpenFOAM-specific build tools
openfoam2006-tutorials.noarch : OpenFOAM tutorials
openfoam2012-common.noarch : OpenFOAM common files
openfoam2012-default.x86_64 : OpenFOAM default installation bundle
openfoam2012-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2012-doc.noarch : OpenFOAM documentation
openfoam2012-tools.x86_64 : OpenFOAM-specific build tools
openfoam2012-tutorials.noarch : OpenFOAM tutorials
openfoam2106-common.noarch : OpenFOAM common files
openfoam2106-default.x86_64 : OpenFOAM default installation bundle
openfoam2106-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2106-doc.noarch : OpenFOAM documentation
openfoam2106-tools.x86_64 : OpenFOAM-specific build tools
openfoam2106-tutorials.noarch : OpenFOAM tutorials
openfoam2112-common.noarch : OpenFOAM common files
openfoam2112-default.x86_64 : OpenFOAM default installation bundle
openfoam2112-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2112-doc.noarch : OpenFOAM documentation
openfoam2112-tools.x86_64 : OpenFOAM-specific build tools
openfoam2112-tutorials.noarch : OpenFOAM tutorials
openfoam2206-common.noarch : OpenFOAM common files
openfoam2206-default.x86_64 : OpenFOAM default installation bundle
openfoam2206-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2206-doc.noarch : OpenFOAM documentation
openfoam2206-tools.x86_64 : OpenFOAM-specific build tools
openfoam2206-tutorials.noarch : OpenFOAM tutorials
openfoam2212-common.noarch : OpenFOAM common files
openfoam2212-default.x86_64 : OpenFOAM default installation bundle
openfoam2212-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2212-doc.noarch : OpenFOAM documentation
openfoam2212-tools.x86_64 : OpenFOAM-specific build tools
openfoam2212-tutorials.noarch : OpenFOAM tutorials
openfoam2306-common.noarch : OpenFOAM common files
openfoam2306-default.x86_64 : OpenFOAM default installation bundle
openfoam2306-devel.noarch : OpenFOAM source code headers and wmake build chain
openfoam2306-doc.noarch : OpenFOAM documentation
openfoam2306-tools.x86_64 : OpenFOAM-specific build tools
openfoam2306-tutorials.noarch : OpenFOAM tutorials
==================================================== Name Matched: openfoam ====================================================
openfoam1912.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam1912.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2006.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2006.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2012.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2012.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2106.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2106.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2112.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2112.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2206.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2206.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2212.src : Free, Open Source, Computational Fluid Dynamics Package
openfoam2212.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2306.x86_64 : Free, Open Source, Computational Fluid Dynamics Package
openfoam2306.src : Free, Open Source, Computational Fluid Dynamics Package

Related posts

A New Approach to MPI in Apptainer

A New Approach to MPI in Apptainer

Jun 27, 2023

Apptainer / Singularity

Apptainer 1.1.0 Is Here, with Improved Security and User Experience

Apptainer 1.1.0 Is Here, with Improved Security and User Experience

Sep 29, 2022

Apptainer / Singularity

Apptainer Official PPA for Ubuntu Is Now Available!

Apptainer Official PPA for Ubuntu Is Now Available!

Feb 2, 2023

Apptainer / Singularity

123
17
>>>