Forward

by Brock Taylor

Collaboration through open source may become a necessity for application developers to keep pace with the rapid introduction of technology in high performance computing. With the rapid injection of new silicon, new performance software, and managing all the existing components, developing for high performance computing is stress-inducing. 

The efforts led by Yoshiaki Senda, however, demonstrate the value and benefits of how an open source community can meet the challenge of keeping pace. The original effort to make a containerized version of OpenRadioss with Apptainer quickly spawned additional collaborations. Yoshiaki’s efforts detailed in this article demonstrate how effective the open source community can be. 

Open source does not mean easy! Altair, Intel, and CIQ engineers tackled some tough issues, but the process was simpler and faster because of the community. This is likely just the beginning of what will become a steady stream of innovations going into OpenRadioss, and ultimately into the professional version of Radioss offered by Altair. This has already been fun to watch and play a role in.

Summary

This article describes two things: (1) How to create an Intel oneAPI build of OpenRadioss Apptainer container image from a daily released repository, and (2) How to run the 2010 Toyota Yaris Detailed Finite Element Model Yaris Impact Model in LS-DYNA® format using the Intel oneAPI build of OpenRadioss Apptainer container. We tested these procedures on Rocky Linux 8.6 and Rocky Linux 9.

Installation

The steps below have been tested and validated on Rocky Linux 8.6 and 9.0. If you are planning to try this out yourself, please make sure you have enough disk space. 80GB or above is recommended to test OpenRadioss examples.

Apptainer

The first step is to install Apptainer on Rocky Linux. Since the release of Apptainer version 1.1.0, the RPM package has been merged into the upstream EPEL repository, making installation and updating easier than ever.

sudo dnf install -y epel-release 

For alternate installation methods, check the Apptainer documentation.

Definition file

A definition file is like a set of blueprints for building a container. The OpenRadioss definition file sets up a base container and then pulls in the necessary tools and libraries, grabs the OpenRadioss source code from the main branch that is available on OpenRadioss GitHub repository, builds it, and sets the proper runtime environment.

The Apptainer definition file for the Intel oneAPI build of OpenRadioss container is merged into OpenRadioss GitHub repository. Let’s clone the OpenRadioss repository to build the OpenRadioss container.

git clone https://github.com/OpenRadioss/OpenRadioss.git 

Build an Apptainer image for Intel oneAPI build of OpenRadioss

Now we can build the OpenRadioss main Apptainer image with the following command without root privilege. It is a recommended best security practice to build containers without elevated privileges.

Build the OpenRadioss container together with Intel MPI and Intel MKL using Intel oneAPI compilers (ifx, icx, icpx) with the following command.

apptainer build --fakeroot openradioss_intel.sif openradioss_intel.def

Copy this Apptainer image to somewhere in your PATH .

mkdir -p ~/bin cp openradioss_intel.sif ~/bin

Containerized! We’re ready to utilize OpenRadioss using this openradioss_intel.sif Apptainer image on your compute resources. This can run on your laptop, workstation, on-prem HPC cluster, and cloud resources.

Intel MPI

Before running a model, install Intel MPI on the host system if it’s not already installed.

sudo cp OpenRadioss/Apptainer/oneAPI.repo /etc/yum.repos.d/oneAPI.repo 

Set environment variables:

source /opt/intel/oneapi/lib/env/compiler_rt_vars.sh 

Simulation setup

Now that we have the container, we will want to test it with the Toyota Yaris workload. This is a crash simulation, and we will use ParaView to visualize the result.

Download the 2010 Toyota Yaris Impact Model in LS-DYNA® format

Download the model from CCSA, download the OpenRadioss launch file for this simulation. Please see further details about the OpenRadioss launch file here.

wget -q https://media.ccsa.gmu.edu/model/2010-toyota-yaris-detailed-v2j.zip unzip 2010-toyota-yaris-detailed-v2j.zip cd 2010-toyota-yaris-detailed-v2j wget -q -O YarisOpenRadioss.key https://openradioss.atlassian.net/wiki/download/attachments/30539777/YarisOpenRadioss.key?api=v2

Running Intel oneAPI build OpenRadioss using Intel MPI

Go to the model directory and then set up environment variables for Intel oneAPI runtime libraries.

cd 2010-toyota-yaris-detailed-v2j/ source /opt/intel/oneapi/lib/env/compiler_rt_vars.sh export PATH=/opt/intel/oneapi/lib/intel64/bin:$PATH export LD_LIBRARY_PATH=/opt/intel/oneapi/lib/intel64:$LD_LIBRARY_PATH

Run starter first. This pre-processing tool reads the data, initializes the simulation, checks the model, decomposes the domain for parallel processing and outputs simulation configuration and restart files. Make sure to set the -np parameter for the number of processors you will use in the second step below.

openradioss_intel.sif starter_linux64_intel -i YarisOpenRadioss.key -np 8

Then, run engine that is the actual solver using Intel MPI. Here we specify 8 MPI processes to run this simulation. This takes about 15 hours with “12th Gen Intel(R) Core(TM) i7-12700” and consumes about 13GB memory. For storage, at least 80GB of storage (SSD/HDD) is recommended to build and run this simulation.

mpirun -np 8 openradioss_intel.sif engine_linux64_intel_impi -i YarisOpenRadioss_0001.rad ************************************************************************ OpenRadioss Engine Non-linear Finite Element Analysis SoftwareLinux 64 bits, Intel compiler, Intel MPI************************************************************************ OpenRadioss SoftwareCOPYRIGHT (C) 1986-2022 Altair Engineering, Inc.Licensed under GNU Affero General Public License.See License file.************************************************************************ ROOT: YarisOpenRadioss RESTART: 0001 25 NUMBER OF HMPP PROCESSES 8

Output

If it runs successfully, it will show you:

 ELAPSED TIME     =      11648.55 s
                          3:14:08
 ESTIMATED SPEEDUP=         53.03



     NORMAL TERMINATION



     TOTAL NUMBER OF CYCLES  :  200848

Post-processing using ParaView

Now we want to see our results using ParaView. The Toyota Yaris simulation output files are in ANIM format. We need to convert them to VTK format and install ParaView.

Convert output files ANIM to VTK format

To visualize ANIM format with ParaView, we need to convert ANIM format to VTK format. The main OpenRadioss includes a converter utility anim_to_vtk inside tools directory.

A directory listing shows we have 21 output files from our simulation to convert, and each file has a three digit number, from 001 to 021, at the end of the name.

$ ls YarisOpenRadiossA* YarisOpenRadiossA001 YarisOpenRadiossA005 YarisOpenRadiossA009 YarisOpenRadiosA013 YarisOpenRadiossA017 YarisOpenRadiossA021 3YarisOpenRadiossA002 YarisOpenRadiossA006 YarisOpenRadiossA010 YarisOpenRadiosA014 YarisOpenRadiossA018 4YarisOpenRadiossA003 YarisOpenRadiossA007 YarisOpenRadiossA011 YarisOpenRadiosA015 YarisOpenRadiossA019 5YarisOpenRadiossA004 YarisOpenRadiossA008 YarisOpenRadiossA012 YarisOpenRadiosA016 YarisOpenRadiossA020

The following command will convert all 21 files from ANIM format to VTK format.

$ seq -f YarisOpenRadiossA%03g 021 | xargs -L 1 -I{} sh -c 'openradioss_intel.sif anim_to_vtk_linux64_gf "$1" > "$1.vtk"' -- {}

Now our directory listing shows our simulation results in VTK format, ready for rendering by ParaView.

$ ls YarisOpenRadiossA*.vtk YarisOpenRadiossA001.vtk YarisOpenRadiossA006.vtk YarisOpenRadiossA011.vtk YarisOpenRadiossA016.vtk YarisOpenRadiossA021.vtk 3YarisOpenRadiossA002.vtk YarisOpenRadiossA007.vtk YarisOpenRadiossA012.vtk YarisOpenRadiossA017.vtk 4YarisOpenRadiossA003.vtk YarisOpenRadiossA008.vtk YarisOpenRadiossA013.vtk YarisOpenRadiossA018.vtk 5YarisOpenRadiossA004.vtk YarisOpenRadiossA009.vtk YarisOpenRadiossA014.vtk YarisOpenRadiossA019.vtk 6YarisOpenRadiossA005.vtk YarisOpenRadiossA010.vtk YarisOpenRadiossA015.vtk YarisOpenRadiossA020.vtk

Install ParaView

Download ParaView from their official website and place it in your preferred location.

$ wget -O ParaView-5.10.1-MPI-Linux-Python3.9-x86_64.tar.gz "https://www.paraview.org/paraview-downloads/download.php?submit=Download&version=v5.10&type=binary&os=Linux&downloadFile=ParaView-5.10.1-MPI-Linux-Python3.9-x86_64.tar.gz" $ tar zxvf ParaView-5.10.1-MPI-Linux-Python3.9-x86_64.tar.gz $ sudo mkdir -p /opt/ParaView $ sudo mv ParaView-5.10.1-MPI-Linux-Python3.9-x86_64 /opt/ParaView/5.10.1 $ rm -rf ParaView-5.10.1-MPI-Linux-Python3.9-x86_64.tar.gz $ export PATH=/opt/ParaView/5.10.1/bin:$PATH $ export LD_LIBRARY_PATH=/opt/ParaView/5.10.1/lib:$LD_LIBRARY_PATH

Rendering the results

To visualize our results, simply call ParaView, providing it with which file to render. Here is the command and resulting visualization for YarisOpenRadiossA012.vtk.

$ paraview --data=YarisOpenRadiossA012.vtk

The pictures below show stress magnitude. The image is a rendering of YarisOpenRadiossA012.vtk. The result shows that most of the stress is absorbed at the front end and much less at the passenger cabin.