CIQ

OpenRadioss with Visualization Using ParaView on Oracle Cloud

OpenRadioss with Visualization Using ParaView on Oracle Cloud
Yoshiaki SendaJune 23, 2023

This guide provides you with a fast-track path to run OpenRadioss simulations and use ParaView to visualize the results. CIQ created builds of OpenRadioss and ParaView and packaged them into a single Rocky Linux 9 image published in the Oracle Cloud marketplace. In this guide, we’ll provide some of the details of the effort and, most importantly, the instructions to try out OpenRadioss simulations quickly. By following the instructions in this guide, you can try a sample workload or give your own workload a go!

OpenRadioss is an industry-proven open source Finite Element Analysis solver recently provided by Altair. More information, community support of the code, and additional workloads may be found on the OpenRadioss community website.

ParaView is an award-winning open source visualization application. 

Rocky Linux is a popular Linux distribution across all domains, including scientific applications like OpenRadioss. CIQ is a founding partner in Rocky Linux, the Rocky Linux Enterprise Software Foundation, and also provides professional support for Rocky Linux. 

As a prerequisite to executing this workflow, you will need to have an account on Oracle Cloud. This guide does not cover the instructions for setting up an account.

Minimizing the cost of running in the cloud

The instructions for executing OpenRadioss and ParaView in Oracle Cloud take into account that running in the cloud is not free. With that in mind, this section provides some background on minimizing the cost of running in the cloud.

The primary cost-efficient consideration comes down to selecting the right compute instance shape for each part of the workflow. OpenRadioss executes its solver on CPUs, while ParaView needs GPU power for the visualization. The cost of a GPU instance shape is much higher, typically, than a shape with just CPUs. If we choose one instance shape that satisfies both requirements, then the overall cost of execution will be greater. Instead, this guide uses a two-step workflow. The first uses a CPU only shape to execute the OpenRadioss simulation. The second uses a shape with an NVIDIA GPU to visualize the results. While there are additional steps to perform, this is overall a more cost effective path. Otherwise, while OpenRadioss is solving your workload, you’re paying for a GPU that is sitting idle.

Workflow step 1: OpenRadioss simulation

In this example, we will run an OpenRadioss crash simulation using a 2019 generic mid-size ADS vehicle model from CCSA. For the first step in the workflow, we will use an AMD CPU shape with 64 CPU cores to run the OpenRadioss workload.

Create the instance on Oracle Cloud

Create an instance that has 64 cores and 64 GB of memory for the OpenRadioss workload.

This screen appears right after logging in to Oracle Cloud. Click the top left corner to open the menu.

Click Compute > Instances.

Click “Create Instance.”

It shows many options, but first we are going to choose the Rocky Linux image and then the shape of the instance.

Click “Change image.”

Select “Rocky Linux,” and then select the “OpenRadioss and ParaView on Rocky Linux 9.1” from the list.

Click “Change shape,” maximize “Number of OCPUs” to 64 cores, and specify “Amount of memory (GB)” to 64 GB.

This is after you choose the image and shape.

Click “Save private key” and “Save public key” to download keys for SSH. We use the private key to SSH log-in to the instance we created here, and we use the public key when we create other GPU-equipped instances for visualization.

Now you have two keys under the Downloads directory (depending on your environment and how you saved those keys); one is like “ssh-key-yyyy-mm-dd.key.”  This is the private key that we use for SSH log-in, and the other is like “ssh-key-yyyy-mm-dd.key.pub.” This is the public key that we use when we create other instances.

Open your terminal and then type the following to attach read write permission only to the current user; otherwise, SSH log-in fails. This procedure is only required for MacOS and Linux users.

$ chmod 600 ~/Downloads/ssh-key-2023-05-08.key

Click “Create.”

The instance is in a “provisioning” state for usually less than a minute.

When your instance becomes “running” state, you can SSH log-in to this instance. Sometimes you need to wait a bit after an instance becomes a “running” state, due to the cloud-init script still running, etc.

SSH

Let’s SSH log-in to the instance just created. For SSH log-in, we need the IP address of the instance, username, and SSH key. The IP address of the instance is shown on the “Instance access” section of the Instance details page. The username is “rocky” and the SSH key is the private key that we downloaded when we created the instance.

Open your terminal (when you use Windows, it’s “Windows Terminal”; when you use MacOS, it’s “Terminal”), and then type the following to SSH log-in to the instance.

$ ssh -i ~/Downloads/ssh-key-2023-05-08.key rocky@129.153.31.77

This is what you see with a successful SSH log-in to the instance.

Type “htop” and hit Enter.

This shows the instance we just created has 128 cores. This is because AMD Simultaneous Multithreading (SMT) is enabled. We use 64 cores for our simulation, since AMD SMT and Intel HT don't contribute to the performance of OpenRadioss and HPC applications in general.

Type “q” to quit “htop.”

Run the simulation

Preparing input files

Move to “Simulation” directory.

Unarchive “2019-generic-mid-size-ads-vehicle-v1.zip.”

Copy “ADSOpenRadioss.key.”

Move to “2019-generic-mid-size-ads-vehicle-v1" directory.

$ cd Simulation
$ unzip 2019-generic-mid-size-ads-vehicle-v1.zip

Archive:  2019-generic-mid-size-ads-vehicle-v1.zip

2019 Generic Mid-size ADS Vehicle Model

Center for Collision Safety and Analysis

George Mason University

  inflating: 2019-generic-mid-size-ads-vehicle-v1/README.txt

  inflating: 2019-generic-mid-size-ads-vehicle-v1/combine.key

  inflating: 2019-generic-mid-size-ads-vehicle-v1/gmu-ccsa-generic-mid-size-ads-vehicle-v1.key

  inflating: 2019-generic-mid-size-ads-vehicle-v1/wall.key

$ cp ADSOpenRadioss.key 2019-generic-mid-size-ads-vehicle-v1

$ cd 2019-generic-mid-size-ads-vehicle-v1
Decompose simulation input data

Use “tmux,” terminal multiplexer, here. This is convenient when you want to exit your terminal while the simulation is still running or when you lose connection while the simulation is still running. “Tmux” creates a session, and you can detach from that session without stopping your simulation.

$ tmux

Decompose simulation input data for 64 processes MPI parallel simulation.

$ openradioss.sif
Apptainer> starter_linux64_gf -i ADSOpenRadioss.key -np 64
Run simulation

Run the crash simulation. It takes about 40 minutes.

Apptainer> mpirun -np 64 engine_linux64_gf_ompi -i ADSOpenRadioss_0001.rad

This is optional, but let’s check out CPU usage with “htop” for fun. 

Press keys “Ctrl + b” and then press key “d.” This makes you detach from the current session, such as the OpenRadioss simulation terminal session.

Then, type the following and press Enter.

$ htop

This shows the simulation utilizes 64 cores on your instance.

Let’s get back to the OpenRadioss simulation session. Type “q” to quit “htop” and then type the following to attach to the session:

$ tmux attach -t 0

When the simulation is finished, it will show you something like this:

64 cores with “ESTIMATED SPEEDUP" of 63.62 looks like pretty good performance. If you wish to accelerate more, you will need multiple nodes set up, and an HPC cluster. That is indeed an interesting topic, but it is not covered in this article.

Prepare output data for ParaView visualization

OpenRadioss produces visualization data in ANIM format, but ParaView does not have a plugin for reading ANIM format. That means we need to convert ANIM format data to VTK format, which is the primary supported data format by ParaView.

Let’s convert the ANIM format data to VTK format data. It takes about 5 minutes to finish.

$ seq -f ADSOpenRadiossA%03g 076 | xargs -I{} sh -c 'openradioss.sif anim_to_vtk_linux64_gf "$1" > "$1.vtk"' -- {}

If you didn’t finish your simulation during the previous step, don’t worry. The custom image we use here has sample VTK files. Please use VTK files under ~/Visualization if you don’t have your own data.

Terminate the CPU instance

We have done CPU-intensive workloads such as the OpenRadioss simulation, so let’s terminate the instance. Please keep in mind that we will reuse the disk (boot volume) that we just used for the OpenRadioss simulation.

Click “Terminate.”

Please leave the following checkbox unchecked; we would like to use this boot volume that contains data for visualization. We can create a GPU-equipped instance from this boot volume in later steps.

Click “Terminate instance.”

It takes less than a few minutes to be terminated.

After the instance has terminated, please move on to the next step.

Workflow step 2: visualizing with ParaView

Create an instance for the visualization workload from the same boot drive that is used for the OpenRadioss simulations.

Click “Create instance.”

Click “Change image.”

Click “My images,”

and then click “Boot volumes.”

Choose Boot volume and click “Select image.”

Next, click “Change shape.”

This time, we use the NVIDIA GPU-equipped shape. Click “Specialty and previous generation.”

Click the “VM.GPU.A10.1” shape that has a NVIDIA A10 GPU and 15 cores Intel Xeon processor, and then click “Select shape.”

Now it looks like this:

Next, let’s look at SSH key settings. This time we upload the public key that we downloaded when we created an instance for the OpenRadioss simulation.

Click “Upload public key files (.pub),” and then click “Browse.”

Select your public key and then click “open.”

Now it looks like this:

We are ready to go! Click “Create.”

SSH

After the newly created instance becomes the “Running” state, SSH log-in to the instance.

Username is the same as “rocky” and the ssh key is also the same “ssh-key-yyyy-mm-dd.key”; only the IP address is changed here. Please check the new IP address that is assigned to the instance.

$ ssh -i ~/Downloads/ssh-key-2023-05-08.key rocky@150.136.107.70

Optional: Verify the GPU driver is loaded

If you follow these instructions verbatim, you’ll be following a path that has been verified, meaning everything should be working. It’s never a bad idea, though, to do a quick verification that the GPU is up and running. You can use the following command to verify that the NVIDIA driver is running.

$ nvidia-smi

You can also monitor GPU activity using “nvtop”:

$ nvtop

Type “q” to quit “nvtop.”

Start ParaView server

$ pvserver

Client side

Install the ParaView client 

Download the exact same version (v5.11) of the ParaView client from the official download page https://www.paraview.org/download/. You can use Windows, Linux, and MacOS for client side OS.

SSH port forward

Before you connect to the ParaView server on OCI, forward port 11111 on the GPU instance to port 11111 on the local PC using SSH. This way, we don’t need to open port 11111 on the OCI side.

$ ssh -i ~/Downloads/ssh-key-2023-05-08.key -L 11111:localhost:11111 rocky@<INSTANCE IP HERE>

Connect the ParaView client to the ParaView server on OCI

Launch ParaView and click File > Connect.

Connect to “localhost:1111.”

Click “Open.”

Double click “Visualization.”

Select “ADSOpenRadiossA..vtk” and click “OK.”

Click “Apply.”

Now you can see the rendered VTK file.

Set view direction to “-Y.”

Select “2DLEM_Stress_(lower).”

Set Range “0-256.”

Change “Time” to 23.

Lastly, click “ParaView” and then click “About ParaView” when you use MacOS; that shows the GPU device and GPU driver version that the ParaView server running on Oracle cloud resource is being used for this session.

Terminate the GPU instance

After finishing the visualization, let’s terminate the GPU instance. This time, we can delete the boot drive.

Check “Permanently delete the attached boot volume” and then click “Terminate instance.”

Summary

We walked through the car crash simulation to its visualization using the newly published OpenRadioss / ParaView Rocky Linux 9-based image on OCI and appropriate shape for each task.

Appendix A: Attaching extra drive

If you’d like to play with all of the examples in this image, understand that the default boot drive doesn’t have enough space. In this section, we provide instructions on how to create extra drives and attach them to your instance.

Click the top left corner to open the menu, and then click “Storage,” “Block Volumes.”

Click “Create Block Volume.”

Type a name for the block volume and then click “Create Block Volume.” The default size of the block volume is 1024 GB.

Provisioning will take less than a few minutes.

When block volume becomes an available state, we are good to go.

Go back to the instance page and click “Attached block volumes” > “Attach block volume.”

Choose the drive you created.

Click “Attach.”

Attaching the block volumes takes about a minute.

It’s ready!

Click the menu on the right, and click “iSCSI commands & information.”

Copy command to “connect.”

Go back to your terminal and paste the command:

Check that the disk is attached correctly using “lsblk.”

Create a partition using “parted.”

$ sudo parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary 0% 100%
(parted) quit

Create a filesystem using  “mkfs.ext4.”

$ sudo mkfs.ext4 /dev/sdb1

Create a mount point “/data.”

$ sudo mkdir -p /data

Mount the extra drive to “/data.”

$ sudo mount /dev/sdb1 /data

Change permission to 777.

$ sudo chmod 777 /data

Now that you have enough space on “/data” to try all examples under “/Simulations,” copy examples from “/Simulations” to “/data” and then try the examples.

Appendix B: Detaching extra drive

Unmount “/data.”

$ sudo umount /data

Open the instance page, and click “Attached block volumes.” Open the menu and click “iSCSI commands & information.”

Copy the “Disconnect” command.

Paste the “Disconnect” command to terminal.

Click “Detach.”

Appendix C: Using openradioss.sif with /data 

Copy the example from “~/Simulations” to “/data.”

$ cp ~/Simulations/Neon1m11_2017.zip /data

Unarchive the zip file.

$ cd /data
$ unzip Neon1m11_2017.zip
$ cd Neon1m11_2017

Use the Apptainer shell command with bind mount option to bind mount “/data” to openradioss container.

$ apptainer shell -B /data /usr/local/bin/openradioss.sif
Apptainer> pwd
/data/Neon1m11_2017
Apptainer>

Running a Neon 1M benchmark example takes about an hour to finish. 

Apptainer> starter_linux64_gf -i NEON1M11_0000.rad -np 64
Apptainer> mpirun -np 64 engine_linux64_gf_ompi -i NEON1M11_0001.rad

When the simulation is finished, you will see an output similar to this:

                  ** CUMULATIVE CPU TIME SUMMARY **

CONTACT SORTING.............: .1063E+05     8.07 % 
CONTACT FORCES..............: .1297E+04     0.98 % 
ELEMENT FORCES..............: .7143E+05    54.23 % 
KINEMATIC COND..............: .4094E+04     3.11 % 
INTEGRATION.................: .1015E+05     7.71 % 
ASSEMBLING..................: .1563E+05    11.87 % 
OTHERS (including I/O)......: .1849E+05    14.04 % 
TOTAL.......................: .1317E+06   100.00 % 

                   ** MEMORY USAGE STATISTICS **

 TOTAL MEMORY USED .........................:    12585 MB
 MAXIMUM MEMORY PER PROCESSOR...............:      246 MB
 MINIMUM MEMORY PER PROCESSOR...............:      184 MB
 AVERAGE MEMORY PER PROCESSOR...............:      196 MB

                   ** DISK USAGE STATISTICS **

 TOTAL DISK SPACE USED .....................:       3086 MB
 ANIMATION/H3D/TH/OUTP SIZE ................:        719 MB
 RESTART FILE SIZE .........................:       2366 MB

 ELAPSED TIME     =       2069.68 s
                          0:34:29
 ESTIMATED SPEEDUP=         63.64

     NORMAL TERMINATION      
     USER BREAK 
     TOTAL NUMBER OF CYCLES  :  100001
Apptainer>

Running the Taurus 10M benchmark example takes about an hour to finish.

Apptainer> cp ~/Simulations/Taurus10M.zip /data
Apptainer> cd /data
Apptainer> unzip Taurus10M.zip
Apptainer> cd T10M
Apptainer> sed -i '/^\/RUN/a 0.00201' TAURUS_A05_FFB50_0001.rad
Apptainer> starter_linux64_gf -i TAURUS_A05_FFB50_0000.rad -np 64
Apptainer> mpirun -np 64 engine_linux64_gf_ompi -i TAURUS_A05_FFB50_0001.rad

When the simulation is finished, you will see an output similar to this:

                  ** CUMULATIVE CPU TIME SUMMARY **

CONTACT SORTING.............: .2968E+05    14.60 % 
CONTACT FORCES..............: .2441E+05    12.01 % 
ELEMENT FORCES..............: .9330E+05    45.91 % 
KINEMATIC COND..............: .2055E+04     1.01 % 
INTEGRATION.................: .1684E+05     8.28 % 
ASSEMBLING..................: .1068E+05     5.25 % 
OTHERS (including I/O)......: .2628E+05    12.93 % 
TOTAL.......................: .2032E+06   100.00 % 

                   ** MEMORY USAGE STATISTICS **

TOTAL MEMORY USED .........................:    59767 MB
MAXIMUM MEMORY PER PROCESSOR...............:     4070 MB
MINIMUM MEMORY PER PROCESSOR...............:      831 MB
AVERAGE MEMORY PER PROCESSOR...............:      933 MB

                   ** DISK USAGE STATISTICS **

TOTAL DISK SPACE USED .....................:      17730 MB
ANIMATION/H3D/TH/OUTP SIZE ................:       1202 MB
RESTART FILE SIZE .........................:      16528 MB

ELAPSED TIME     =       3226.62 s
                         0:53:46
ESTIMATED SPEEDUP=         62.99

     NORMAL TERMINATION      
     TOTAL NUMBER OF CYCLES  :   10103

Running a Toyota Camry model takes about 2 hours to finish.

Apptainer> cp ~/Simulations/2012-toyota-camry-detailed-v5a.zip /data
Apptainer> cp ~/Simulations/CamryOpenRadioss.key
Apptainer> cd /data
Apptainer> unzip 2012-toyota-camry-detailed-v5a.zip
Apptainer> cp CamryOpenRadioss.key 2012-toyota-camry-detailed-v5a
Apptainer> cd 2012-toyota-camry-detailed-v5a
Apptainer> starter_linux64_gf -i CamryOpenRadioss.key -np 64
Apptainer> mpirun -np 64 engine_linux64_gf_ompi -i CamryOpenRadioss_0001.rad

When the simulation is finished, you will see an output similar to this:

                  ** CUMULATIVE CPU TIME SUMMARY **

CONTACT SORTING.............: .5613E+05    10.56 % 
CONTACT FORCES..............: .1404E+06    26.40 % 
..INCLUDING CONTACT NORMALS:  .6792E+05    12.77 % 
ELEMENT FORCES..............: .1793E+06    33.71 % 
KINEMATIC COND..............: .1355E+05     2.55 % 
INTEGRATION.................: .2393E+05     4.50 % 
ASSEMBLING..................: .3340E+05     6.28 % 
OTHERS (including I/O)......: .8511E+05    16.01 % 
TOTAL.......................: .5318E+06   100.00 % 

                   ** MEMORY USAGE STATISTICS **

TOTAL MEMORY USED .........................:    30993 MB
MAXIMUM MEMORY PER PROCESSOR...............:      563 MB
MINIMUM MEMORY PER PROCESSOR...............:      453 MB
AVERAGE MEMORY PER PROCESSOR...............:      484 MB

                   ** DISK USAGE STATISTICS **

TOTAL DISK SPACE USED .....................:       4958 MB
ANIMATION/H3D/TH/OUTP SIZE ................:       4958 MB
RESTART FILE SIZE .........................:          0 MB
 
ELAPSED TIME     =       8324.31 s
                          2:18:44
 ESTIMATED SPEEDUP=         63.88

     NORMAL TERMINATION      
     USER BREAK 
     TOTAL NUMBER OF CYCLES  :   84061

Running a Toyota Yaris model takes about 3 hours to finish.

Apptainer> cp ~/Simulations/2010-toyota-yaris-detailed-v2j.zip /data
Apptainer> cp ~/Simulations/YarisOpenRadioss.key
Apptainer> cd /data
Apptainer> unzip 2010-toyota-yaris-detailed-v2j.zip
Apptainer> cp YarisOpenRadioss.key 2010-toyota-yaris-detailed-v2j
Apptainer> cd 2010-toyota-yaris-detailed-v2j
Apptainer> starter_linux64_gf -i YarisOpenRadioss.key -np 64
Apptainer> mpirun -np 64 engine_linux64_gf_ompi -i YarisOpenRadioss_0001.rad

When the simulation is finished, you will see an output similar to this:

                  ** CUMULATIVE CPU TIME SUMMARY **

CONTACT SORTING.............: .8854E+05    11.71 % 
CONTACT FORCES..............: .1441E+06    19.06 % 
..INCLUDING CONTACT NORMALS:  .8145E+05    10.77 % 
ELEMENT FORCES..............: .2944E+06    38.94 % 
KINEMATIC COND..............: .1924E+05     2.54 % 
INTEGRATION.................: .2373E+05     3.14 % 
ASSEMBLING..................: .5593E+05     7.40 % 
OTHERS (including I/O)......: .1301E+06    17.20 % 
TOTAL.......................: .7561E+06   100.00 % 

                   ** MEMORY USAGE STATISTICS **

TOTAL MEMORY USED .........................:    25495 MB
MAXIMUM MEMORY PER PROCESSOR...............:      450 MB
MINIMUM MEMORY PER PROCESSOR...............:      375 MB
AVERAGE MEMORY PER PROCESSOR...............:      398 MB

                   ** DISK USAGE STATISTICS **

TOTAL DISK SPACE USED .....................:       9194 MB
ANIMATION/H3D/TH/OUTP SIZE ................:       5386 MB
RESTART FILE SIZE .........................:       3808 MB

 ELAPSED TIME     =      11840.29 s
                          3:17:20
 ESTIMATED SPEEDUP=         63.86

     NORMAL TERMINATION      
     TOTAL NUMBER OF CYCLES  :  200848

If you would like a pdf version of this tutorial to share with your colleagues, please provide your email address below, and we’ll gladly send you a copy.

Related posts

2023 Holiday Gift Guide for Rocky Linux Users

2023 Holiday Gift Guide for Rocky Linux Users

Dec 19, 2023

Rocky Linux

Why Rocky Linux Is a Rock-Solid Choice in an Economic Downturn

Why Rocky Linux Is a Rock-Solid Choice in an Economic Downturn

Jan 18, 2023

Rocky Linux

6 Signs That It's Time to Move to Rocky Linux

6 Signs That It's Time to Move to Rocky Linux

Feb 23, 2023

Rocky Linux

123
39
>>>