Deploying a Dell PowerEdge and Cornelis Omni-Path Cluster with Warewulf

CIQ develops and supports open source software for HPC that simplifies your life as a cluster administrator. I recently had the opportunity to deploy and test this software in a Dell PowerEdge cluster with Cornelis Omni-Path networking, with equipment provided in generous partnership by Dell and Cornelis. Both PowerEdge and Omni-Path are frequently deployed in the HPC space due to their performance, reliability, and cost-effectiveness, so they provide a good platform to showcase how Warewulf makes it easy to deploy a cohesive HPC cluster at any scale.

Setting up the control node

The control node is a Dell PowerEdge R6525 equipped with 2x AMD EPYC 73F3 CPUs, 512GiB system memory, and a 100Gb Cornelis Omni-Path HFI. In this cluster, it runs the Warewulf server, fabric manager, and Slurm controller, starting from a Rocky Linux 8.6 base.

Installing Warewulf

Installing Warewulf v4 on the control node is a simple three-step process:

• Install the Warewulf package.

• Update the Warewulf configuration file.

• Direct Warewulf to configure the system.

# dnf install https://github.com/hpcng/warewulf/releases/download/v4.4.1/warewulf-4.4.1-1.git_d6f6fed.el8.x86_64.rpm
# vi /etc/warewulf/warewulf.conf
# wwctl configure -a

Most of the work in that process goes into the configuration file; but, even then, there’s not much that is required. For reference, here’s the entirety of our warewulf.conf:

WW_INTERNAL: 43
ipaddr: 10.10.10.30
netmask: 255.255.255.0
network: 10.10.10.0
warewulf:
  port: 9873
  secure: false
  update interval: 60
  autobuild overlays: true
  host overlay: true
  syslog: false
dhcp:
  enabled: true
  template: default
  range start: 10.10.10.200
  range end: 10.10.10.209
  systemd name: dhcpd
tftp:
  enabled: true
  systemd name: tftp
nfs:
  enabled: true
  export paths:
path: /home
    export options: rw,sync
    mount options: defaults
    mount: true
path: /opt
    export options: ro,sync,no_root_squash
    mount options: defaults
    mount: true

The first step is to define ipaddr, netmask, and network. These three parameters configure the network for the control node itself and define the network and network interface that Warewulf uses to provision the cluster nodes.

Also important is dhcp range start and range end. These define the addresses that Warewulf uses during PXE provisioning. (Of particular note: these are not the addresses used by cluster nodes after they have been provisioned: nodes only use addresses from this range when they are downloading their node image.)

The Warewulf configuration file enables dhcp, tftp, and nfs by default, and we use these default settings in this test cluster. These services are then automatically configured by Warewulf using wwctl configure -a.

The fabric manager

The control and cluster nodes are connected via a single externally-managed 24-port Omni-Path 100 switch, and the control node serves as the fabric manager for the OPA network.

The OPA fabric manager software is available as a standard part of Rocky Linux.

# dnf install opa-fm
# systemctl enable –now opafm
# systemctl status opafm opa-basic-tools
# opainfo
hfi1_0:1                           PortGID:0xfe80000000000000:001175010178e59f
   PortState:     Active
   LinkSpeed      Act: 25Gb         En: 25Gb        
   LinkWidth      Act: 4            En: 4           
   LinkWidthDnGrd ActTx: 4  Rx: 4   En: 3,4         
   LCRC           Act: 14-bit       En: 14-bit,16-bit,48-bit       Mgmt: True 
   LID: 0x00000001-0x00000001       SM LID: 0x00000001 SL: 0 
         QSFP Copper,       3m  Hitachi Metals    P/N IQSFP26C-30       Rev 02
   Xmit Data:               2658 MB Pkts:             15940870
   Recv Data:              11217 MB Pkts:             15170908
   Link Quality: 5 (Excellent)

Configuring Slurm

The control node is also running a Slurm controller from packages provided by OpenHPC.

# dnf install http://repos.openhpc.community/OpenHPC/2/CentOS_8/x86_64/ohpc-release-2-1.el8.x86_64.rpm
# dnf install slurm-slurmctld-ohpc
# cp /etc/slurm/slurm.conf{.example,}
# vi /etc/slurm/slurm.conf
# systemctl enable --now slurmctld munge
# scontrol ping
Slurmctld(primary) at admin1.dell-ciq.lan is UP

Configuring Slurm is somewhat out-of-scope for this article, but let’s go over some of the most pertinent directives for this deployment:

NodeName=c[5-8] Sockets=2 CoresPerSocket=64 ThreadsPerCore=1
PartitionName=opa Nodes=c[5-8] MaxTime=24:00:00 State=UP Oversubscribe=EXCLUSIVE
SlurmctldParameters=enable_configless

We’ve defined our four nodes, c[5-8] in Slurm, along with their basic CPU information. These four nodes are added to an opa partition that we can support jobs to. Finally, slurmctld itself is configured to enable “configless” mode, which allows compute nodes to get slurm.conf from the controller when they first connect, without having to distribute the config file to all nodes manually.

Compute nodes

The four compute nodes are a Dell PowerEdge C6525 multi-node server. Each node is equipped with 2x AMD EPYC 7702 CPUs, 256GiB system memory, and a 100Gb Cornelis Omni-Path HFI.

Adding compute nodes to Warewulf

Adding compute nodes to Warewulf is relatively straightforward. Our four nodes are named c[5-8], and each has an IP address for its default interface, its IPMI interface, and its OPA interface. Settings that are the same across all nodes, such as netmask, gateway, and IPMI credentials, are added to the default node profile.

# wwctl node add c5 --discoverable=yes --ipaddr=10.10.10.41 --ipmiaddr=172.29.235.41
# wwctl node add c6 --discoverable=yes --ipaddr=10.10.10.42 --ipmiaddr=172.29.235.42
# wwctl node add c7 --discoverable=yes --ipaddr=10.10.10.43 --ipmiaddr=172.29.235.43
# wwctl node add c8 --discoverable=yes --ipaddr=10.10.10.44 --ipmiaddr=172.29.235.44

# wwctl node set c5 --netname=opa --ipaddr=10.10.12.41
# wwctl node set c6 --netname=opa --ipaddr=10.10.12.42
# wwctl node set c7 --netname=opa --ipaddr=10.10.12.43
# wwctl node set c8 --netname=opa --ipaddr=10.10.12.44

# wwctl profile set default \
> --netmask=255.255.255.0 --gateway=10.10.10.30 \
> --ipmiuser=root --ipmipass=[redacted] \
> --ipminetmask=255.255.255.0 --ipmiinterface=lanplus

# wwctl profile set default --netname=opa --type=InfiniBand \
> --netdev=ib0 --onboot=true --netmask=255.255.255.0

Defining compute node images

One of the best features of Warewulf v4 is its support for OCI container images. This makes it near trivial to build and maintain compute node images for Warewulf clusters.

The compute node image in this environment is built on a community Warewulf Rocky Linux 8 image, and adds support for OpenHPC, Slurm, Apptainer, and OPA.

FROM ghcr.io/hpcng/warewulf-rockylinux:8.6

RUN dnf -y install \
      dnf-plugins-core \
      epel-release \
      http://repos.openhpc.community/OpenHPC/2/CentOS_8/$(uname -m)/ohpc-release-2-1.el8.$(uname -m).rpm \
    && dnf config-manager --set-enabled powertools \
    && dnf -y install \
         apptainer \
         ohpc-base-compute \
         ohpc-slurm-client \
         slurm-libpmi-ohpc \
         lmod-ohpc \
         chrony \
         opa-basic-tools \
         libpsm2 \
    && systemctl enable munge \
    && systemctl enable slurmd \
    && mkdir -p /var/spool/slurm \
    && dnf -y clean all

We could build and push this container to a registry to then import into Warewulf; but Warewulf can also import a saved OCI archive, which Podman can generate directly. Once the container is imported into Warewulf, we can set it as the container for the default node profile.

# podman build --tag compute-slurm-opa:latest compute-slurm-opa/
# podman save compute-slurm-opa:latest -o compute-slurm-opa.tar
# wwctl container import $(readlink -f compute-slurm-opa.tar) compute-slurm-opa
# wwctl profile set default --container=compute-slurm-opa

Overlays

We use a couple of overlays to customize the configuration of the cluster nodes. First, our slurm overlay captures distributes the munge key (used by Slurm for authentication within the cluster) and the defines the configuration server for slurmd.

# wwctl overlay create slurm
# wwctl overlay mkdir slurm /etc
# wwctl overlay mkdir slurm /etc/munge
# wwctl overlay mkdir slurm /etc/sysconfig
# wwctl overlay edit slurm /etc/sysconfig/slurmd
# wwctl overlay show slurm /etc/sysconfig/slurmd
SLURMD_OPTIONS="--conf-server admin1.dell-ciq.lan:6817"
# wwctl overlay import slurm /etc/munge/munge.key
# wwctl overlay list -a slurm 
OVERLAY NAME                   FILES/DIRS  
slurm                          /etc/        
slurm                          /etc/munge/  
slurm                          /etc/munge/munge.key
slurm                          /etc/sysconfig/
slurm                          /etc/sysconfig/slurmd

It’s important for cluster nodes to have synchronized time. Our chrony overlay configures compute nodes to synchronize time with the control node.

# wwctl overlay create chrony
# wwctl overlay import --parents chrony /etc/chrony.conf
# wwctl overlay edit chrony /etc/chrony.conf
# wwctl overlay show chrony /etc/chrony.conf | tail -n1
server admin1.dell-ciq.lan iburst

We add our newly-defined overlays to the default profile and rebuild overlays for the nodes.

# wwctl profile set default --wwinit wwinit,slurm,chrony
# wwctl overlay build

Booting the nodes

Since we configured IPMI when we added the nodes, we can use Warewulf’s IPMI support to power on our nodes remotely. Powering them on one-at-a-time ensures that node discovery correctly assigns the intended name to each discovered node.

# wwctl power on c5
# wwctl power on c6
# wwctl power on c7
# wwctl power on c8

Running a test application

In the end, we’re left with a working Slurm cluster, complete with the ability to run arbitrary containers directly from the internet with Apptainer!

$ srun --partition=opa apptainer run docker://alpine:latest sh -c 'echo $(hostname): Hello, world!'
INFO:    Using cached SIF image
c5: Hello, world!

Alternatively, OpenHPC packages can be installed on the control node and, since /opt is shared via NFS from the control node to cluster nodes, these packages are available at runtime through the lmod system.

$ srun --partition=opa sh -c 'module avail'

-------------------------- /opt/ohpc/pub/modulefiles ---------------------------
   cmake/3.24.2    hwloc/2.7.0         os            prun/2.2
   gnu9/9.4.0      libfabric/1.13.0    pmix/4.2.1    ucx/1.11.2

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

Next steps

This article was originally meant to include a broad comparison of different MPI applications; but in the process of putting that together, I found a lot more interesting information than I wanted to just have here at the end. Be sure to read my next article, "Benchmarking Containerized MPI with Apptainer on Dell PowerEdge and Cornelis Omni-Path."