MPI (Message Passing Interface)

The Message Passing Interface (MPI) is a standardized, portable specification for passing data between processes in a parallel program. It defines a common set of operations that let processes running on different cores and nodes exchange data and synchronize their work, so a single program can spread across dozens, hundreds, thousands, or even millions of CPU cores at once. MPI is one of the two primary paradigms of parallel computing in high-performance computing (HPC), used wherever the pieces of a workload must communicate as they compute.

What is MPI?

MPI is a standard, not a single piece of software. It specifies operations for intra-node and inter-node communication of data between CPU cores, and multiple organizations build implementations that conform to it. Because they all implement the same standard, code written to MPI is portable across the clusters and supercomputers that support it. Common implementations include Open MPI, MPICH (originally from Argonne National Laboratory and the basis for several vendor libraries), and Intel MPI. There are also specialized variants that provide similar operations for communication directly between GPUs and other devices.

Why MPI matters

Many HPC workloads are tightly coupled: the calculation on one node depends on data produced by others, so the processes must trade information repeatedly as the job runs. MPI gives these programs a standard, efficient way to do that, letting a single application harness the combined power of an entire cluster on one problem. This distinguishes MPI from the embarrassingly parallel paradigm, where tasks are independent and need little communication. MPI is the tool of choice when the parts of a problem cannot be solved in isolation.

How MPI works

An MPI program launches many copies of itself as separate processes, each with a unique rank. Processes are grouped into communicators, and they coordinate through a set of well-defined operations:

• Point-to-point communication moves data directly between two processes.
• Collective communication exchanges or aggregates data across a whole group, such as broadcasting from one process to all others or gathering results from all processes.
• Derived data types describe complex data structures for transfer.
• Process topologies map processes onto logical geometries that match a problem's layout.
• Parallel I/O lets many processes read and write shared files concurrently.

MPI also defines communication modes that control when a send completes. Blocking sends wait for the transfer to progress before returning, while non-blocking sends let a process continue computing and overlap communication with useful work. Other modes, such as buffered and ready sends, trade safety for performance in specific situations.

MPI and the interconnect

MPI performance depends heavily on the network beneath it. On large clusters, MPI messages travel over a high-speed interconnect, and performance benefits greatly from RDMA, which moves data between nodes without involving each operating system's networking stack. Without RDMA, the overhead of processing many network requests in the OS would bog down a large MPI job long before the interconnect's full bandwidth could be used. Together, a fast interconnect and RDMA let an MPI application scale to very large core counts while keeping communication overhead low.