MPI Libraries

On the HPC cluster, there are two different kinds of MPI libraries: OpenMPI and Intel MPI (impi)

The OpenMPI library is an open source implementation for the MPI protocol. The Intel MPI library is an optimized implementation for the MPI protocol. Different versions compiled with different compilers are available. A list is given by typing

module avail openmpi
module avail impi

For performance reasons the latest release of OpenMPI / impi should be used. The following instructions and examples are based on the environment hpc-env/6.4, which is currently our environment for the newest stable software versions that we install. Therefor we encourage you to use this environment when working with MPI.

Compiling with OpenMPI

Before compiling load the actual module of OpenMPI, e.g.

module load OpenMPI/2.1.1-GCC-6.4.0-2.28

for using the GNU compiler.

The compilation can be done by following wrappers of OpenMPI:

Name	Description
mpicc	C compiler
mpic++, mpiCC or mpicxx	C++ compiler
mpif77	Fortran 77 compiler
mpif90	Fortran 90 compiler

These programs are only wrappers which means that the scripts sets additional flags for OpenMPI (e.g. include path, flags for linking OpenMPI libraries, ...). For using the Intel Compiler please use the module

module load OpenMPI/2.1.0-iccifort-2018a

Below there is a list of all environment variables for setting other compiler.

Environment variable	Description
OMPI_CC	Set the C compiler
OMPI_CXX	Sets the C++ compiler
OMPI_F77	Sets the Fortran 77 compiler
OMPI_FC	Sets the Fortran 90 compiler

However, in general the module's toolchain (as indicated by the version suffix, e.g iccifort-2018a) already defines the compiler that will used by the wrappers listed above. In principle it should be possible to change the default behaviour but this is not recommended. In case of Fortran 90 or newer, the module files will most likely cause problems.

Compiling with Intel MPI

Before compiling an MPI program you need to load the actual module for Intel MPI, e.g. with the commands

$ module load hpc-env/6.4
$ module load impi/2018a-iccifort-2018a

Other versions of Intel MPI are available, you may have to load a different environment before you can load them (use module spider impi to find out which versions exist and which environments have to be loaded). Note, that the following has been tested for impi/2018a-iccifort-2018a, other versions may behave slightly different.

In general, you can use Intel MPI with the Intel compilers or the GNU compilers. Both families of compilers are loaded together with the Intel MPI module, and it is recommended that use the versions that are loaded automatically. Compilation of your MPI program is done using a wrapper command, e.g.

$ mpicc MPI_HelloWorld.c -o MPI_HelloWorld

for a C-program. The wrapper will call one of the usual compilers (as used for non-MPI programs) and passes any given options to that compiler (i.e. you can simply change the compile command to e.g. mpicc). The available wrapper commands are listed below (and also here). Note: the actual compiler called by the standard wrappers is from the GNU compiler family. You can verify this by typing e.g.:

$ mpicc
This script invokes an appropriate specialized C MPI compiler driver.
The following ways (priority order) can be used for changing default
compiler name (gcc):
  1. Command line option:  -cc=<compiler_name>
  2. Environment variable: I_MPI_CC (current value )
  3. Environment variable: MPICH_CC (current value )

The compilation can be done by following wrappers of Intel MPI:

Name	Description
mpicc	C compiler
mpicxx	C++ compiler
mpifc	Fortran compiler
mpif77	Fortran 77 compiler
mpif90	Fortran 90 compiler

These programs are only wrappers which means that the scripts sets additional flags for Intel MPI (e.g. include path, flags for linking Intel MPI libraries, ...) use other compilers (e.g. GNU compiler, Intel Compiler). The compilers can be choosen by setting environment variables, e.g.

 export I_MPI_CC=icc

for using the Intel C compiler. Below there is a list of all environment variables for setting the compiler.

Environment variable	Description
I_MPI_CC	Set the C compiler for the mpicc script
I_MPI_CXX	Sets the C++ compiler for the mpicxx script
I_MPI_FC	Sets the Fortran for the mpifc script
I_MPI_F77	Sets the Fortran 77 compiler for the mpif77 script
I_MPI_F90	Sets the Fortran 90 compiler for the mpif90 script

Alternatively for GNU compilers and Intel Compilers there exists following wrapper scripts which needs no special environment variables settings.

Wrapper for Intel Compiler	Wrapper for GNU Compiler	Description
mpiicc	mpigcc	C compiler
mpiicpc	mpigxx	C++ compiler
mpiifort		Fortran 77 compiler