Difference between revisions of "MPI Libraries"

From HPC users
Jump to navigationJump to search
(Created page with "On the HPC cluster, we have two different kinds of MPI libraries: '''OpenMPI''' and ''intel MPI'' ('''impi''') The [http://www.open-mpi.org OpenMPI] library is an open source...")
 
(added hpc-env/6.4)
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
On the HPC cluster, we have two different kinds of MPI libraries:
On the HPC cluster, there are two different kinds of MPI libraries:
'''OpenMPI''' and ''intel MPI'' ('''impi''')
'''OpenMPI''' and ''Intel MPI'' ('''impi''')


The [http://www.open-mpi.org OpenMPI] library is an open source implementation for the MPI protocol.  
The [http://www.open-mpi.org OpenMPI] library is an open source implementation for the MPI protocol.  
Line 8: Line 8:
  module avail openmpi
  module avail openmpi
  module avail impi
  module avail impi
The currently installed versions are available...
... on envirnoment ''hpc-uniol-env'':
'''OpenMPI/1.10.4'''-GCC-4.9.4
'''OpenMPI/2.0.1'''-GCC-5.4.0
'''OpenMPI/2.0.1'''-GCC-6.2.0
'''OpenMPI/2.0.2'''-GCC-5.4.0-2.26
'''impi/5.1.3.210'''
...on environment ''hpc-env/6.4'':
'''OpenMPI/2.1.0'''-iccifort-2018a
'''OpenMPI/2.1.1'''-GCC-6.4.0-2.28
'''impi/2018a'''-iccifort-2018a
'''impi/2018.1.163'''-GCC-6.4.0-2.28


For performance reasons the latest release of OpenMPI / impi should be used.
For performance reasons the latest release of OpenMPI / impi should be used.
The following instructions and examples are based on the environment '''hpc-env/6.4''', which is currently our environment for the newest stable software versions that we install.
Therefor we encourage you to use this environment when working with MPI.


== Compiling with OpenMPI ==
== Compiling with OpenMPI ==
Before compiling [[User environment - The usage of module|load the actual module]] of OpenMPI, e.g.  
Before compiling [[User environment - The usage of module|load the actual module]] of OpenMPI, e.g.  


  module load openmpi/1.8.4/gcc
  module load OpenMPI/2.1.1-GCC-6.4.0-2.28


for using the GNU compiler.
for using the GNU compiler.
Line 42: Line 63:
These programs are only wrappers which means that the scripts sets additional flags for OpenMPI (e.g. include path, flags for linking OpenMPI libraries, ...). For using the Intel Compiler please use the module  
These programs are only wrappers which means that the scripts sets additional flags for OpenMPI (e.g. include path, flags for linking OpenMPI libraries, ...). For using the Intel Compiler please use the module  


  module load openmpi/1.8.4/intel
  module load OpenMPI/2.1.0-iccifort-2018a


Below there is a list of all environment variables for setting other compiler.
Below there is a list of all environment variables for setting other compiler.
Line 66: Line 87:
|}
|}
</center>
</center>
However, in general the module's toolchain (as indicated by the version suffix, e.g iccifort-2018a) already defines the compiler that will used by the wrappers listed above. In principle it should be possible to change the default behaviour but this is not recommended. In case of Fortran 90 or newer, the module files will most likely cause problems.


== Run parallel programs ==
== Compiling with Intel MPI ==
The typical call to launch a MPI program within an SGE script is
Before compiling an MPI program you need to [[User environment - The usage of module|load the actual module]] for Intel MPI, e.g. with the commands
 
  mpirun -machinefile $TMPDIR/machines -np $NSLOTS <MPI_program> <MPI_program_options>
 
Please don't forget to load the correct OpenMPI module before (the same OpenMPI module which was used for compilation)!


On FLOW the communication will be done over InfiniBand (automatically). Due to new virtual nodes without InfiniBand the explicit setting of InfiniBand usage by
$ module load hpc-env/6.4
setting the environment variable ''OMPI_MCA_btl'' by
$ module load impi/2018a-iccifort-2018a


  export OMPI_MCA_btl="openib,sm,self"
Other versions of Intel MPI are available, you may have to load a different environment before you can load them (use <tt>module spider impi</tt> to find out which versions exist and which environments have to be loaded). Note, that the following has been tested for <tt>impi/2018a-iccifort-2018a</tt>, other versions may behave slightly different.


or by using the ''mpirun'' or ''mpiexec'' command line option
In general, you can use Intel MPI with the Intel compilers '''or''' the GNU compilers. Both families of compilers are loaded together with the Intel MPI module, and it is recommended that use the versions that are loaded automatically. Compilation of your MPI program is done using a wrapper command, e.g.


  mpirun -mca btl "openib,sm,self" ...
  $ mpicc MPI_HelloWorld.c -o MPI_HelloWorld


is depricated and can causes problems on the ''vx*'' nodes!
for a C-program. These wrapper commands set the required additional flags for Intel MPI (e.g. include path, flags for linking Intel MPI libraries, ...) which are then used by the underlying compiler (e.g. GNU compiler, Intel Compiler), in addition to any other options given to the wrapper. The available wrapper commands are listed below (and also [https://software.intel.com/en-us/mpi-developer-reference-linux-compilation-commands here]). '''Note:''' the actual compiler called by the standard wrappers is from the GNU compiler family. You can verify this by typing e.g.:


$ mpicc
This script invokes an appropriate specialized C MPI compiler driver.
The following ways (priority order) can be used for changing default
compiler name (gcc):
  1. Command line option:  -cc=<compiler_name>
  2. Environment variable: I_MPI_CC (current value '')
  3. Environment variable: MPICH_CC (current value '')


Alternatively, and to get more details, you can also use the command


== Compiling with Intel MPI ==
$ mpicc --version
Before compiling [[User environment - The usage of module|load the actual module]] of Intel MPI by


module load impi/5.0.0.028/32/intel
(likewise for other wrappers) which will give the name and version of the underlying compiler. In the first command (w/o <tt>--version</tt> you can also see how to change the default behaviour of the wrapper. Therefore,


for Intel and
$ mpicc -cc=icc ...


module load impi/5.0.0.028/64/gcc
would compile your program using the Intel C compiler.


for the GNU compiler. The compilation can be done by following wrappers of Intel MPI:
The following standard wrappers of Intel MPI can be used for compilation of your MPI programs:
<center>
<center>
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"  
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"  
|- style="background-color:#ddddff;"
|- style="background-color:#ddddff;"
! Name
! Name
!Description
! Description
! Default compiler
! Compiler cmd-line option
! Compiler ENV variable
|-
|-
| ''mpicc''
| ''mpicc''
| C compiler
| C compiler
| <tt>gcc</tt>
| <tt>-cc=<compiler_name></tt>
| <tt>I_MPI_CC</tt>
|-
|-
| ''mpicxx''
| ''mpicxx''
| C++ compiler
| C++ compiler
| <tt>g++</tt>
| <tt>-cxx=<compiler_name></tt>
| <tt>I_MPI_CXX</tt>
|-
|-
| ''mpifc''
| ''mpifc''
| Fortran compiler
| Fortran compiler
| <tt>gfortran</tt>
| <tt>-fc=<compiler_name></tt>
| <tt>I_MPI_FC</tt>
|-
|-
| ''mpif77''
| ''mpif77''
| Fortran 77 compiler
| Fortran 77 compiler
| <tt>gfortran</tt>
| <tt>-f77=<compiler_name></tt>
| <tt>I_MPI_F77</tt>
|-
|-
| ''mpif90''
| ''mpif90''
| Fortran 90 compiler
| Fortran 90 compiler
| <tt>gfortran</tt>
| <tt>-f90=<compiler_name></tt>
| <tt>I_MPI_F90</tt>
|-
|-
|}
|}
</center>
</center>


These programs are only wrappers which means that the scripts sets additional flags for Intel MPI (e.g. include path, flags for linking Intel MPI libraries, ...) use other compilers (e.g. GNU compiler, Intel Compiler). The compilers can be choosen by setting environment variables, e.g.
As you can see, each of the wrappers above used a GCC default compiler. This can be changed by chosing the compiler using a cmd-line option as shown above or by setting the corresponding environment variables, e.g.


   export I_MPI_CC=icc
   export I_MPI_CC=icc


for using the Intel C compiler. Below there is a list of all environment variables for setting the compiler.
for using the Intel C compiler.  


<center>
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"
|- style="background-color:#ddddff;"
! Environment variable
!Description
|-
| ''I_MPI_CC''
| Set the C compiler for the ''mpicc'' script
|-
| ''I_MPI_CXX''
| Sets the C++ compiler for the ''mpicxx'' script
|-
| ''I_MPI_FC''
| Sets the Fortran for the ''mpifc'' script
|-
| ''I_MPI_F77''
| Sets the Fortran 77 compiler for the ''mpif77'' script
|-
| ''I_MPI_F90''
| Sets the Fortran 90 compiler for the ''mpif90'' script
|-
|}
</center>
Alternatively for GNU compilers and Intel Compilers there exists following wrapper scripts which needs no special environment variables settings.
Alternatively for GNU compilers and Intel Compilers there exists following wrapper scripts which needs no special environment variables settings.


Line 173: Line 193:
|}
|}
</center>
</center>
'''Important note for Fortran 90/2008''': Newer Fortran standards can use special module files. Intel MPI in general supports the Fortran 90/95 standard (if the underlying compiler supports it which is usually the case nowadays). Therefore, you can include the MPI module with
use mpi
If you are using the Intel compiler <tt>ifort</tt> you can also use the new Fortran 2008 standard with
use mpi_f08
which is recommended (mainly for code robustness). This standard is '''not''' supported for the combination Intel MPI and GCC compiler (see [https://software.intel.com/en-us/articles/intel-mpi-library-release-notes-linux also here])

Latest revision as of 12:29, 28 June 2019

On the HPC cluster, there are two different kinds of MPI libraries: OpenMPI and Intel MPI (impi)

The OpenMPI library is an open source implementation for the MPI protocol. The Intel MPI library is an optimized implementation for the MPI protocol. Different versions compiled with different compilers are available. A list is given by typing

module avail openmpi
module avail impi

The currently installed versions are available...

... on envirnoment hpc-uniol-env:

OpenMPI/1.10.4-GCC-4.9.4
OpenMPI/2.0.1-GCC-5.4.0
OpenMPI/2.0.1-GCC-6.2.0
OpenMPI/2.0.2-GCC-5.4.0-2.26

impi/5.1.3.210

...on environment hpc-env/6.4:

OpenMPI/2.1.0-iccifort-2018a
OpenMPI/2.1.1-GCC-6.4.0-2.28

impi/2018a-iccifort-2018a
impi/2018.1.163-GCC-6.4.0-2.28


For performance reasons the latest release of OpenMPI / impi should be used. The following instructions and examples are based on the environment hpc-env/6.4, which is currently our environment for the newest stable software versions that we install. Therefor we encourage you to use this environment when working with MPI.

Compiling with OpenMPI

Before compiling load the actual module of OpenMPI, e.g.

module load OpenMPI/2.1.1-GCC-6.4.0-2.28

for using the GNU compiler.

The compilation can be done by following wrappers of OpenMPI:

Name Description
mpicc C compiler
mpic++, mpiCC or mpicxx C++ compiler
mpif77 Fortran 77 compiler
mpif90 Fortran 90 compiler

These programs are only wrappers which means that the scripts sets additional flags for OpenMPI (e.g. include path, flags for linking OpenMPI libraries, ...). For using the Intel Compiler please use the module

module load OpenMPI/2.1.0-iccifort-2018a

Below there is a list of all environment variables for setting other compiler.

Environment variable Description
OMPI_CC Set the C compiler
OMPI_CXX Sets the C++ compiler
OMPI_F77 Sets the Fortran 77 compiler
OMPI_FC Sets the Fortran 90 compiler

However, in general the module's toolchain (as indicated by the version suffix, e.g iccifort-2018a) already defines the compiler that will used by the wrappers listed above. In principle it should be possible to change the default behaviour but this is not recommended. In case of Fortran 90 or newer, the module files will most likely cause problems.

Compiling with Intel MPI

Before compiling an MPI program you need to load the actual module for Intel MPI, e.g. with the commands

$ module load hpc-env/6.4
$ module load impi/2018a-iccifort-2018a

Other versions of Intel MPI are available, you may have to load a different environment before you can load them (use module spider impi to find out which versions exist and which environments have to be loaded). Note, that the following has been tested for impi/2018a-iccifort-2018a, other versions may behave slightly different.

In general, you can use Intel MPI with the Intel compilers or the GNU compilers. Both families of compilers are loaded together with the Intel MPI module, and it is recommended that use the versions that are loaded automatically. Compilation of your MPI program is done using a wrapper command, e.g.

$ mpicc MPI_HelloWorld.c -o MPI_HelloWorld

for a C-program. These wrapper commands set the required additional flags for Intel MPI (e.g. include path, flags for linking Intel MPI libraries, ...) which are then used by the underlying compiler (e.g. GNU compiler, Intel Compiler), in addition to any other options given to the wrapper. The available wrapper commands are listed below (and also here). Note: the actual compiler called by the standard wrappers is from the GNU compiler family. You can verify this by typing e.g.:

$ mpicc
This script invokes an appropriate specialized C MPI compiler driver.
The following ways (priority order) can be used for changing default
compiler name (gcc):
  1. Command line option:  -cc=<compiler_name>
  2. Environment variable: I_MPI_CC (current value )
  3. Environment variable: MPICH_CC (current value )

Alternatively, and to get more details, you can also use the command

$ mpicc --version

(likewise for other wrappers) which will give the name and version of the underlying compiler. In the first command (w/o --version you can also see how to change the default behaviour of the wrapper. Therefore,

$ mpicc -cc=icc ...

would compile your program using the Intel C compiler.

The following standard wrappers of Intel MPI can be used for compilation of your MPI programs:

Name Description Default compiler Compiler cmd-line option Compiler ENV variable
mpicc C compiler gcc -cc=<compiler_name> I_MPI_CC
mpicxx C++ compiler g++ -cxx=<compiler_name> I_MPI_CXX
mpifc Fortran compiler gfortran -fc=<compiler_name> I_MPI_FC
mpif77 Fortran 77 compiler gfortran -f77=<compiler_name> I_MPI_F77
mpif90 Fortran 90 compiler gfortran -f90=<compiler_name> I_MPI_F90

As you can see, each of the wrappers above used a GCC default compiler. This can be changed by chosing the compiler using a cmd-line option as shown above or by setting the corresponding environment variables, e.g.

 export I_MPI_CC=icc

for using the Intel C compiler.

Alternatively for GNU compilers and Intel Compilers there exists following wrapper scripts which needs no special environment variables settings.

Wrapper for Intel Compiler Wrapper for GNU Compiler Description
mpiicc mpigcc C compiler
mpiicpc mpigxx C++ compiler
mpiifort Fortran 77 compiler

Important note for Fortran 90/2008: Newer Fortran standards can use special module files. Intel MPI in general supports the Fortran 90/95 standard (if the underlying compiler supports it which is usually the case nowadays). Therefore, you can include the MPI module with

use mpi

If you are using the Intel compiler ifort you can also use the new Fortran 2008 standard with

use mpi_f08

which is recommended (mainly for code robustness). This standard is not supported for the combination Intel MPI and GCC compiler (see also here)