Python 2016

Introduction

Python is a widely used high-level programming language for general-purpose programming, created by Guido van Rossum and first released in 1991. An interpreted language, Python has a design philosophy which emphasizes code readability (notably using whitespace indentation to delimit code blocks rather than curly braces or keywords), and a syntax which allows programmers to express concepts in fewer lines of code than possible in languages such as C++ or Java. The language provides constructs intended to enable writing clear programs on both a small and large scale.

Installed version

The following versions of Python are currently installed on the cluster:

Installed version

The currently installed versions are
On environment hpc-uniol-env
Python/2.7.3-goolf-5.2.01
Python/2.7.11-intel-2016b
Python/2.7.12
Python/3.5.2
Python/3.6.1-goolf-5.2.01

 On environment hpc-env/6.4
Python/2.7.14-foss-2017b
Python/2.7.14-intel-2018a
Python/3.6.3-foss-2017b
Python/3.6.3-intel-2018a
Python/3.7.0-foss-2017b
Python/3.7.0-intel-2018a

Further, there are modules for Biopython and ScientificPython (version:2.9.4 based on Python 2.7.12) installed. Biopython is a set of freely available tools for biological computation written in python. It is distributed to develop python libraries and applications which address the needs of current and future work in bioinformatics. Scientificpython is a collection of python modules for scientific computing. It contains support for geometry, mathematical functions, statistics, pyhsical units, IO, visualization and parallelization.

There are even more additional packages installed:

• numpy
• scipy
• nose
• ..

A complete list, including the version numbers of the installed packages can be obtained with the command

pip list

Python has to be loaded in order for this command to work. If any packages that you need are missing, you can either contact the Scientific Computing and we will install it or you can install it on your own in your $HOME-directory (instructions at the end of this article!).

Using Python

If you want to use python on the cluster, you can simply do that by using the following command

module load Python

Since there is more than one verson of Python installed, you can further specify which version you would like to load, e.g.

module load Python/2.7.12 

This will, obviously, load python in version 2.7.12. Loading Python without specifying a version will load version 3.5.2!

On the other hand, if you want to use the latest version of Python, you have to change the environment first.

module load hpc-env/6.4

After that, you just have to choose which Toolchain should be underlying: You can either use Foss or Intel:

module load Python/3.7.0-intel-2018a
or
module load Python/3.7.0-foss-2017b

Using Python and MPI

For parallel scripts the Python installation contains the package mpi4py. To launch a parallel Python script inside an SLURM jobscript, use command line

mpirun YOUR_PYTHON_SCRIPT.py

An easy example could look like this:

1. Create a new file called hello_world.py and add the following lines of code:

from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
print "hello world from process ", rank

2. Create a new file called hello_world.job and add the following lines of code:

#!/bin/bash
               
#SBATCH --ntasks=1                  
#SBATCH --mem=2G                  
#SBATCH --time=0-2:00  
#SBATCH --job-name PYTHON-MPI-TEST              
#SBATCH --output=python-mpi-test.%j.out        
#SBATCH --error=python-mpi-test.%j.err          
 
module load Python

mpirun -n 5 python hello_world.py

3. Make sure both files are in the same directory

4. Submit the job with the command

sbatch -p carl.p hello_world.job

Installing Packages in your HOME-Directory

Using pip

Most Python packages can be installed easily using the pip-command. In that case, the --user-option allows a local user based installation. E.g., to install the package PYPACKAGE use the command (after loading the python module):

 pip install --user PYPACKAGE

If the installation is successful, the corresponding files are installed in

 $HOME/.local/lib/pythonx.y/site-packages

where x.y corresponds to the Python version currently loaded. Also, pip list should show the new package in the list. Note, that you can only manage one set of package this way. If you want to use different environments, for example because different packages require different versions of the same dependency, you need to use e.g. virtual environments (venv, see below) or even conda (see Anaconda).

Virtual Environments

The venv module supports creating lightweight virtual environments, each with their own independent set of Python packages (see venv-documentation for more details). To use venv you can run the following commands:

module load hpc-env/8.3                        # optional, select the hpc-env you want to use to load Python
module load Python                             # load the most recent Python available in hpc-env
python -m venv $HOME/venvs/my_env              # create a virtual environment in the given path
source $HOME/venvs/my_env/bin/activate         # activate the environment
python -m pip install -U pip setuptools wheel  # update pip, setuptools, and wheel

After that you are ready to install the packages you need with pip. For example, if you want to install scipy (although, we do have a module for that called SciPy-bundle):

python -m pip install scipy

This will automatically install all required dependencies as well. Other useful commands are

python -m pip list                             # show all installed packages
python -m pip uninstall scipy                  # uninstall packages
deactivate                                     # leave the virtual environment

Note that there is a limitation of the number of files you can have in $HOME, so you should not install too many virtual environments in your account. If you no longer need an environment, you can completely remove it by deleting the corresponding folder:

rm -r $HOME/venvs/my_env

If you do this accidentally, you can still recover the directory from the snapshots for up to 30 days.

Documentation

The full documentation of Python can be found on the following website: Python Documentation