R 2016
Introduction
R is a programming language and environment commonly used in statistical computing, data analytics and scientific research.
Installed version
The currently installed versions of R are
on environment hpc-env/8.3:
R/4.0.2-foss-2019b R/4.1.0-foss-2019b-2021.06
on environment hpc-env/6.4:
R/3.4.4-intel-2018a R/3.5.2-intel-2018a R/3.6.1-intel-2018a
on environment hpc-uniol-env:
R/3.3.1
To find all installed versions you can use the command
$ module -r spider ^R/
and to show the available versions in the currently loaded environment use
$ module -r available ^R/
Note, that starting with version 4.1.0 we have changed the way how we provide the almost 1.000 R-packages we install per default. Using any of the modules listed above you will get a similar experience when using R. However, modules with an extra suffix in the form yyyy.mm indicate the year and month of the installation, allowing us to provide updates to the packages without touching the original installation.
Additional installed packages
The R release contains a lot of additional packages. After loading and starting R ("module load R" and simply "R" on the command line), you can generate a list of all of them by using the following commands
ip <- as.data.frame(installed.packages()[,c(1,3:4)]) rownames(ip) <- NULL ip <- ip[is.na(ip$Priority),1:2,drop=FALSE] print(ip, row.names=FALSE)
You will receive a list of every package and its related version. It should look like this:
Package Version
abc 2.1
abc.data 1.0
abind 1.4-3
acepack 1.3-3.3
adabag 4.1
Additional installed R-Modules
In case new R-packages are being requested, the original R-module (e.g.: R/4.0.2-foss-2019b) usually will not be modified, but instead the HPC team installs a new loadable module which contains the required package as well as all necessary dependency-packages. The advantage of this variant is that the integrity of the original R package remains secured (no change of the individual package versions, etc.). The disadvantage here, however, is that the presence of individual installed packages remains hidden to many users, provided that only the R module is loaded.
So, if you do not find the package you need, it can often be worthwhile to search for already installed R-packages. Either by searching specifically for the required package name (ml av <desired_package>), or by searching for the R suffix.
The following command can be used to find all packages that are based on R:
ml av R-
Among others, this will print the module R-bundle-Bioconductor/3.12-foss-2019b-R-4.0.2, which itself provides a variety of other R packages.
Installing your own packages
If your are missing an R-packages you can contact Scientific Computing or, alternatively install the package in your own HOME directory. To do so you should first create a directory on the cluster, e.g. with
$ mkdir -p $HOME/R/lib
This would create a directory 'R' with the subdirectory 'lib' in your HOME folder. Now, we need to create two files and in some lines. To make this as easy as possible, you can just copy the following line into your console:
echo -en "R_LIBS_USER=\"~/R/lib\"\n" >> $HOME/.Renviron
Afterwards, the file .Renvrion should include the following line (if the file did not exit before, it will only contain this line):
$ cat $HOME/.Renviron ... R_LIBS_USER="~/R/lib"
You can choose a different location for installing your R-libraries if you wish (by setting R_LIBS_USER in .Renviron differently). There are also alternative mirrors, set your preferred on in .Rprofile.
Once this is done, you start R on the login node and begin installing packages:
$ R
> install.packages("lme4", repo = "https://ftp.gwdg.de/pub/misc/cran", lib = "~/R/lib")
> library(lme4)
> library(car)
In this example, the packages lme4 will be downloaded from the GWDG CRAN mirror (if you omit the option repo you will be able to select a mirror from a list) and installed in the directory given by the lib-option (if you omit that option, you will be asked if you want to install in a personal folder). Please note that R will not check if a package is already installed and will always reinstall a package by overwriting the previous install (you can program a logic for that into your R programs). You may want to separate package installation from the execution of jobs.
The package lme4 is already installed in the global R-installation (version 1.1-12) whereas the installation above will install a newer version (1.1-13 or newer). When you load the package with library(lme4), the installation in your $HOME folder will be used. You can verify this with the R-command
> sessionInfo() ... lme4_1.1-13 ...
The next call library(car) will load the globally installed package car. So you do not need to install every package in your $HOME, only those packages that are missing or when you require an updated version.
If you encounter problems when installing an R package in your $HOME, for example because a non-R dependency is missing, then please contact Scientific Computing. Also note, that some package may require to use the installer from the BioConductor package.
Using multiple R versions
In case you are planning to use multiple R versions or want to migrate from one version to the next, you may have to reinstall the packages in your own personal library. For example, going from R/3.5.2 to R/3.6.1 or R/4.0.2 probably requires to reinstall all packages (an update to R/3.5.3, i.e. a bugfix release, should not be a problem). If required, you can also keep packages for multiple versions, in which case you should create directories of the form $HOME/R/x.y/lib, e.g.
$ mkdir -pv $HOME/R/4.0/lib
and then set
R_LIBS_USER="~/R/%v/lib"
in $HOME/.Renviron. The %v will be expanded the version of R that you are using and allows you to have multiple lib-directories.
Using R on the HPC cluster
If you want to use R on the HPC cluster, you will have to load its module. You can do that by using the command
module load R
Since R is installed on multiple environments and in different versions, possibly you will have to change environment and specify the version
module load hpc-env/6.4 module load R/3.5.2-intel-2018a
Basic Job Script for R
Suppose you want to create 100 random numbers and calculate their mean and standard deviation. In R the commands for that would be:
x <- runif(100,0.0,1.0) mean(x) sd(x)
If you want to do the calculation on the cluster store the above commands in a file named e.g. Rtest.R. Then create a job script Rtest.sh with the content:
#!/bin/bash #SBATCH --job-name=Rtest #SBATCH --partition=carl.p #SBATCH --time=24:00:0 #SBATCH --mem=5000M # load modules module load R # run R Rscript ./Rtest.R
and submit a job with the command:
sbatch Rtest.sh
The output of R will appear in a file called slurm-<jobid>.out once the job has been completed. Instead of the command
Rscript ./Rtest.R
you can also use
R CMD BATCH ./Rtest.R
in which case you would find (a slightly different output) in a file called Rtest.Rout. Try out the different commands (and maybe also additional options that can be passed) to see which serves your needs best.
Using batchtools
In some situations you may need to run the same R-program multiple times. This can be achieved with the approach described below using foreach and doMPI. Another option is the R-package batchtools as described on this page
Usage of R and MPI
For parallelization the package doMPI is installed. To launch an parallel R script inside a SLURM script please use command line
mpirun R --slave -f SCRIPTNAME SCRIPT_CMDLINE_OPTIONS
to enable SLURM to control all processes of your script. Please do not use the batch starting sequence R CMD BATCH!
The corresponding parallel environment in the SLURM submission script is specified by
#SBATCH --ntasks=NUMBER_OF_TASKS
Note for doMPI:
- Before you start R with the mpirun command you have to unset the environment variable R_PROFILE in your SLURM-Script. Otherwise the MPI processes were not spawned. Please add following line to your jobscript:
unset R_PROFILE
- Please use mpi.quit() at the end of your script. Otherwise it will not end.
- Here a small example R script for doMPI (it writes the current rank of MPI in b):
#!/usr/bin/env Rscript
#
# file name: test_dompi.R
#
library("doMPI")
# doMPI start
cl <- startMPIcluster()
registerDoMPI(cl)
# parallel foreach due to %dopar% using the MPI cluster
# note that one MPI process is the master (rank 0)and
# distributes the work (iterations) among the other
# processes (ranks 1 to (mpi.comm.size(0)-1))
# rnorm returns a vector with three elements, the
# option .combine="rbind" makes a table with 10 rows
b<-foreach(i=1:10, .combine="rbind") %dopar% {
my_rank<-as.integer(mpi.comm.rank(0))
rnorm(3, my_rank, 0.01) # return three random values near my_rank
}
closeCluster(cl)
print(b)
mpi.quit()
and he corresponding SLURM-script
#!/bin/bash #SBATCH --job-name=test_dompi #SBATCH --time=24:00:0 #SBATCH --mem-per-cpu=2G #SBATCH --output=dompi-test.%j.out #SBATCH --error=dompi-test.%j.err #SBATCH --ntasks=4 # load modules module load hpc-env/8.3 module load R # unset the environment variable which is needed for Rmpi # but makes problems with doMPI unset R_PROFILE # run R in parallel (mpirun knows the number of tasks requested) mpirun R --slave -f ./test_dompi.R
Usage of NetCDF and R
A package for NetCDF has been installed together with R. In order to use it, please add the command
module load netCDF
to your job script before starting R. Your R-script should include a line
library(ncdf)
to load the NetCDF library. Please refer to the documentations of NetCDF and R for more informations.
Documentation
You can look up anything about R on their
