Singularity 2016

Introduction

Singularity is a portable application stack packaging and runtime utility. With Singularity, you can provide whole working environments and install the OS, and software you need to use within a container. In the best case, "this means that you don’t have to ask your cluster admin to install anything for you - you can put it in a Singularity container and run." [1]

Installed version

Singularity is installed as part of the Linux OS on the head and compute nodes. That means you do not need to load any modules (anymore) to use Singularity on the cluster. The exact version can be determined from the command

$ singularity version
3.4.0-1.2.el7

Any containers you have created with Singularity 3.x should be compatible with this version, older 2.x containers should be rebuild.

Note: The earlier modules with Singularity 2.x should no longer be used and will be removed on February 29th, 2020.

Using Singularity

Singularity can be a very powerful tool to create containers with your favoured OS environment, and all the tools you need - completely without help or permission of the HPC administrators. Since you need root permissions to build your own containers, you will need to prepare them on your local device. In the following chapters, we will show you exemplarily how to create your own containers, fill them with the software you need, and how to use them to start your computations. But first, Singularity must be installed on a local system where you have administration rights (e.g. sudo.

Installation

The installation of singularity is described in the documentation. Below, the steps for Ubuntu are listed, which work similarly for other Linux distributions. The installation is done in three steps:

1. Install dependencies: To do so, open a terminal and copy-paste the following commands:

$ sudo apt-get update && sudo apt-get install -y \
    build-essential \
    libssl-dev \
    uuid-dev \
    libgpgme11-dev \
    squashfs-tools \
    libseccomp-dev \
    wget \
    pkg-config \
    git \
    cryptsetup

2. Install Go: Singularity 3.x is written primarily in Go, and you will need Go 1.13 or above installed to compile it from source. The following commands are one way to do so:

$ export VERSION=1.13.7 OS=linux ARCH=amd64 && \
    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
    rm go$VERSION.$OS-$ARCH.tar.gz

where you can change the version as needed. Then setup the environment to complete the Go installation:

$ echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
    echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
    source ~/.bashrc

You may want to repeat this step for other users on the same system if they need Go for other reasons.

Now, Singularity should be installed on your system. Just to be sure, check the version of the newly installed software:

singularity version

Singularity should display the installed version.
Should this be the case, you can remove the folder containing the source files we originally needed for the installation (~/singularity_install).

Creating Containers

There are three ways to create containers which will be discussed:

• A) Downloading solely OS containers and manually adding software packets
• B) Creating installation scripts, including OS and post-install commands
• C) Searching for 'off-the-shelf scripts' from Singularitys repository

Below we will briefly cover each of the methods mentioned.

A) Download and modify

Always remember: The following commands must be executed on your own local machine!
To download an image only containing CentOS, type in the following command:

$ sudo singularity build --sandbox centos7.img/ docker://centos:7

This will create a directory centos7.img on your local computer which contains a basic Linux installation and can be modified to your needs.

As you can see, with docker://centos:, we used a docker image from the Docker hub. A lot of distributions, like Fedora, Ubuntu, Debian, etc.´, are available that way.

Now, you can start a shell within the container environment and install the packets you need:

$ sudo singularity shell --writable centos7.img
# # Now, every shell command will be executed within the container centos.img (idicated by the prompt changing from $ to #)
# yum update -y
# yum install vim -y
# # To escape from the container shell, type in:
# exit

In this example, we switched to the containers shell, did some updates, installed the text editor vim, and finally exited from the containers shell with exit. As you probably noticed, the commands within the container are executed as root because we ran Singularity with sudo.

When you finished modifying the image, create a read-only image file for the productive use (which additionally makes is very space efficient):

sudo singularity build centos7.simg centos7.img/

B) Working with your own recipe (Bootstrapping)

By creating a personal recipe (also called definition file = .def), you have the advantages of creating your personal environment, choosing the OS, and installing the software that fits your needs perfectly. When the recipe is created, a new container can be bootstrapped with one single command at any time.

Start by creating a script file, e.g. with the vim editor...

vim my_container.def

... and fill it with the settings you prefer. You can comment out with #, so that you can remember the purpose of every line, if you want to store the recipe file for some time.

# Chose an OS, preferably from the official docker images
Bootstrap: docker
From: centos:latest

# Set environment variables
%environment
PATH=new/directory:$PATH

# Set labels 
%labels
AUTHOR Monika Testuser, monika.testuser@uol.de

# Run post-install commands. Use them just like you would do within the targeted operating system 
%post
yum -y update && yum -y install wget vim

# Copy files into the container.
%files
source/path/to/file.txt target/path/to/file.txt

After saving and closing the file, you start the installation with:

sudo singularity build --sandbox my_container.img my_container.def

Since we used the argument --sandbox, we can modify the newly built image afterwards like shown on A)

sudo singularity shell --writable my_container.img
touch testfile.txt
yum -y install vim 
exit

Like in A), create a read-only image file when you finished modifying the image:

sudo singularity build my_container.simg my_container.img

C) Use an 'off-the-shelf' image

On the Singularity Hub, you have access to a lot of pre-puilt images and the corresponting recipes.

Here, we have two options: A) Downloading the complete image files or B) downloading the corresponding recipe. For downloading the ready-to-use image, Singularity has a pull function.

# Pulling a tensorflow image from the singularity hub:
singularity pull shub://marcc-hpc/tensorflow

This command will download a read-only image file. You use it to start calculations but to change or modify it, you would have to change the image structure. Should you need to do any changes, consider using the recipe. On the project site of tensorflow (for example), you can download the recipe for the container image. When downloaded, the recipe can be extended by any desired argument and be build as --writable or --sandbox if any further changes are desired.

https://singularity-hub.org/containers/2725/download/recipe

Like in A) and B, finalize your image into a .simg file when you are done modifying.

 sudo singularity build my_container.simg my_container.img

Transferring your Container to CARL/EDDY and start working

If your container is ready for computing and finalized by building into a .simg file, you will probably want to transfer it to the cluster (CARL or EDDY). For this task, we can use the tool scp, which should be pre installed on every common linux distribution. The usage is very straightforward:

scp <container_file> abcd1234@carl.hpc.uni-oldenburg.de:/user/abcd1234/
# Where abcd1234 is your domain user name.

Now you can login to CARL, load the Singularity module, and start using your container:

ssh abcd1234 carl.hpc.uni-oldenburg.de
module load hpc-env/6.4
module load Singularity

At Putting in Practice, we show exemplarily, how using a container could look like.

Putting in Practice

Now, we will show you how to use Singularity from the first step to the last command.
Let's say, we want to convert a video file into a different type: flv to mp4, because flv files can't be handled by many video players.
We already stored our jellyfish clip in our HPCs Download directory:

#on CARL/EDDY:
cd ~/Downloads
wget http://mirrors.standaloneinstaller.com/video-sample/jellyfish-25-mbps-hd-hevc.flv

We choose HandBrakeCLI as our converting tool. Since converting files can be very compute intensive, we decide to use the HPC Cluster. The only problem is, that HandBrakeCLI is not available as a loadable module. This is where Singularity comes in handy.
After we installed Singularity, we are ready to go:
On our local machine, we create an image by writing a definition file with vim HandBrake.def...

Bootstrap: docker
From: ubuntu:latest

%post
# install dependencies 
apt-get -y update
apt-get -y install autoconf automake build-essential cmake git libass-dev libbz2-dev libfontconfig1-dev libfreetype6-dev libfribidi-dev libharfbuzz-dev libjansson-dev liblzma-dev libmp3lame-dev libogg-dev libopus-dev libsamplerate-dev libspeex-dev libtheora-dev libtool libtool-bin libvorbis-dev libx264-dev libxml2-dev libvpx-dev m4 make nasm patch pkg-config python tar yasm zlib1g-dev libdrm-dev

# pull source files, make & install HandBrakeCLI & delete the install-dir
mkdir /handbrake-install
cd /handbrake-install
git clone https://github.com/HandBrake/HandBrake.git && cd HandBrake
./configure --launch-jobs=$(nproc) --launch --disable-gtk
make --directory=build install
cd /
rm -rf /handbrake-install

... and build the image with this recipe:

sudo singularity build HandBrake.simg HandBrake.def 

--> Since we did not need to modify anything after the building process, we skipped --sandbox and directly created a .simg file!

Let's transfer the new image to our cluster:

scp HandBrake.simg abcd1234@carl.hpc.uni-oldenburg.de:/user/abcd1234/

At last, we can actually work with our new image.
On our local system, we would just call an exec command to HandBrakeCLI in the container:

singularity exec HandBrake.simg HandBrakeCLI -i ~/Downloads/jellyfish-25-mbps-hd-hevc.flv -o ~/Downloads/jellyfish-25-mbps-hd-hevc.mp4

#For better understanding: after singularity exec, every command and argument is transferred
#into the container. If HandBrakeCLI had been installed on our system, it would have been ignored anyway.

But since we work on the cluster, we will create a job to use the SLURM queuing system. First, we create a job file with vim...

vim HandBrake.job

... and then we will fill it with sbatch arguments and the actual commands we want it to get done:

#!/bin/bash

#SBATCH --nodes=1                    
#SBATCH --ntasks=1                  
#SBATCH --mem=2G                  
#SBATCH --time=0-1:00                
#SBATCH --output=slurm.%j.out        
#SBATCH --error=slurm.%j.err          
#SBATCH --mail-type=END,FAIL         
#SBATCH --mail-user=your.name@uol.de 

# Load the modules we need:
module load hpc-env/6.4
module load Singularity/2.6.1-GCCcore-6.4.0

# Actually work with the singularity container:
singularity exec HandBrake.simg HandBrakeCLI -i ~/Downloads/jellyfish-25-mbps-hd-hevc.flv -o ~/Downloads/jellyfish-25-mbps-hd-hevc.mp4

Afterwards, we submit the job:

sbatch -p carl.p HandBrake.job

In this example, we reserved one hour on one node on the carl.p partition, which should be more than enough for this simple task. For more Information about our SLURM system and the corresponding options, visit our page about SLURM

But that's it! We used the exec command to call the containerized HandBrakeCLI and processed the file outside of the container. This is possible, since Singularity automatically binds your $HOME path. And now, our beloved clip of jellyfish is watchable on nearly every video player (even if this should not be the main purpose of our HPC cluster (; ).

Working outside of $HOME

Singularity will automatically bind your $HOME directory so that you can enter your files from within the container. If you need other paths to be connected to your container, you can can do this with the --bind command:

singularity shell --bind $WORK centos7.simg

Troubleshooting

sudo: singularity: command not found

If this error occurs on your local machine, sudo may be missing a path to Singularitys installation directory. With sudo visodu this can be changed by adding the path to a specific line:

Just add :/usr/local/bin to the line containing secure_path so it looks like this:
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

mkdir: cannot create directory <chosen_directory>: Permission denied

Probably, the directory that you want wo work on was not made known to Singularity.
You need to use the option --bind to set the directory, e.g.:

singularity run --bind $WORK /gss/work/abcd1234/my_image.simg -d /gss/work/abcd1234/project_folder/ -r
### With --bind $WORK Singularity gets the permission to write on /gss/work/abcd1234

More Examples for Using Singularity

You can find more examples on how to use Singularity on the HPC cluster here.

Documentation

For further information developers offer a detailed website with further information, and a Quick Stard Guide.