Singularity 2016

Introduction

Singularity is a portable application stack packaging and runtime utility. With Singularity, you can provide whole working environments and install the OS, and Software you need to use within a container. In the best case, "this means that you don’t have to ask your cluster admin to install anything for you - you can put it in a Singularity container and run."[¹ http://singularity.lbl.gov/]

Installed version

The currently installed version is available on environment hpc-env/6.4

Singularity/2.4.2-GCCcore-6.4.0
Singularity/2.6.1-GCCcore-6.4.0 (D)

Using Singularity

Singularity can be a very powerful tool to create containers with your favoured OS environment, and all the tools you need - completely without the help or permission of the HPC administrators. In order to use Singularity, you need to create your own containers on a local device since you need root permissions to do so which we can't grant our users on our cluster. In the following chapters, we will show you exemplarily how to create your own containers, fill them with the software you need, and how to use them to start your computations. But first, Singularity must be installed on a local system.

Installation

In our example, we use CentOS 7. For other distributions, the procedure should be similar, but some steps might slightly differ, especially the needed dependencies for the installation.

To install Singularity on your system, you need to install at least two dependencies , clone the GitHub repository and perform confingure/make/make install. To make this as simple as possible, open a terminal, and copy-paste this lines:

cd ~ && mkdir singularity_install && cd singularity_install
sudo yum -y install libarchive-devel squashfs-tools
git clone https://github.com/singularityware/singularity.git
cd singularity
git fetch --all
git checkout 2.6.1
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

Now, Singularity should be installed on your system. Just to be sure, check the version of the newly installed software:

singularity --version

Singularity should display the installed version.
Should this be the case, you can remove the folder containing the source files we originally needed for the installation.

Creating Containers

There are three ways to create containers:

• A) Downloading solely OS containers and manually adding software packets
• B) Creating installaton scripts, including OS and post-install commands
• C) Searching for 'ready to go scripts' from Singularitys repository

Below we will briefly cover each of the methods mentioned.

A) Download and modify

Always remember: The following commands must be executed on your own local machine!
To download an image only containing CentOS, type in the following command:

sudo singularity build --sandbox centos7.simg docker://centos:7

Now, switch into the container environment and install the packets needed:

sudo singularity shell --writable centos7.simg
# Now, every shell command will be forwarded to the container centos.simg
yum update -y
yum install vim -y
# To escape from the container shell, type in:
exit

In this example, we switched to the containers shell, installed the text editor vim, and finally exited from the containers shell with exit.

Singularity will automatically bind your $HOME directory so that you can enter your filesvfrom within the container. If you need other paths to be connected to your container, you can can do this with the --bind command:

singularity shell --bind $WORK centos7.simg

B) Working with your own recipe (Bootstrapping)

By creating a personal recipy, you have the advantages of creating your personal environment, chosing the OS and installing the software that fits your needs perfectly. When the recipe is created, a new container can be bootstrapped with one single command at any time.

Start by creating a script file, e.g. with the vim editor...

vim my_container.def

... and fill it with the settings you prefer:

# Chose an OS, preferably from the official docker images
Bootstrap: docker
From: centos:latest
# Run post-install commands. Use them just like you would do within the targeted operating system

# Set environment variables
%environment
PATH=new/directory:$PATH

# Set labels 
%labels
AUTHOR Monika Testuser, monika.testuser@uol.de

%post
yum -y update && yum -y install wget vim

# Copy files into the container.
%files
source/path/to/file.txt target/path/to/file.txt

After saving and closing the file, you start the installation with:

sudo singularity build --sandbox my_container.img my_container.def

Afterwards, you can modify the newly built image like shown on A)

sudo singularity shell --writable my_container.img
touch testfile.txt
exit

C) Use an 'off-the-shelf' image

On the Singularity Hub, you have access to a lot of pre-puilt images and the corresponting recipes.

Here, we have two options: A) Downloading the complete image files or B) downloading the corresponding recipe. For downloading the ready-to-use image, Singularity brings a pull function.

# Pulling a tensorflow image from the singularity hub:
singularity pull shub://marcc-hpc/tensorflow

This command will download a read-only image file. You use it to start calculations but you can't change or modify it any more. Should you need to do any changes, consider using the recipe. On the project site of tensorflow (for example), you can download the recipe for every container image. When downloaded, the recipe can be extended by any desired argument and be build as --writable or --sandbox.

https://singularity-hub.org/containers/2725/download/recipe

Transferring your Container to CARL/ERLE

If your container is ready for computing, you will probably want to transfer it to the cluster (CARL or EDDIE). For this task, we can use the tool scp, which should be pre installed on every common linux distribution. The usage is very straightforward:

sudo scp <container_file> abcd1234>@carl.hpc.uni-oldenburg.de/home/abcd1234/
# Where abcd1234 should be your domain user name.

If your container is packed as a folder (--sandbox),you have to copy recursively with -r. And do not forget to copy with sudo! Otherwise, the files will probably not be copied entirely which can be overseen very easily.

Troubleshooting

sudo: singularity: command not found
If this error occurs on your local machine, sudo may be missing a path to Singularitys installation directory. With sudo visodu this can be changed by adding the path to a specific line:

Just add :/usr/local/bin to the line containing secure_path so it looks like this:
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

#######WORK IN PROGRESS#######

If you want to find out more about Singularity on the HPC Cluster, you can use the command

module spider Singularity

This will show you basic informations e.g. a short description, and the currently installed version.

To load the desired version of the module, use the command, e.g.

module load hpc-env/6.4
module load Singularity

Always remember: this command is case sensitive!

To get familiar with the commands, and options available for Singularity, you can use the help function:

singularity --help

Documentation

The developers offer a detailed website with further information, and a Quick Stard Guide.