File system and Data Management

The HPC cluster offers access to two important file systems. In additions the compute nodes of the HERO-II cluster have local storage devices for high I/O demands. The different file systems available are documented below. Please follow the guidelines given for best pratice.

Storage Hardware

A GPFS Storage Server (GSS) serves as a parallel file system for the HPC cluster. The total (net) capacity of this file system is about 900 TB and the read/write performance is up to 17/12 GB/s over FDR Infiniband. It is possible to mount the GPFS on your local machine using SMB/NFS (via the 10GE campus network). The GPFS should be used as the primary storage device for HPC, in particular for data that is read/written by the compute nodes. Currently, the GPFS offers no backup functionality (i.e. deleted data can not be recovered).

The central storage system of the IT services is used to provide the NFS-mounted $HOME-directories. The central storage system offers very high availability, snapshots, backup and should be used permanent storage, in particular everything that cannot be recovered easily (program codes, initial conditions, final results, ...).

File Systems

The following file systems are available on the cluster

File System	Environment Variable	Path	Device	Data Transfer	Backup	Used for	Comments
Home	$HOME	/user/abcd1234	Isilon Storage System	NFS over 10GE	yes	critical data that cannot easily be reproduced (program codes, initial conditions, results from data analysis)	high-availability file-system, snapshot functionality, can be mounted on local workstation
Data	$DATA	/gss/data/abcd1234	GPFS	FDR Infiniband	not yet	important data from simulations for on-going analysis and long term (project duration) storage	parallel file-system for fast read/write access from compute nodes, can be mounted on local workstation
Work	$WORK	/gss/work/abcd1234	GPFS	FDR Infiniband	no	data storage for simulation runtime, pre- and post-processing, short term (weeks) storage	parallel file-system for fast read/write access from compute nodes, temporarily store larger amounts of data
Scratch	$TMPDIR	/scratch/<job-specific-dir>	local disk or SSD	local	no	temporary data storage during job runtime	directory is created at job startup and deleted at completion, job script can copy data to other file systems if needed

Quotas

Quotas are used to limit the storage capacity for each user on each file system (except $TMPDIR which is not persistent after job completion). The following table gives an overview of the default quotas. Users with a particular high demand in storage can contact Scientific Computing to have their quotas increased (reasonably and depending available space).

File System	Hard Quota	Soft Quota	Grace Period
$HOME	1 TB	none	none
$DATA	2 TB	none	none
$WORK	10 TB	2 TB	30 days

The $WORK file system is special regarding quotas. It uses a soft limit and a hard limit whereas all other file systems only have a hard limit.

A hard limit means, that you cannot write more data to the file system than allowed by your personal quota. Once the quota has been reached any write operation will fail (including those from running simulations).

A soft limit, on the other hand, will only trigger a grace period during which you can still write data to the file system (up to the hard limit). Only when the grace period is over you can no longer write data, again including from running simulations (also note that you cannot write data even if you below the hard but above the soft limit).

What does this mean for $WORK? You can store data on $WORK for as long as you want while you are below the soft limit. When you get above the soft limit (e.g. during a long, high I/O simulation) the grace period starts. You can still produce more data within the grace period and below the hard limit. Once you delete enough of your data on $WORK to get below the soft limit the grace period is reset. This system forces you to clean up data no longer needed on a regular basis and helps to keep the GPFS storage system usable for everyone.

Scratch space / TempDir

In addition to the storage in $HOME, $WORK and $DATA, the nodes also have local space called $TMPDIR. The amount of space is different for every node type:

• mpcs-nodes: approx. 800 GB (HDD)
• mpcl-nodes: approx. 1.7 TB (HDD)
• mpcp-nodes: approx. 1.7 TB (SSD)
• mpcb-nodes: approx. (SSD)
• mpcg-nodes: approx. (HDD)

Important note: This storage space is only used for files temporary created during the execution of the job. After the job has finished the files will be deleted. If you need local storage for your job, please add the following line to your jobscript

#SBATCH --gres:tmpdir:100G

This will reserve 100GB local, temporary storage for your job.

Keep in mind that you can add only one gres (generic resource) per jobscript. Multiple lines will not be accepted. If you need an additional GPU for your job, the "--gres"-command should look like this:

#SBTACH --gres:tmpdir:100G, gpu:1

Managing access rights of your folders

Different to the directory structure on HERO and FLOW, the folder hierarchy on CARL and EDDY is flat and less clustered together. We no longer have multiple directories stacked in each other. This leads to an inevitable change in the access right management. If you dont change the access rights for your directory on the cluster, the command "ls -l /user" will show you something like this:

If you look at the first line shown in the screenshot above, you can see the following informations:

drwx--S---  3 abgu0243  agtheochem        226  2. Feb 11:55 abgu0243

drwx--S---

• the first letter marks the filetype, in this case its a "d" which means we are looking at an directory.
• the following chars are the access rights: "rwx--S---" means: Only the owner can read, write and execute the folder and everything thats in it.
• The "S" stands for "Set Group ID". This will, additional to the rights of the executing (read and write respectively) User, apply the rights of the group which owns the filed/directory etc. This was an temporary option to ensure safety while we were opening the cluster for the public. Its possible that the "S" isnt set when you are looking at this guide, that is okay and intended.

abgu0243

• current owner of the file/directory

agtheochem

• current group of the file/directory
• this is your primary group. You can check your secondary group with the command "groups $(whoami)". It will output something like this: "abcd1234: group1 group2".

abgu0243

• current name of the file/directory

Basically we will need three commands to modify our access rights on the cluster:

• chmod
• chgrp
• chown

You will most likey just need one or maybe two of these commands, nevertheless we listed and described them for the integrity of this wiki.

chmod

The command chmod will directly modify the access rights of your desired file/directory. chmod has two different modes, symbolic- and octal-mode, but the syntaxes are pretty much the same:

• symbolic mode

chmod [OPTION]... MODE[,MODE]... FILE...

• octal-mode

chmod [OPTION]... OCTAL‐MODE FILE...

The following table will show the three different user categories for each of the modes.

Usertype	Symbolic mode	Octal-mode
Owner of the file	u	1. digit
Group of the file	g	2. digit
Other users	o	3. digit
Owner, group and others	a

Possible access rights are: r = read, w = write, x = execute

Examples for the symbolic mode

• allow your group to read, but not to write or execute your file

chmod g=r yourfile.txt

"=" will always clear every access right and set the ones you want. For example, if the file mentioned above was readable, writable and executable, the command "chmod g=r yourfile.txt" will make the file only readable. Beside that you can use "+" to add specific rights and "-" to remove them.

• allow other users to read and write, but not to execute your file

chmod o=rw yourfile.txt or chmod o+rw yourfile.txt

Examples for the octal mode

For comparison, we will be using the same examples as in the symbolic mode shown above:

• allow your group to read, but not to write or execute your file (owner can still read, write and execute)

chmod 744 yourfile.txt 

• allow other users to read and write, but not to execute your file

chmod 766 yourfile.txt

An easy way to calculate these numbers is using the following tool: chmod-calculator.com

chgrp

The command chgrp (or change group) will change the group of your file/directory. The syntax of the command looks like this:

chgrp [OPTION] GROUP FILE

Example

Change the group of your file/directory

chgrp yourgrp /user/abcd1234/randomdirectory

chown

The command chown (or: change owner) does what you think it does: it changes the owner of a file or directory. The syntax of the command looks like this:

chown [OPTION]... [OWNER][:[GROUP FILE...

Note: In the most cases you will not need to use this command!

Examples

• How can prevent anyone else from reading, modifying, or even seeing files/directories in my directories?

cd /user
chmod u=rwx /abcd1234/your_folder

• How can I grant access to a file/folder for another member of my primary group?

cd /user
chmod g=x /abcd2134/your_folder

Members of your group will only see the folder that you mentioned, other folders arent visible.

Adding "rx" will grant the same rights, but everything else is visible (not accessible though).

cd /user
chmod g=rx /abcd1234/your_folder

• How can I grant access to a file/folder for another member of one of my secondary groups?

cd /user
chgrp -R your_secondary_group /abcd1234/your_folder

This will grant "your_secondary_group" the set rights for this group to your folder "/abcd1234/your_folder".

• How can I grant access to a file/folder for anyone without anyone being able to access files/folders stored in the same directory?

cd /user
chmod o=x /abcd1234/your_folder

NOTE: We used "=" in every example. Remember that using "=" (instead of "+" which adds rights) will reset all other rights that are applied to the file/folder and will only set the ones you added!

Storage Systems Best Practice

Here are some general remarks on using the storage systems efficiently:

• Store all files and data that cannot be reproduced (easily) on $HOME for highest protection.
• Carefully select files and data for storage on $HOME and compress them if possible as storage on $HOME is the most expensive.
• Store large data files that can be reproduced with some effort (re-running a simulation) on $DATA.
• Data created or used during the run-time of a simulation should be stored in $WORK and avoid reading from or writing to $HOME, in particular if you are dealing with large data files.
• If possible avoid writing many small files to any file system.
• Clean up your data regularly.