Brief Introduction to HPC Computing

A brief introduction to the usage of the HPC facilities, targeted at new and unexperienced HPC users is given below. The introduction is based on various minimal examples that illustrate how to compile serial and parallel programs as well as how to submit and monitor actual jobs using SGE.

Misc

how to use local storage for I/O intense serial jobs (or parallel jobs that run on a single host)

Consider a situation where your particular application is rather I/O intense so that the speed of your program suffers from the amount of I/O operations that strain the global file system. Examples might be irregular I/O patterns at a fast pace or an application that has to create, open, close and delete many files. As a remedy in order to overcome such problems you might benefit from using a local scratch disk of an execution host on which your program is actually run. This reduces the amount of network traffic and hence reduces the strain on the global file system. The subsequent example illustrates how to access and use the local storage on a given host for the purpose of storing data during the runtime of the program. In the example, after the program terminates, the output data is copied to the working directory from which the job was submitted from and the local file system on the host is cleaned out. For this matter, consider the examplary C program myExample_tempdir.c

 
#include <stdio.h>

int main(int argc, char *argv[]) {
  FILE *myFile;

  myFile=fopen("my_data/myData.out","w");
  fprintf(myFile,"Test output to local scratch directory\n");

  fclose(myFile);
}
  

which, just for arguments (and to fully explain the job submission script below), is contained in the working directory

 $HOME/wmwr/my_examples/tempdir_example/

The program assumes that there is a directory my_data in the current working directory to which the file myData.out with a certain content (here the sequence of characters Test output to local scratch directory) will be written.

In oder to compile the program via the current gcc compiler, you could first set the stage by loading the proper modules, e.g.

 module clear
 module load sge
 module load gcc/4.7.1

and then compiling via

  gcc myExample_tempdir.c -o myExample_tempdir

to yield the binary myExample_tempdir.

At this point bear in mind that we do not want to execute the binary by hand right away! Instead, we would like to leave it to SGE to determine a proper queue instance (guided by the resources we subsequently will specify for the job) on a host with at least one free slot, where the job will be executed. A proper job submission script, here called myProg_tempdir.sge, that takes care of creating the folder my_data needed by the program myExample_tempdir in order to store its output in a temporal directory on the executing host reads

 
#!/bin/bash

####### which shell to use
#$ -S /bin/bash

####### change to directory where job was submitted from
#$ -cwd

####### maximum walltime of the job (hh:mm:ss)
#$ -l h_rt=0:10:0

####### memory per job slot
#$ -l h_vmem=100M

####### since working with local storage, no need to request disk space

####### name of the job
#$ -N tmpdir_test

####### change current working directory to the local /scratch/<jobId>.<x>.<qInst>
####### directory, available as TMPDIR on the executing host with HOSTNAME
cd $TMPDIR

####### write details to <jobName>.o<jobId> output file
echo "HOSTNAME = " $HOSTNAME
echo "TMPDIR   = " $TMPDIR

####### create output directory on executing host (parent folder is TMPDIR)
mkdir my_data

####### run program
$HOME/wmwr/my_examples/tempdir_example/myExample_tempdir

####### copy the output to the directory the job was submitted from
cp -a ./my_data $HOME/wmwr/my_examples/tempdir/
  

Note that in the above job submission script there is no need to request disk space by setting the resource h_fsize since we are working with local storage provided by the execution host. Submitting the script via

 qsub myProg_tempdir.sge

enqueues the respective job, here having jobId 703914. After successful termination of the job, the folder my_data is moved to the working directory from which the job was originally submitted from. Also, the two job status files tmpdir_test.e703914 and tmpdir_test.o703914 where created that might contain further details associated with the job. The latter file should contain the name of the host on which the job actually ran and the name of the temporal directory. And indeed, cat tmpdir_test.o703914 reveals the file content

 HOSTNAME =  mpcs001
 TMPDIR   =  /scratch/703914.1.mpc_std_shrt.q

Further, the file my_data/myData.out contains the line

 Test output to local scratch directory

as expected. Note that the temporary directory $TMPDIR (here: /scratch/703914.1.mpc_std_shrt.q) on the execution host (here: mpcs001) is automatically cleaned out.