EGSnrc

Overview

The follwing description can also be found here:

NRC’s electron gamma shower (EGS) software tool can meet your specific requirements relating to modelling the passage of electrons and photons through matter. EGSnrc relies on Monte Carlo, which is the most accurate method to model the transport of radiation.

EGSnrc is used to address a broad range of questions about the propagation of radiation in materials. It is particularly well-suited for medical physics purposes, such as the research and development of devices that allow medical professionals to detect radiation, image a patient’s anatomy using x-rays, or deliver a prescribed radiation dose to a tumour while sparing healthy tissue. The software is also employed directly by medical physicists in cancer clinics for research and for verifying radiation treatment plans.

The software is freely available for research and other non-commercial purposes, see [1] for details. The latest version is now available on GitHub[2] and uses a new license model (free GNU license). Please read the documentation and license information on GitHub. By using EGSnrc on the cluster you are accepting the terms of the license. Also cite EGSnrc in any work published, the relevant citation can be found on GitHub as well.

The modules also includes the package BEAMnrc, see [3] for more information. Since version 2016 BEAMnrc is included in EGSnrc, no extra package need installation.

Using EGSnrc on the cluster

Loading the module

The available versions of EGSnrc on the cluster can be listed with

 module av egsnrc

In order to use EGSnrc you have to load the corresponding modules for the desired version with e.g.

 module load egsnrc/2016

after which using

 module list

should show e.g. (or something similar)

 Currently Loaded Modulefiles:
 1) shared         2) gcc/4.8.1      3) sge/6.2u5p2    4) egsnrc/2016

Note, the module for the GCC compiler is loaded automatically if needed.

First time usage

EGSnrc requires a directory in user space to store the so-called user codes. If you are using a version of EGSnrc for the first time on the cluster you need create a directory and to copy these user codes there.

This can be done conveniently by running the provided scripts for EGSnrc

 finalize_egs_foruser

(after you have loaded the module, of course) which will copy and compile the user codes.

The script will ask you for a directory name to store the user codes

 Input the directory where you want to keep your user codes 
      [/user/somegroup/abcd1234/egsnrc]:

It is strongly recommended that you use the following directories which are also the suggested defaults:

• for EGSnrc 2.4.0 the provided default is $HOME/egsnrc)
• for EGSnrc 2016 or later the default is $HOME/egsnrc-version where version is replaced by the version number

The default is accepted by simply typing Return. The default directory is also set by loading the module as can be seen with

 echo $EGS_HOME

after loading the module for the desired version. If you are not using the recommended setting here you will need to change $EGS_HOME after loading the module.

Then you will be asked which user codes should be installed where you can enter e.g. 1 for Compile all. After that you should a few messages about codes being successfully compiled. This may take a few minutes.

After the installation of the user codes you will see the following message:

 *******************************************************************************
 
 The configuration was completed succesfully for user abcd1234.
 
 Your default shell is /bin/bash
 Add the following 3 statements to your .bashrc file: 
 
 export EGS_HOME=/user/hrz/lees4820/egsnrc/
 export EGS_CONFIG=/cm/shared/apps/egsnrc/2.4.0/HEN_HOUSE/specs/x86_64-gcc.config
 . /cm/shared/apps/egsnrc/2.4.0/HEN_HOUSE/scripts/egsnrc_bashrc_additions
 
 ******************************************************************************

You do not need to modify your .bashrc-file as all the required settings are done by loading the module as described above.

For EGSnrc 2.4.0 only: After running finalize_egs_foruser you also need to install the user codes for BEAMnrc with

 finalize_beam_foruser

which does not require any input and will produce a similar output (shorter) as before.

Remember, these steps only need to done once. If you encounter problems, please send the log-file (found in your home-directory) to Scientific Computing.

Testing

You can quickly test if everything is installed correctly by the command

 tutor1 -p tutor_data

which should produce around 100 lines of output in one second.

Running jobs on the cluster

As with any application on the cluster you have to submit a job script to run EGSnrc (see below for an example). A detailed introduction to job submission can be found under SGE Job Management (Queueing) System.

A job submission script for one of the EGSnrc user codes could look like this:

#!/bin/bash

### the following lines are used by SGE to determine your job requirements
### you may want to modify the requested resources, e.g. h_vmem or h_rt,
### if you need more memory or run time (respectively)
#$ -S /bin/bash
#$ -cwd
#$ -l h_rt=1:00:0
#$ -l h_vmem=500M
#$ -l h_fsize=100M
#$ -N egsnrc
#$ -j n


### here the actual job script begins
### you may want to modify this part according to your needs
### the job script should placed in the directory of the user code
### you want to use
echo Job started on `date`   # put a time stamp in log file

# load module EGSnrc
module load gcc
module load egsnrc

# settings (here you need to make your own modifications)
# NOTE: the input file has to be in the directory $EGS_HOME/$USERCODE
#       the PEGS4DATA files can be located in different locations
#       one is the HEN_HOUSE which is automatically set
USERCODE=cavrznrc
INPUTFILE=cavrznrc_template
PEGS4DATA=521icru

# run code, e.g. the example from cavrznrc
$USERCODE -i $INPUTFILE -p $PEGS4DATA

echo Job finished on `date`   # put a time stamp in log file

exit

Copy the job script to a file named e.g. run_egsnrc.sge which should be located in the directory of the user code you want to run. In the example, the user code is cavrznrc, the script is placed in ~/egsnrc/cavrznrc. You will need to modify the job script to your needs. In the first part of the script, resources for the job are requested (e.g. with the line #$ -l h_rt=1:00:0 the maximum run time of the job is set to one hour). Below, the actual job is described. Here you may want to modify, which user code you want to use, which input file and so on. Note, that EGSnrc has a peculiar way of handling files, which means for example that the input file can only be placed in the directory of the user code.

Once the job script is prepared, you can submit it to the SGE by the command

 qsub run_egsnrc

which will respond with

 Your job 32083 ("egsnrc") has been submitted

After that you can check the status of your jobs using the

 qstat

command.

You can submit multiple jobs to the cluster by having multiple sge-scripts or by modifying the sge-script between each qsub-command. Note however, that while you can modify the sge-script, you should not modify the input file when submitting more than one job. Instead you need to make copies for each submitted job.

When you job is completed (i.e. when the job-ID is no longer listed by qstat), you can find the output files from the EGSnrc user code in the usual place. In addition, you will find two more output files, e.g.

 egsnrc.e32083
 egsnrc.o32083

which contain the output normally printed on the screen.

Job Arrays

If you need to run the same code many times with different input files you should consider using a job array. Suppose you have an input file named setup.egsinp containing the following (or similar) lines:

 initial random no. seeds = 1, 6
 ncase = 10000
 nbatch  = 10

If you now make a copy of this file naming it setup_template.egsinp and modify the lines to

 initial random no. seeds = 1, %seed%
 ncase = 1000
 nbatch  = 1

Note how ncase and nbatch have been divided by 10, wheres %seed% is introduced as a placeholder.

Now you could write a short bash-script, e.g.

#!/bin/bash

NPAR=10
FIRSTSEED=6
TEMPLATE="setup_template.egsinp"

for i in `seq $NPAR`
do
   seed=`expr $i + $FIRSTSEED - 1`
   INPFILE=`echo $TEMPLATE | sed s/_template/_w$i/`
   cat $TEMPLATE | sed s/%seed%/$seed/ > $INPFILE
done

saving it as generate_egsinp.sh and giving it execute permissions with chmod a+x generate_egsinp. Now you can run this script

 generate_egsinp

which will generate a numbered sequence of 10 input files:

 setup_w1.egsinp  
 setup_w2.egsinp
       ....
 setup_w10.egsinp

Then you can use a job array to run the same job script on each of the input files. The following lines need to be added to the job script from above:

 #$ -t 1-10
 #$ -tc 2

The first one tells SGE that this job is an array, and that tasks number from 1 to 10 should be generated (it is also possible to define different ranges and increments, e.g. #$ -t 1000-1500:2 will generate tasks for every even number between (and including) 1000 and 1500). The second line limits the number of tasks that can run at the same time to 2 (this line is optional, and 2 is just an example).

In addition, you also need to modify the line where the input file is defined:

 INPUTFILE=setup_w$(printf "%i" $SGE_TASK_ID)

The number of the current task is stored in the variable $SGE_TASK_ID and can be used to create a unique file name for the input file (the exact details for this depend on your file names, of course).

Once the modified job script is ready, you can submit as before with

 qsub run_egsnrc_array.sge

but now your script will run multiple times with as many output.