View source for Queues and resource allocation

The thing about queues is that, ideally, you don't have to worry about them. In general you only specify resources for the job you are about to submit. 
In doing so you provide enough information for the scheduler to decide in which queue the job belongs in. Hence, you ''explicitly'' allocate resources and ''implicitly'' choose
a queue. However, in some cases, namely when it comes to the problem of running a job on, say, particular hardware components of the cluster, it is beneficial
to know the resources that need to be allocated in order to access a proper queue running on that component.

Albeit you (as a user) should worry more about ''specifying resources'' instead of ''targeting queues'' it is useful to disentangle the relationship between
certain queues that are implemented on the HPC system and the resources that need to be specified in order for the scheduler to address that queue. Also 
some of you might be familiar with the concept of queues and prefer to think in terms of them.

== Listing all possible queues ==

Now, thinking in terms of queues, you might be interested to see which queues there are on the HPC system. Logged in to your HPC account, you obtain a full list of all 
possible queues a job might be placed in by typing the command <code>qconf -sql</code>. <code>qconf</code> is a grid engine configuration tool which, among other
things, allows you to list existing queues and queue configurations. In casual terms, the sequence of options <code>-sql</code> demands: show (<code>s</code>) queue (<code>q</code>) list (<code>l</code>).
As a result you might find the following list of queues:
  <nowiki>
cfd_him_long.q
cfd_him_shrt.q
cfd_lom_long.q
cfd_lom_serl.q
cfd_lom_shrt.q
cfd_xtr_expr.q
cfd_xtr_iact.q
glm_dlc_long.q
glm_dlc_shrt.q
glm_qdc_long.q
glm_qdc_shrt.q
mpc_big_long.q
mpc_big_shrt.q
mpc_std_long.q
mpc_std_shrt.q
mpc_xtr_ctrl.q
mpc_xtr_iact.q
mpc_xtr_subq.q
uv100_smp_long.q
uv100_smp_shrt.q
  </nowiki>

== Obtaining elaborate information for a particular queue ==

So as to obtain more details about the configuration of a particular queue you just need to specify that queue. E.g. to get elaborate 
information on the queue <code>mpc_std_shrt.q</code>, just type
  qconf -sq mpc_std_shrt.q
which yields
  <nowiki>
qname                 mpc_std_shrt.q
hostlist              @mpcs
seq_no                10000,[mpcs001.mpinet.cluster=10001], \
                      [mpcs002.mpinet.cluster=10002], \
                      ...
                      [mpcs123.mpinet.cluster=10123], \
                      [mpcs124.mpinet.cluster=10124]
load_thresholds       np_load_avg=1.75,slots=0
suspend_thresholds    NONE
nsuspend              1
suspend_interval      00:05:00
priority              0
min_cpu_interval      00:05:00
processors            UNDEFINED
qtype                 BATCH
ckpt_list             NONE
pe_list               impi impi41 linda molcas mpich mpich2_mpd mpich2_smpd \
                      openmpi smp mdcs
rerun                 FALSE
slots                 12
tmpdir                /scratch
shell                 /bin/bash
prolog                root@/cm/shared/apps/sge/scripts/prolog_mpc.sh
epilog                root@/cm/shared/apps/sge/scripts/epilog_mpc.sh
shell_start_mode      posix_compliant
starter_method        NONE
suspend_method        NONE
resume_method         NONE
terminate_method      NONE
notify                00:00:60
owner_list            NONE
user_lists            herousers
xuser_lists           NONE
subordinate_list      NONE
complex_values        h_vmem=23G,h_fsize=800G,cluster=hero
projects              NONE
xprojects             NONE
calendar              NONE
initial_state         default
s_rt                  INFINITY
h_rt                  192:0:0
s_cpu                 INFINITY
h_cpu                 INFINITY
s_fsize               INFINITY
h_fsize               INFINITY
s_data                INFINITY
h_data                INFINITY
s_stack               INFINITY
h_stack               INFINITY
s_core                INFINITY
h_core                INFINITY
s_rss                 INFINITY
h_rss                 INFINITY
s_vmem                INFINITY
h_vmem                INFINITY
  </nowiki>
Among the listed resource attributes some contain valuable information:
* <code>pe_list</code>: specifies the list of [[SGE_Job_Management_(Queueing)_System#Parallel_environments_.28PEs.29 | parallel environments]] available for the queue. 
* <code>hostlist</code>: specifies the list of hosts on which the respective queue is implemented. <p>Here, the name of the hostlist is <code>@mpcs</code>. You can view the full list by means of the command <code>qconf -shgrpl @mpcs</code>, where <code>-shgrpl</code> stands for show (<code>s</code>) host group (<code>hgrp</code>) list (<code>l</code>).</p>
* <code>complex_values</code>:  A list of [http://docs.oracle.com/cd/E19080-01/n1.grid.eng6/817-5677/6ml49n2bh/index.html#i998850| complex resource attributes] a user might allocate for his jobs using the <code>#$ -l</code> option in his job submission script. <p>E.g., the queue configuration value <code>h_vmem</code> is used for the virtual memory size. This complex value is defined as a consumable variable which is limiting the amount of total allocatable memory on a host. An entry in the <code>complex_values</code> list of the queue configuration defines that the a job in the queue can consume this variable of the defined amount, in this case the job can allocate up to 23G of memory. </p> <p>In contrast the complex variable <code>cluster</code> is not a consumable. This variable is set for each host to distinguish the HPC component (either HERO of FLOW). An entry in the queue configuration defines that the queue needs this specific value on a host for using it.</p>
* <code>slots</code>: A consumable complex variable which defines the number of slots usable for this queue on the host. This variable is set to the specific value (usually the number of cores) on each host.
* <code>h_rt</code>: specifies a requestable resource of type <tt>time</tt>. A submitted job is only eligible to run in this queue, if the specified maximal value of <code>h_rt=192</code>h is not exceeded.
* <code>userlist</code>: list of users that are eligible to place jobs in the queue. Most queues are restricted to either all HERO users, comprised by the userlist <code>herousers</code>, or to all FLOW users, comprised by the userlist <code>flowusers</code>. However, some queues, as, e.g., the queue <code>uv100_smp_shrt.q</code> exhibit a user list that is much shorter.

== Requestable resources ==
The type and amount of requestable resources differs from queue to queue. To facilitate intuition compare, e.g., the main resources for <code>mpc_std_shrt.q</code> and <code>mpc_std_long.q</code>:
  <nowiki>
$ qconf -sq mpc_std_shrt.q | grep "qname\|hostlist\|complex_values\|h_rt\|pe_list"
qname                 mpc_std_shrt.q
hostlist              @mpcs
pe_list               impi impi41 linda molcas mpich mpich2_mpd mpich2_smpd openmpi smp mdcs
complex_values        h_vmem=23G,h_fsize=800G,cluster=hero
h_rt                  192:0:0 
  </nowiki>
and
  <nowiki>
$ qconf -sq mpc_std_long.q | grep "qname\|hostlist\|complex_values\|h_rt\|pe_list"
qname                 mpc_std_long.q
hostlist              @mpcs
pe_list               impi_long linda_long molcas_long smp_long
complex_values        h_vmem=23G,h_fsize=800G,cluster=hero,longrun=true
h_rt                  INFINITY
 </nowiki>

Note that both queues run on the same hosts, i.e. both have identical hostlists. However, the requestable resource <code>h_rt</code> and the list of complex values associated to both queues differs. At this point, details 
on the resource <code>h_rt</code> can once more be obtained using the <code>qconf</code> command:
  <nowiki>
$ qconf -sc | grep "h_rt\|#"
#name                   shortcut       type        relop   requestable consumable default  urgency   
#----------------------------------------------------------------------------------------------------
h_rt                    h_rt           TIME        <=      YES         NO         0:0:0    0  
</nowiki>
As can be seen, the relation operator associated to <code>h_rt</code> reads ''lower or equal''. I.e., so as to be eligible to be placed in the short queue, a job is not allowed to request more than 
192h of running time. Regarding the long queue, there is no upper bound on the running time and a job with proper allocated resources might be put in this queue.

Further, note that the long queue features one complex value more than the short queue, namely <code>longrun</code>. Details about this resource are:
  <nowiki>
$ qconf -sc | grep "longrun\|#"
#name                   shortcut       type        relop   requestable consumable default  urgency   
#----------------------------------------------------------------------------------------------------
longrun                 lr             BOOL        ==      FORCED      NO         FALSE    0
  </nowiki>
So, <code>longrun</code> is of type BOOL and has the default value FALSE. In order to place a job in the long queue one has to explicitly request to set <code>longrun=true</code>, see [[SGE_Job_Management_(Queueing)_System#Runtime| here]]. 

As a further detail, consider the requestable resource <code>h_vmem</code>. Details about this resource are:
  <nowiki>
$ qconf -sc | grep "h_vmem\|#"
#name                   shortcut       type        relop   requestable consumable default  urgency   
#----------------------------------------------------------------------------------------------------
h_vmem                  h_vmem         MEMORY      <=      YES         YES        1200M    0
  </nowiki>
I.e. it is specified as being a ''consumable'' resource. Say, you submit a single slot job to the short queue (which, by default, offers 23G per host), requesting <code>h_vmem=4G</code>. Then, 
this amount of memory is consumed, leaving 19G for further usage.

Finally, consider the resource <code>h_fsize</code>, used on HERO to specify the required scratch disk size. The resource details read:
  <nowiki>
$ qconf -sc | grep "h_fsize\|#"
#name                   shortcut       type        relop   requestable consumable default  urgency   
#----------------------------------------------------------------------------------------------------
h_fsize                 h_fsize        MEMORY      <=      YES         JOB        10G      0 
  </nowiki>
Hence, a job is eligible to run on either queue only if the requested amount of scratch disk size is less than 800G. In effect, if a job needs more than that it cannot run on a standard HERO node at all.
In such a case one has to request a big node, see [[SGE_Job_Management_(Queueing)_System#Local_disk_space_.28HERO_only.29 | here]].

== Listing slots occupied via a particular queue ==
So as to list how many jobs there are running through a particular queue on hosts within its associated hostlist you can simply use the command <code>qstat</code>. E.g., to list the jobs that run via the
queue <code>mpc_std_shrt.q</code> on the host in its hostlist (i.e. the hosts contained in <code>@mpcs</code>) simply type:
  <nowiki>
$ qstat -f -l qname=mpc_std_shrt.q 

queuename              qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
mpc_std_shrt.q@mpcs001 BP    0/9/12         9.27     lx26-amd64    
---------------------------------------------------------------------------------
mpc_std_shrt.q@mpcs002 BP    0/9/12         9.11     lx26-amd64    
---------------------------------------------------------------------------------
mpc_std_shrt.q@mpcs003 BP    0/8/12         12.19    lx26-amd64    
...
---------------------------------------------------------------------------------
mpc_std_shrt.q@mpcs017 BP    0/6/12         6.31     lx26-amd64    
 873601 0.50500 ksp_L1024  alxo9476     r     08/25/2013 01:37:51     1 31
...
  </nowiki>
Information on how many jobs are running via the specified queue are given by the three-tuple of numbers in the third column of the list. These specify the number of reserved/used/total slots on the
respective host. Your own jobs are listed underneath their respective host-entry as you can see for one of my jobs running on host <code>mpcs017</code>.
E.g., as can be seen from the first entry in the list, the host <code>mpcs001</code> has 9 out of 12 possible slots occupied with jobs supplied via the <code>mpcs_std_shrt.q</code>.
In principle there are 3 slots left which might be occupied by jobs from other queues that run on that host (if, e.g., enough memory resources are available to do so). As a detail, in order to list the overall
number of jobs on a particular host you might use <code>qstat</code> in conjunction with the <code>hostname</code> keyword to filter for host <code>mpcs001</code>. E.g., to see in detail what is going on at host <code>mpcs001</code>
you might type
  <nowiki>
$ qstat -f -l hostname=mpcs001

queuename              qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
mpc_std_shrt.q@mpcs001 BP    0/9/12         10.21    lx26-amd64    
---------------------------------------------------------------------------------
mpc_std_long.q@mpcs001 BP    0/0/12         10.21    lx26-amd64      
  </nowiki>
Apparently an instance of the queues <code>mpc_std_shrt.q</code> and <code>mpc_std_long.q</code> are running on that host (this does not come as a surprise since both queues have identical hostlists). 
However there are only 9 out of 12 slots occupied. In principle, the scheduler follows a ''fill up'' rule wherein jobs are assigned to a host until it is filled up before the next host is considered. According to the 
above list, host <code>mpcs002</code> has already 9 slots filled while host <code>mpcs001</code> still has 3 vacant slots. Why is this? Well, there are many possible reasons for that. In 90 percent of the cases the reason is that, albeit host <code>mpcs001</code>
offers further slots, it cannot offer further memory for a job. That this is also the case here you might check by monitoring the current value of the consumable resource <code>h_vmem</code> for that host. Therefore you simply have
to type:
  <nowiki>
$ qstat -F -l hostname=mpcs001 | grep "qname\|h_vmem"

	hc:h_vmem=360.000M
	qf:qname=mpc_std_shrt.q
	hc:h_vmem=360.000M
	qf:qname=mpc_std_long.q 
  </nowiki>
This shows that for both queues only <code>h_vmem=360M</code> are available. Usually there is no job requesting less than that amount of memory and consequently the slots will not be occupied!

== Listing the overall number of free slots ==

A great tool that allows you to view the current cluster status in terms of occupied/free slots and nodes is <code>qfreenodes</code>. Called on the command line it lists the 
details of occupied/free slots and nodes for all availables host-types:
  <nowiki>
$ qfreenodes

Free nodes:
   cfdl:   0 (   0 cores) of 122 (1464 cores)
   cfdh:  25 ( 300 cores) of  64 ( 768 cores)
   cfdx:   5 (  40 cores) of   7 (  56 cores)
   glmd:   8 (  32 cores) of  42 ( 168 cores)
   mpcb:   0 (   0 cores) of  20 ( 240 cores)
   mpcs:   6 (  60 cores) of 130 (1548 cores)
   mpcx:   0 (   0 cores) of   0 (   0 cores)

Flagged free nodes (not included above):
(potentially not usable due to errors, use -c for more information)
   cfdx:   1
   glmd:   2
   mpcb:   1

Cores:
   cfdl:   12 free, 1452 used,    0 reserved
   cfdh:  300 free,  468 used,    0 reserved
   cfdx:   55 free,    1 used,    0 reserved
   glmd:  136 free,   32 used,    0 reserved
   mpcb:   69 free,  171 used,    0 reserved
   mpcs:  502 free, 1046 used,    0 reserved
   mpcx:    0 free,    0 used,    0 reserved
  </nowiki>
However, note that the ''free'' vs. ''used'' description has to be taken with a grain of salt, since, due to a lack of other resources (e.g. memory, as discussed above), a free slot might actually not be able to be occupied by means of a job.  Here it can be usefull to use the command <code>qhost</code> with the specific resource requirements. E.g. to see if a single core job which needs 30G memory will run you can  type
  <nowiki>
qhost -l h_vmem=30G -l slots=1
HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
-------------------------------------------------------------------------------
cfdh040                 lx26-amd64     12  0.00   47.1G    1.4G     0.0     0.0
cfdh042                 lx26-amd64     12  0.00   47.1G    1.4G     0.0     0.0
...
mpcb015                 lx26-amd64     12  0.00   47.1G  235.6M     0.0     0.0
mpcb016                 lx26-amd64     12  0.00   47.1G  241.8M     0.0     0.0
mpcb017                 lx26-amd64     12  0.00   47.1G  188.3M     0.0     0.0
mpcb020                 lx26-amd64     12  0.00   47.1G  235.4M     0.0     0.0
  </nowiki>
Here you get a list which fulfill the specified requirements.

== Examples: addressing particular hardware components/queues==
Subsequently, a few examples for the resource specifications required to address particular [[HPC_Facilities_of_the_University_of_Oldenburg | hardware components]] / [[Queues_and_resource_allocation#Listing_all_possible_queues| queues]] are
detailed. For completeness, a list relating the different hosts and queue instances (separately for both HPC components) running thereon is given below:
 
{| border="1" class="wikitable"
|+ HPC component: FLOW
! Hosts
! Host group list
! List of queue instances
|-
| width='160pt'| cfdh001 … cfdh064
| width='100pt'| @cfdh
| width='400pt'| cfd_him_shrt.q, cfd_him_long.q
|-
| cfdl001 … cfdl122
| @cfdl
|  cfd_lom_shrt.q, cfd_lom_long.q, cfd_lom_serl.q
|-
| cfdx003 … cfdx007
|
|  cfd_xtr_expr.q
|-
| cfdx001 ... cfdx002
|
|  cfd_xtr_iact.q
|}

{| border="1" class="wikitable"
|+ HPC component: HERO
! Hosts
! Host group list
! List of queue instances
|-
| width='160pt' | glmd001 … glmd042
| width='100pt' | @glmd
| width='400pt' | glm_dlc_shrt.q, glm_dlc_long.q
|-
| glmq001 … glmq015
| @glmq
|  glm_qdc_shrt.q, glm_qdc_long.q
|-
| mpcb001 … mpcb020
| @mpcb
|  mpc_big_shrt.q, mpc_big_long.q
|-
| mpcs001 … mpcs124
| @mpcs
|  mpc_std_shrt.q, mpc_std_long.q
|-
| mpcs125 … mpcs130
| @mpcx
|  mpc_xtr_subq.q, mpc_xtr_iact.q, mpc_xtr_ctrl.q
|-
| uv100
|
|  uv100_smp_shrt.q, uv100_smp_long.q
|}

As pointed out earlier, in some cases it might be useful to know which resource allocation statements
in your job submission script allow to choose a particular queue instance (or node type) for a job.
Below, the example program <code>myExample</code> is the simple serial program shown [[Brief_Introduction_to_HPC_Computing#Example_using_the_GNU_compiler_collection_.28gcc.29| here]].

=== Resource allocation statements required for <code>mpc_std_shrt.q</code> ===

To set up a job submission script that contains all the resource allocation statements needed to place a job in an instance
of the <code>mpc_std_shrt.q</code> queue, lets list the characteristic properties of that queue using a call to the <code>qconf</code> tool, first:

  <nowiki>
$ qconf -sq mpc_std_shrt.q | grep "qname\|hostlist\|complex_values\|h_rt"

qname                 mpc_std_shrt.q
hostlist              @mpcs
complex_values        h_vmem=23G,h_fsize=800G,cluster=hero
h_rt                  192:0:0
  </nowiki>

As can be seen, a job placed in this queue will run on one of the hosts contained in the hostlist <code>@mpcs</code>, i.e. one of the execution hosts <code>mpcs001 … mpcs124</code>. So as to
eventually (!; see discussion below) place a job in a queue instance of this type (e.g. the queue instance <code>mpc_std_shrt.q@mpcs001</code>), your job submission script should
look similar to the following:
  <nowiki>
#!/bin/bash

#$ -S /bin/bash
#$ -cwd

## resource allocation statements:
#$ -l h_rt=0:10:0
#$ -l h_vmem=200M
#$ -l h_fsize=100M

#$ -N test

./myExample
  </nowiki>
Albeit you ''should'' provide estimates for the requested resources, you are not forced to do so. Most of the resources you might want to request have meaningful default values:
* default scratch space requirement: <code>h_fsize=10G</code>,
* default memory requirement: <code>h_vmem=1200M</code>,
* default running time: <code>h_rt=0:0:0</code>. 

Thus, at least for the requested running time you should provide a nonzero estimate. 

'''NOTE:'''
* If your job needs to run longer than 192h (i.e. 8d), you need to request an instance of a long queue by adding a line with content <code>#$ -l longrun=True</code> to your job submission script, see [[Queues_and_resource_allocation#Resource_allocation_statements_required_for_mpc_std_long.q| here]].
* If your job needs more than 800G scratch space, you need to request a queue instance on a big node by adding a line with content <code>#$ -l bignode=True</code> to your job submission script, see [[Queues_and_resource_allocation#Resource_allocation_statements_required_for_mpc_big_shrt.q| here]].
* The same resource allocation statements suffice for the queue <code>mpc_xtr_subq.q</code>, with queue instances on the host in <code>@mpcx</code> (i.e. nodes <code>mpcs125…mpcs130</code>). If nodes <code>mpcs001…mpcs124</code> are filled up already, the scheduler resorts to one of these queue instances as an alternative. The difference is: on hosts <code>mpcs125…mpcs130</code>, 2 slots, 4G memory and 200G scratch space are reserved for other, interactive tasks. One way to ensure this is to set up a separate queue with queue instances on these hosts. Since, in principle, the concept of a queue is hidden from the user (who, in principle, has to worry about resources, only) this is a convenient approach.
* A list of [[SGE_Job_Management_(Queueing)_System#Parallel_environments_.28PEs.29| parallel environments]] available for this queue can be listed by calling <code>qconf -sq mpc_std_shrt.q</code> and having a look at the entry <code>pe_list</code>. The usage of different parallel environments is illustrated [[Brief_Introduction_to_HPC_Computing| here]].

=== Resource allocation statements required for <code>mpc_std_long.q</code> ===
To set up a job submission script that contains all the resource allocation statements needed to place a job in an instance
of the <code>mpc_std_long.q</code> queue, lets list the characteristic properties of that queue using a call to the <code>qconf</code> tool, first:

  <nowiki>
$ qconf -sq mpc_std_long.q | grep "qname\|hostlist\|complex_values\|h_rt\|pe_list"

qname                 mpc_std_long.q
hostlist              @mpcs
pe_list               impi_long linda_long molcas_long smp_long
complex_values        h_vmem=23G,h_fsize=800G,cluster=hero,longrun=true
h_rt                  INFINITY
  </nowiki>

As can be seen, a job placed in this queue will run on one of the hosts contained in the hostlist <code>@mpcs</code>, i.e. one of the execution hosts <code>mpcs001 … mpcs124</code>.
So as to place a job in a queue instance of this type (e.g. the queue instance mpc_std_long.q@mpcs001), your job submission script should look similar to the following:

  <nowiki>
#!/bin/bash

#$ -S /bin/bash
#$ -cwd

## resource allocation statements:
#$ -l longrun=True
#$ -l h_rt=200:0:0
#$ -l h_vmem=200M
#$ -l h_fsize=100M

#$ -N test

./myExample
  </nowiki>

Albeit you ''should'' provide estimates for the requested resources, you are not forced to do so. Most of the resources you might want to request have meaningful default values:
* default scratch space requirement: <code>h_fsize=10G</code>,
* default memory requirement: <code>h_vmem=1200M</code>,
* default running time: <code>h_rt=0:0:0</code>
* default <code>longrun</code> flag: <code>False</code>. 

Thus, at least for the requested running time you should provide a nonzero estimate (ideally larger that 192h) and you need to force the <code>longrun</code> flag to take the boolean value <code>True</code>. 

If you need a parallel environment please note that there might be different names for the parallel environments (as you can see above).

'''NOTE:'''
* This queue is appropriate when your job needs less than 800G scratch space and more than 192h to finish.
* If your job needs more than 800G scratch space, you need to request a queue instance on a big node by adding a line with content <code>#$ -l bignode=True</code> to your job submission script, see [[Queues_and_resource_allocation#Resource_allocation_statements_required_for_mpc_big_shrt.q| here]].
* A list of [[SGE_Job_Management_(Queueing)_System#Parallel_environments_.28PEs.29| parallel environments]] available for this queue can be listed by calling <code>qconf -sq mpc_std_long.q</code> and having a look at the entry <code>pe_list</code>. The usage of different parallel environments is illustrated [[Brief_Introduction_to_HPC_Computing| here]].

=== Resource allocation statements required for <code>mpc_big_shrt.q</code> ===
To set up a job submission script that contains all the resource allocation statements needed to place a job in an instance
of the <code>mpc_big_shrt.q</code> queue, lets list the characteristic properties of that queue using a call to the <code>qconf</code> tool, first:

  <nowiki>
$ qconf -sq mpc_big_shrt.q | grep "qname\|hostlist\|complex_values\|h_rt\|pe_list"

qname                 mpc_big_shrt.q
hostlist              @mpcb
pe_list               impi linda molcas openmpi smp mdcs
complex_values        h_vmem=46G,h_fsize=2100G,cluster=hero,bignode=true
h_rt                  192:0:0
  </nowiki>

As can be seen, a job placed in this queue will run on one of the hosts contained in the hostlist <code>@mpcb</code>, i.e. one of the execution hosts <code>mpcb001 … mpcb020</code>.
So as to place a job in a queue instance of this type (e.g. the queue instance mpc_big_shrt.q@mpcb001), your job submission script should look similar to the following:

  <nowiki>
#!/bin/bash

#$ -S /bin/bash
#$ -cwd

## resource allocation statements:
#$ -l bignode=True
#$ -l h_rt=01:0:0
#$ -l h_vmem=30G
#$ -l h_fsize=100M

#$ -N test

./myExample
  </nowiki>

Albeit you ''should'' provide estimates for the requested resources, you are not forced to do so. Most of the resources you might want to request have meaningful default values:
* default scratch space requirement: <code>h_fsize=10G</code>,
* default memory requirement: <code>h_vmem=1200M</code>,
* default running time: <code>h_rt=0:0:0</code>
* default <code>bignode</code> flag: <code>False</code>. 

Thus, at least for the requested running time you should provide a nonzero estimate and you need to force the <code>bignode</code> flag to take the boolean value <code>True</code>. 

'''NOTE:'''
* This queue is appropriate when your job needs less that 192h to finish and needs more than 800G scratch space or more memory a standard node is able to offer (i.e. 23G).
* There is also a ''long'' version of this queue which might be addressed by setting the <code>longrun</code> flag to the boolean value <code>True</code>, i.e. by adding a line with content <code>#$ -l longrun=True</code> to your job submission script.
* A list of [[SGE_Job_Management_(Queueing)_System#Parallel_environments_.28PEs.29| parallel environments]] available for this queue can be listed by calling <code>qconf -sq mpc_big_shrt.q</code> and having a look at the entry <code>pe_list</code>. The usage of different parallel environments is illustrated [[Brief_Introduction_to_HPC_Computing| here]].

=== Resource allocation statements required for <code>glm_dlc_shrt.q</code> ===
To set up a job submission script that contains all the resource allocation statements needed to place a job in an instance
of the <code>glm_dlc_shrt.q</code> queue, lets list the characteristic properties of that queue using a call to the <code>qconf</code> tool, first:

  <nowiki>
$ qconf -sq glm_dlc_shrt.q |  grep "qname\|hostlist\|complex_values\|h_rt\|slots\|pe_list"

qname                 glm_dlc_shrt.q
hostlist              @glmd
pe_list               impi impi41 linda molcas mpich mpich2_mpd mpich2_smpd openmpi smp mdcs
slots                 4
complex_values        h_vmem=7G,h_fsize=250G,cluster=hero,golem=true
h_rt                  192:00:00
  </nowiki>

As can be seen, a job placed in this queue will run on one of the hosts contained in the hostlist <code>@glmd</code>, i.e. one of the execution hosts <code>glmd001 … glmd042</code>.
So as to place a job in a queue instance of this type (e.g. the queue instance glm_dlc_shrt.q@glmd001), your job submission script should look similar to the following:
  
  <nowiki>
#!/bin/bash

#$ -S /bin/bash
#$ -cwd

## resource allocation statements:
#$ -l golem=True
#$ -l h_rt=0:01:0
#$ -l h_vmem=200M
#$ -l h_fsize=100M

#$ -N test

./myExample
  </nowiki>

Albeit you ''should'' provide estimates for the requested resources, you are not forced to do so. Most of the resources you might want to request have meaningful default values:
* default scratch space requirement: <code>h_fsize=10G</code>,
* default memory requirement: <code>h_vmem=1200M</code>,
* default running time: <code>h_rt=0:0:0</code>
* default <code>golem</code> flag: <code>False</code>. 

Thus, at least for the requested running time you should provide a nonzero estimate and you need to force the <code>golem</code> flag to take the boolean value <code>True</code>. 

'''NOTE:'''
* The hosts in the hostlist <code>@glmd</code> stem from [http://www.csc.uni-oldenburg.de/index.html  GOLEM] (acronym for "Grossrechner Oldenburg für explizit multidisziplinäre Forschung"), the cluster for scientific computing (CSC) previously used by several groups in theoretical chemistry, physics and mathematical modeling.
* In contrast to the 12 slots a standard node on HERO offers, these nodes feature 4 slots, only.
* In comparison to the standard nodes on HERO, these nodes offer less scratch space: <code>h_fsize=250G</code>.
* There is also a ''long'' version of this queue which might be addressed by setting the <code>longrun</code> flag to the boolean value <code>True</code>, i.e. by adding a line with content <code>#$ -l longrun=True</code> to your job submission script.
* A list of [[SGE_Job_Management_(Queueing)_System#Parallel_environments_.28PEs.29| parallel environments]] available for this queue can be listed by calling <code>qconf -sq glm_dlc_shrt.q</code> and having a look at the entry <code>pe_list</code>. The usage of different parallel environments is illustrated [[Brief_Introduction_to_HPC_Computing| here]].

=== Resource allocation statements required for <code>glm_qdc_shrt.q</code> ===
To set up a job submission script that contains all the resource allocation statements needed to place a job in an instance
of the <code>glm_qdc_shrt.q</code> queue, lets list the characteristic properties of that queue using a call to the <code>qconf</code> tool, first:

  <nowiki>
$ qconf -sq glm_qdc_shrt.q |  grep "qname\|hostlist\|complex_values\|h_rt\|slots\|pe_list"

qname                 glm_qdc_shrt.q
hostlist              @glmq
pe_list               impi impi41 linda molcas mpich mpich2_mpd mpich2_smpd openmpi smp mdcs
slots                 8
complex_values        h_vmem=15G,h_fsize=800G,cluster=hero,golem_quad=true
h_rt                  192:00:00
 </nowiki>

As can be seen, a job placed in this queue will run on one of the hosts contained in the hostlist <code>@glmq</code>, i.e. one of the execution hosts <code>glmd001 … glmd015</code>.
So as to place a job in a queue instance of this type (e.g. the queue instance glm_qdc_shrt.q@glmq001), your job submission script should look similar to the following:
  
  <nowiki>
#!/bin/bash

#$ -S /bin/bash
#$ -cwd

## resource allocation statements:
#$ -l golem_quad=True
#$ -l h_rt=0:01:0
#$ -l h_vmem=200M
#$ -l h_fsize=100M

#$ -N test

./myExample
  </nowiki>

Albeit you ''should'' provide estimates for the requested resources, you are not forced to do so. Most of the resources you might want to request have meaningful default values:
* default scratch space requirement: <code>h_fsize=10G</code>,
* default memory requirement: <code>h_vmem=1200M</code>,
* default running time: <code>h_rt=0:0:0</code>
* default <code>golem_quad</code> flag: <code>False</code>. 

Thus, at least for the requested running time you should provide a nonzero estimate and you need to force the <code>golem_quad</code> flag to take the boolean value <code>True</code>. 

'''NOTE:'''
* The hosts in the hostlist <code>@glmd</code> stem from [http://www.csc.uni-oldenburg.de/index.html  GOLEM] (acronym for "Grossrechner Oldenburg für explizit multidisziplinäre Forschung"), the cluster for scientific computing (CSC) previously used by several groups in theoretical chemistry, physics and mathematical modeling.
* In contrast to the 12 slots a standard node on HERO offers, these nodes feature 8 slots, only.
* There is also a ''long'' version of this queue which might be addressed by setting the <code>longrun</code> flag to the boolean value <code>True</code>, i.e. by adding a line with content <code>#$ -l longrun=True</code> to your job submission script.
* A list of [[SGE_Job_Management_(Queueing)_System#Parallel_environments_.28PEs.29| parallel environments]] available for this queue can be listed by calling <code>qconf -sq glm_qdc_shrt.q</code> and having a look at the entry <code>pe_list</code>. The usage of different parallel environments is illustrated [[Brief_Introduction_to_HPC_Computing| here]].

=== Resource allocation statements required for <code>uv100_smp_shrt.q</code> ===
The characteristic properties of that queue <code>uv100_smp_shrt.q</code> can be listed using a call to the <code>qconf</code> tool:

  <nowiki>
$ qconf -sq uv100_smp_shrt.q | grep "qname\|hostlist\|user_list\|complex_values\|h_rt\|pe_list"

qname                 uv100_smp_shrt.q
hostlist              uv100.cm.cluster
pe_list               impi openmpi smp
user_lists            uv100users
xuser_lists           NONE
complex_values        slots=108,h_fsize=10000G,cluster=hero,uv100=true
h_rt                  192:00:00  </nowiki>

'''NOTE:'''
* This queue has a restricted user list. It can only be accessed by the Theoretical Chemistry group.
* Further information on the usage of the uv100 can be found [[Using_the_Altix_UV_100_system| here]].

== Overview: addressing particular hardware components/queues ==

As an overview, the resource specifications required to address particular [[HPC_Facilities_of_the_University_of_Oldenburg | hardware components]] / [[Queues_and_resource_allocation#Listing_all_possible_queues| queues]] (as discussed in the preceding section) are summarized in the subsequent table:

{| border="1" class="wikitable"
|- style="border-bottom: 2px solid black;" 
! width="120" | Queue
! width="100" | resource name
! width="100" | default value
! width="150" | upper limit
|-
| rowspan="4" style="border-bottom: 2px solid black;"| mpc_std_shrt.q 
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>192:00:0</tt>
|- 
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 23G</tt>
|- 
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>800G</tt>
|- style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>1548 [smp: 12]</tt> 
|-
| rowspan="5" style="border-bottom: 2px solid black;" | mpc_std_long.q
| <tt>longrun</tt>
| <tt>False</tt>
| 
|- 
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>Inf</tt>
|-
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 23G</tt>
|- 
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>800G</tt>
|-style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>1548 [smp_long: 12]</tt>
|-
| rowspan="5" style="border-bottom: 2px solid black;" | mpc_big_shrt.q
| <tt>bignode</tt>
| <tt>False</tt>
|
|-
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>192:00:0</tt>
|-
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 46G</tt>
|- 
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>2100G</tt>
|-style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>240 [smp: 12]</tt>
|-
| rowspan="6" style="border-bottom: 2px solid black;" | mpc_big_long.q
| <tt>bignode</tt>
| <tt>False</tt>
|
|-
| <tt>longrun</tt>
| <tt>False</tt>
|
|- 
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>Inf</tt>
|-
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 46G</tt>
|-
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>2100G</tt>
|-
|-style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>240 [smp_long: 12]</tt>
|-
| rowspan="5" style="border-bottom: 2px solid black;" | glm_dlc_shrt.q
| <tt>golem</tt>
| <tt>False</tt>
|
|-
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>192:00:0</tt>
|-
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 7G</tt>
|-
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>250G</tt>
|-
|-style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>168 [smp: 4]</tt>
|-
| rowspan="6" style="border-bottom: 2px solid black;"| glm_dlc_long.q
| <tt>longrun</tt>
| <tt>False</tt>
|
|-
| <tt>golem</tt>
| <tt>False</tt>
|
|-
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>Inf</tt>
|-
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 7G</tt>
|-
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>250G</tt>
|-
|-style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>168 [smp_long: 4]</tt>
|-
| rowspan="5" style="border-bottom: 2px solid black;"| glm_qdc_shrt.q
| <tt>golem_quad</tt>
| <tt>False</tt>
|
|-
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>192:00:0</tt>
|-
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 15G</tt>
|-
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>800G</tt>
|-
|-style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>120 [smp: 8]</tt>
|-
| rowspan="6" style="border-bottom: 2px solid black;" | glm_qdc_long.q
| <tt>longrun</tt>
| <tt>False</tt>
|
|-
| <tt>golem_quad</tt>
| <tt>False</tt>
|
|-
| <tt>h_rt</tt>
| <tt>0:00:0</tt>
| <tt>Inf</tt>
|-
| <tt>h_vmem</tt>
| <tt>1200M</tt>
| <tt> 15G</tt>
|-
| <tt>h_fsize</tt>
| <tt>10G</tt>
| <tt>800G</tt>
|-
|-style="border-bottom: 2px solid black;"
| <tt>slots</tt>
|
| <tt>120 [smp_long: 8]</tt>
|}

'''NOTE:'''
* The numerical value for the resource variable <tt>slots</tt> refers to the overall number of slots available for the respective queue. The values for <tt>smp</tt>/<tt>smp_long</tt> listed in parentheses refer to the maximal number of slots available for the [[SGE_Job_Management_(Queueing)_System#Parallel_environments_.28PEs.29| parallel environments]] <tt>smp</tt> and <tt>smp_long</tt>. The <tt>smp</tt> parallel environments require all the requested slots to be located on the same execution host. Therefore, the maximum number of slots for this parallel environment is limited by the number of slots offered by the respective host machines.
* How to display all the details for a particular queue is explained [[Queues_and_resource_allocation#Obtaining_elaborate_information_for_a_particular_queue| above]].