Difference between revisions of "HPC User Wiki 2016"

From HPC users
Jump to navigationJump to search
 
(354 intermediate revisions by 5 users not shown)
Line 1: Line 1:
= Documentation for High-Performance Computing at the University Oldenburg =
= Documentation for High-Performance Computing at the University Oldenburg =
<!-- Table of Contents von der Startseite entfernen-->
__NOTOC__
This is the HPC User Wiki for the HPC clusters CARL and EDDY. If you would like to contribute please contact {{sc}}!


{{warningbox|
{{warningbox|
This HPC User Wiki is for the new HPC cluster and is still under heavy construction! <br><br>
The current wiki for the HPC cluster ROSA and STORM/MOUSE can be found [https://hpcwiki.uol.de/ here]. Unfortunately, the new wiki is far from complete so  you may need to adapt some information from this wiki still. We are working on it :)
 
For editors: the old Wiki should be stored in the current state, i.e. pages from the old Wiki should only be used if they don't need to be changed (minor changes are ok). In case substantial changes are need a new page should be created. In case of a naming conflict add "2016" to page name and use text links. Example:  
<pre>
Page name in old Wiki: Some Old Page
Page name in new Wiki: Some Old Page 2016
Link in old Wiki: [[Some Old Page]]
Link in new Wiki: [[Some Old Page 2016|Some Old Page]]
</pre>
}}
}}


== General Information ==
== General Information ==
<center>
<center>
<br/>
---------------- [[FAQ|FAQ - Frequently Asked Questions]] ----------------
  ------------- Want to stay in contact with the HPC support and other users? Join our [https://mattermost.uni-oldenburg.de/signup_user_complete/?id=byc9quwbjjgc7cy57rwp9aphua Mattermost group!] -------------
------------------ Tired of reading long wiki articles? ------------------
------------- Check out our new [[HPC_Video_Lectures_2016|HPC Video Lectures]]! -------------
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"  
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"  
|- style="background-color:#ddddff;"
|- style="background-color:#ddddff;"
Line 24: Line 29:
|
|
* [[HPC Facilities of the University of Oldenburg 2016| Overview]]
* [[HPC Facilities of the University of Oldenburg 2016| Overview]]
* [[HPC Facilities of the University of Oldenburg#FLOW| FLOW]]
* [[HPC Facilities of the University of Oldenburg 2016#CARL| CARL]]
* [[HPC Facilities of the University of Oldenburg#HERO| HERO]]
* [[HPC Facilities of the University of Oldenburg 2016#EDDY| EDDY]]
* [[HPC Policies| HPC Policies]]
* [[HPC Policies 2016| HPC Policies]]
** [[Cost Sharing Scientific Computing]]
* [[Unix groups| Groups ]]
* [[Unix groups| Groups ]]
* [[Acknowledging_the_HPC_facilities| Acknowledging FLOW/HERO]]
** [[Group Tools|Managing your Group]]
* [[User Meetings]]
* [[Acknowledging_the_HPC_facilities 2016| Acknowledging CARL/EDDY]]
* [[User Meetings | User Meetings / User Representatives]]
 
 
''Other HPC resources:''
* [[HPC Facilities of the University of Oldenburg 2016#GOLD| GOLD]]
* [[ HLRN 2016 | HLRN ]]
|  
|  
* [[Logging in to the system#From within the University (intranet) | From University]]
* [[Quickstart Guide | Quickstart Guide]]
* [[Logging in to the system#From outside the University (internet) | From Home]]
* [[Quickstart Guide#HPC Cluster Overview | HPC Cluster Overview]]
* [[Quickstart Guide#Account creation | Account creation]]
* [[Quickstart Guide#Login | Login]]
* [[Quickstart Guide#File System | File System]]
* [[Quickstart Guide#Software and Environment | Software and Environment]]
* [[Quickstart Guide#Basic Job Submission | Basic Job Submission]]
* [[FAQ | Frequently Asked Questions]]
|
|
* [[User environment - The usage of module| Usage of module]]
* [[Login | Login to the HPC cluster]]
** [[Login#HPC_Account | Getting an HPC account]]
** [[Login#Connect to CARL/EDDY using the Remote Desktop Protocol | Remote Desktop Connection]]
** [[Login#Using SSH Keys for login | Using SSH Keys]]
** [[Login#VPN:_Internet_.2F_From_Home | VPN: Connect from Home]]
* [[User environment - The usage of module 2016| Usage of module]]
** [[Software Environments and their Differences]]
** [[Installed Application Software and Libraries]]
<!-- ** [[List of Modules with Category and Dependency]] -->
* [[File system and Data Management]]
* [[File system and Data Management]]
* [[Mounting Directories of FLOW and HERO#Windows | Shares under Windows]]
* [[Local Mounting of File Systems]]
* [[Mounting Directories of FLOW and HERO#Linux | Shares under Linux]]
** [[Local Mounting of File Systems#Shares under Windows | Shares under Windows]]
** [[Local Mounting of File Systems#Shares under Linux | Shares under Linux]]
** [[Local Mounting of File Systems#Mounting a Directory on Nextcloud | Mounting a Directory on Nextcloud]]
* [[License servers]]
* [[License servers]]
|
|
* [[SGE Job Management (Queueing) System| Overview]]
* [[SLURM Job Management (Queueing) System| Overview]]
* [[SGE Job Management (Queueing) System#Submitting jobs| Submitting ]]
* [[SLURM Job Management (Queueing) System#Submitting jobs| Submitting Jobs ]]
* [[SGE Job Management (Queueing) System#Specifying job requirements| Job requirements ]]
* [[Partitions|Partitions]]
* [[SGE Job Management (Queueing) System#Parallel environments (PEs) | Parallel jobs ]]
* [[SLURM Job Management (Queueing) System#Information on sbatch-options| Information on sbatch-options ]]
* [[SGE Job Management (Queueing) System#Interactive jobs | Interactive jobs ]]
* [[SLURM Job Management (Queueing) System#Documentation| Documentation]]
* [[SGE Job Management (Queueing) System#Monitoring and managing your jobs | Commands ]]
* [[Information on used Resources]]
* [[SGE Job Management (Queueing) System#Array jobs| Job arrays  ]]
* [[Parallel Jobs with MPI]]
* [[SGE Job Management (Queueing) System#Environment variables | Environment variables]]
* [[Interactive Jobs]]
* [[Brief_Introduction_to_HPC_Computing#Checking_the_status_of_the_job | Checking the job status]] [[Brief_Introduction_to_HPC_Computing#Checking_the_status_of_the_job_2| (par. jobs)]]
* [[GPU Usage]]
* [[Brief_Introduction_to_HPC_Computing#Details_for_finished_jobs| Obtaining details for finished jobs]]
* [[Best Practice]]
* [[SGE Job Management (Queueing) System#Documentation | Documentation]]
** [[How to Manage Many Jobs]]
* [[Queues_and_resource_allocation| On Queues and resource allocation]]
|-
|-
|}
</center>
== Application Software and Libraries ==
<center>
---------------------- [[Installed Application Software and Libraries | What's installed and what's new?]] ----------------------
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0" align="center"
|- style="background-color:#ddddff;"
! scope="col" width="200px" | Compiler and Development Tools
! scope="col" width="200px" | Quantum Chemistry
! scope="col" width="400px" colspan="2" | Biology & Medicine
! scope="col" width="200px" | Mathematics, Statistics & Scripting
|- valign="top"
|
* [[ALE 2016 | ALE ]]
* [[CMake 2016 | CMake]]
* [[GNU Compiler 2016 | GNU Compiler]]
* [[Go 2016 | Go]]
* [[Intel Compiler 2016 | Intel Compiler]]
* [[Intel Tools]]
** [[Threading Building Blocks | Threading Building Blocks (TBB)]]
** [[Inspector]]
** [[VTune]]
** [[Advisor]]
* [[Lua 2016 | Lua]]
* [[MPI Libraries | MPI Libraries ]]
** [[MPI Libraries#Compiling with OpenMPI | Compiling with OpenMPI]]
** [[MPI Libraries#Compiling with Intel MPI | Compiling with Intel MPI]]
* [[PGI Compiler 2016 | PGI Compiler]]
** [[OpenACC]]
*[[Parallel 2016 | parallel]]
* [[Qt 2016 | Qt]]
* [[ROS 2016 | ROS]]
* [[Singularity 2016 | Singularity]]
* [[Spark 2016 | Spark]]
* [[Toolchains | Toolchains]]
|
* [[AIMAll 2016 | AIMAll]]
* [[CP2K 2016 | CP2K]]
* [[CRYSTAL]]
* [[DIAMOND 2016 | DIAMOND]]
* [[Gaussian 2016 | Gaussian]]
** [[GaussView]]
* [[GROMACS]]
* [[MOLGW 2016 | MOLGW]]
* [[NAMD]]
* [[NBO 2016 | NBO]]
* [[ORCA 2016 | ORCA]]
* [[pDynamo]]
* [[PLUMED]]
* [[QuantumESPRESSO 2016 | QuantumESPRESSO]]
| 
*[[angsd 2016 | angsd]]
*[[ANTs 2016 | ANTs]]
*[[AUGUSTUS 2016 | AUGUSTUS]]
*[[bam_to_mate_hist 2016 | bam_to_mate_hist]]
*[[bwa-mem2 2016 | bwa-mem2]]
*[[BCFtools]]
*[[Beast]]
*[[BEDTools]]
*[[bgc 2016 | bgc]]
*[[BLAST]]
*[[BLAT]]
*[[BamTools]]
*[[BBMap 2016 | BBMap]]
*[[BioPerl]]
*[[Biopython]]
*[[Bowtie 1/Bowtie 2]]
*[[BRAKER2 2016 | BRAKER2]]
*[[Burrows-Wheeler Aligner]]
*[[canu 2016 | canu]]
*[[cdbfasta 2016 | cdbfasta]]
*[[CD-HIT]]
*[[CellRanger 2016 | CellRanger]]
*[[cutadapt]]
*[[discovardenovo 2016 | discovardenovo]]
*[[Exonerate 2016 | Exonerate]]
*[[FastANI 2016 | FastANI]]
*[[FASTX-Toolkit]]
*[[FastQC]]
*[[FreeSurfer 2016 | FreeSurfer]]
*[[FSL 2016 | FSL]]
*[[GATK]]
*[[GATE 2016 | GATE]]
*[[GenomeThreader 2016 | GenomeThreader]]
*[[GeneMark-EX 2016 | GeneMark-EX]]
*[[GenomeTools 2016 | GenomeTools]]
*[[GffCompare 2016 | GffCompare]]
*[[gffread 2016 | gffread]]
*[[hapflk 2016 |hapflk]]
*[[HISAT2 2016 | HISAT2]]
*[[IGV and IGVTools]]
*[[IQ-TREE 2016 | IQ-TREE]]
*[[ipyrad]]
*[[ ITK 2016 | ITK ]]
*[[jcvi 2016 | jcvi]]
*[[Kraken 2016 | Kraken]]
*[[Kaldi 2016 | Kaldi]]
*[[Kaiju 2016 | Kaiju]]
|
*[[Lighter]]
*[[MAFFT 2016 | MAFFT]]
*[[minimap2 2016 | minimap2]]
*[[MUSCLE 2016 | MUSCLE]]
*[[lpsolve 2016 | lpsolve]]
*[[Lumpy]]
*[[MCScanX 2016 | MCScanX]]
*[[medaka 2016 | medaka]]
*[[MIRTK 2016 | MIRTK]]
*[[MrBayes 2016 | MrBayes]]
*[[MultiQC 2016 | MultiQC]]
*[[NextGenMap]]
*[[NiBabel 2016 | NiBabel]]
*[[nQuire 2016 | nQuire]]
*[[OrthoFinder 2016 | OrthoFinder]]
*[[OMA 2016 | OMA]]
*[[PEST++ 2016 | PEST++]]
*[[Pychopper 2016 | Pychopper]]
*[[picard 2016 | picard]]
*[[Pilon 2016 | Pilon]]
*[[Picard]]
*[[QUAST 2016 | QUAST / QUAST-LG]]
*[[Racon 2016 | Racon]]
*[[RepeatModeler 2016 | RepeatModeler]]
*[[REPET 2016 | REPET]]
*[[RSEM 2016 | RSEM]]
*[[Salmon 2016 | Salmon]]
*[[SAMtools]]
*[[SHAPEIT4 2016 | SHAPEIT4]]
*[[Sickle 2016 | Sickle]]
*[[SPAdes 2016 | SPAdes]]
*[[spaln 2016 | spaln]]
*[[Mindboggle 2016 | Mindboggle]]
*[[SNAP 2016 | SNAP]]
*[[SNAP-HMM 2016 | SNAP-HMM]]
*[[SOAPdenovo2]]
*[[Stacks]]
*[[STAR 2016 | STAR]]
*[[StringTie 2016 | StringTie]]
*[[TopHat 2016 | TopHat]]
*[[TransDecoder 2016 | TransDecoder 2016]]
*[[Trinity]]
*[[TopHat]]
*[[Trimmomatic]]
*[[VCFTools]]
*[[ViennaRNA]]
*[[wtdbg 2016 | wtdbg2 ]]
*[[Fiji/Unet 2016 | Fiji/Unet]]
|
* [[dakota 2016 | dakota]]
* [[Gurobi 2016 | Gurobi]]
* [[Julia 2016 | Julia]]
* [[Maple 2016 | Maple]]
* [[MATLAB 2016 | MATLAB]] 
** [[Configuration MDCS 2016 | Configuration MDCS]]
** [[Basic Examples MDCS 2016 | Basic Examples MDCS]]
** [[Advanced Examples MDCS 2016 | Advanced Examples MDCS]]
** [[MDCS Troubleshooting]]
** [[CVX]]
** [[MDCS with GPUs]]
** [[IQMtools]]
* [[Mmg 2016 | Mmg]]
* [[NEURON 2016 | NEURON]]
*[[osrm-backend 2016 | osrm]]
*[[parallel 2016 | parallel]]
* [[Perl 2016 | Perl]]
* [[Python 2016 | Python]]
** [[JupyterLab]]
* ScientificPython
** [[Anaconda 2016 | Anaconda3 / Miniconda3]]
* [[R 2016 | R]]
** R-bundle-Bioconductor
** [[RStudio 2016 | RStudio]]
* [[RStan 2016 | RStan]]
*[[SALOME 2016 | SALOME ]]
* [[STATA 2016 | STATA]]
* [[TCL 2016 | TCL]]
|-
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"
|- style="background-color:#ddddff;"
! scope="col" width="200px" | Libraries
! scope="col" width="200px" | Physics
! scope="col" width="200px" | Machine Learning
! scope="col" width="178px" | Computational Fluid Dynamics
! scope="col" width="200px" | Visualisation
|- valign="top"
|
* [[BLAS and LAPACK 2016 | BLAS and LAPACK]]
* [[FLAC 2016 | FLAC]] 
* [[libsamplerate 2016 | libsamplerate]]
* [[MariaDB 2016 | MariaDB]]
* [[PyTables 2016 | PyTables]]
* [[SuiteSparse 2016 | SuiteSparse]]
|
* [[EGSnrc 2016 | EGSnrc]]
* [[ElmerFEM 2016 | ElmerFEM]]
* [[FLUKA 2016 | FLUKA]]
* [[fluka.cern | fluka.cern]]
* [[FDTD_Solutions / Lumerical 2016 | FDTD Solutions / Lumerical]]
* [[GEANT4 2016| GEANT4]]
** [[GEANT4 2016#GRAS | GRAS]]
** [[GEANT4 2016#GATE | GATE]]
** [[GEANT4 2016#g4py| g4py]]
* [[HDF5 2016 | HDF5]]
* [[LEDA  2016 | LEDA]]
* [[mumax 2016 | mumax]]
* [[NetCDF 2016 | NetCDF]]
* [[Octopus 2016 | Octopus ]]
* [[PVMOS 2016 | PVMOS]]
* [[SCUFF-EM 2016 | SCUFF-EM]]
* [[libsamplerate 2016 | sndfile-tools]]
* [[VASP 2016 | VASP]]
** [[p4vasp 2016 | p4vasp]]
* [[Yambo 2016 | Yambo]]
|
* [[Caffe]]
* [[Keras 2016 | Keras]]
* [[Tensorflow 2016 | TensorFlow]]
* [[Theano 2016 | Theano]]
* [[KNIME 2016 | KNIME]]
* [[PyTorch 2016 | PyTorch]]
|
* [[ESMF 2016 | ESMF]]
* [[GMT 2016 | GMT]]
* [[Install PALM on Eddy | PALM]]
* [[OpenFOAM_2016 | OpenFOAM]]
* [[THETA_2016 | THETA]]
* [[WRF/WPS_2016 | WRF/WPS]]
|
* [[Ghostscript 2016 | Ghostscript ]]
* [[Mesa 2016 | Mesa]]
* [[NCL]]
* [[ncview]]
* [[paraview]]
* [[PCL 2016 | PCL]]
|}
|}


== Courses and Tutorials ==


Below you can find a list of Courses and Tutorials for our HPC clusters. Currently, a short, two-hour introduction for the new clusters CARL and EDDY is planned. During the summer semester a lecture "Introduction to HPC" is held (2h/week). If you are interested in attending or would like to request a course please contact {{sc}}.
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"
|- style="background-color:#ddddff;"
!Courses and Workshops
!Lectures
!Tutorials
|- valign="top"
|
* Introduction to the new HPC Clusters [[media:Introduction_to_High-Performance-Computing.pdf|[pdf]]]<br>(23.02.2017, 14-16pm c.t., W02 1-148)
* Introduction to the new HPC Clusters [[media:Introduction_to_High-Performance-Computing.pdf|[pdf]]]<br>(14.03.2017, 14-16pm c.t., W02 1-148)
* Introduction for EDDY users for migrating from FLOW [[media:Eddy_Introduction.pdf|[pdf]]]<br>(01.03.2017 and 02.03.2017, 10-12am c.t.)
* [[OpenACC Workshop]]<br>(28./29.3.2017, 10am-6pm, W01 0-008)
* [[Workshop Nvidia HPC SDK]]<br>(06.07.2021, 10am-1pm, online)
* [[Workshop How to AMD]]<br>(04./05.04.2023, 14pm-17pm, online)
* [[HPC Introduction | Introduction to HPC (three day course)]]
** [[HPC Introduction 03/2023 | upcoming or most recent course]]<br>(27.03-29.03.2023)
** [[HPC Introduction | all course]]
|
* [[Lecture HPC Introduction | Introduction to High-Performance Computing]]
<!-- [[HPC Video Lectures 2016 | HPC Video Lectures ]] -->
|
*[[HPC_Video_Lectures_2016 | HPC Video Lectures]]
*[[Tutorials MDCS 2016 | Matlab Tutorials]]
:*[[Audio Data Processing]]
:*[[Using the MEX Compiler]]
*[[R Tutorial]]
|}
</center>
== Contact ==
In case you need HPC support you can use the contacts listed below. The servicedesk of IT services can be contacted for all questions regarding HPC, and are recommended in particular when someone is absent (e.g. due to a vacation).
<center>
{| style="background-color:#eeeeff;" cellpadding="10" border="1" cellspacing="0"
|- style="background-color:#ddddff;"
!HPC Resource
!Name
!E-Mail
|- valign="top"
| CARL and EDDY
| General HPC Support<br> and Servicedesk
| [mailto:hpcsupport@uni-oldenburg.de hpcsupport@uni-oldenburg.de] <br> [mailto:servicedesk@uni-oldenburg.de servicedesk@uni-oldenburg.de]
|- valign="top"
| CARL
| Stefan Harfst<br> Fynn Schwietzer
| [mailto:stefan.harfst@uni-oldenburg.de stefan.harfst@uni-oldenburg.de]<br> [mailto:fynn.schwietzer1@uni-oldenburg.de fynn.schwietzer1@uni-oldenburg.de]
|-
|- valign="top"
| EDDY
| Wilke Trei
| [mailto:wilke.trei@forwind.de wilke.trei@forwind.de]
|-
|}
|}
</center>
</center>
<br/>
---------------- [[Migration|Migration to the new HPC clusters]] ----------------
The old wiki for HERO and FLOW is found [[Welcome to the HPC User Wiki of the University of Oldenburg | here]].<br><br>
<span style="color: green">[[HPC Support Internal]]</span> <br>
[[Intel CPU Bug]]<br>
[[Globus Data Transfer]]

Latest revision as of 08:06, 27 June 2024

Documentation for High-Performance Computing at the University Oldenburg

This is the HPC User Wiki for the HPC clusters CARL and EDDY. If you would like to contribute please contact Scientific Computing!

The current wiki for the HPC cluster ROSA and STORM/MOUSE can be found here. Unfortunately, the new wiki is far from complete so you may need to adapt some information from this wiki still. We are working on it :)

General Information


---------------- FAQ - Frequently Asked Questions ---------------- 

 ------------- Want to stay in contact with the HPC support and other users? Join our Mattermost group! -------------

------------------ Tired of reading long wiki articles? ------------------ 
------------- Check out our new HPC Video Lectures! -------------
HPC Facilities Quickstart Guide User environment Job Management (Queueing) System


Other HPC resources:

Application Software and Libraries

----------------------  What's installed and what's new? ----------------------
Compiler and Development Tools Quantum Chemistry Biology & Medicine Mathematics, Statistics & Scripting
Libraries Physics Machine Learning Computational Fluid Dynamics Visualisation

Courses and Tutorials

Below you can find a list of Courses and Tutorials for our HPC clusters. Currently, a short, two-hour introduction for the new clusters CARL and EDDY is planned. During the summer semester a lecture "Introduction to HPC" is held (2h/week). If you are interested in attending or would like to request a course please contact Scientific Computing.

Courses and Workshops Lectures Tutorials

Contact

In case you need HPC support you can use the contacts listed below. The servicedesk of IT services can be contacted for all questions regarding HPC, and are recommended in particular when someone is absent (e.g. due to a vacation).

HPC Resource Name E-Mail
CARL and EDDY General HPC Support
and Servicedesk 		hpcsupport@uni-oldenburg.de
servicedesk@uni-oldenburg.de
CARL Stefan Harfst
Fynn Schwietzer 		stefan.harfst@uni-oldenburg.de
fynn.schwietzer1@uni-oldenburg.de
EDDY Wilke Trei wilke.trei@forwind.de 


---------------- Migration to the new HPC clusters ----------------

The old wiki for HERO and FLOW is found here.

HPC Support Internal
Intel CPU Bug
Globus Data Transfer

Retrieved from "https://wiki.hpcuser.uni-oldenburg.de/index.php?title=HPC_User_Wiki_2016&oldid=7295"