Ipyrad
Introduction
The software ipyrad is an interactive toolkit for assembly and analysis of restriction-site associated genomic data sets (e.g., RAD, ddRAD, GBS) for population genetic and phylogenetic studies. [1]
At the moment, there is no central installation of ipyrad, however, you can easily install it yourself using Anaconda3 as described below.
Installation
To install ipyrad you first need to load a module for Anaconda3. In this example, we use Anaconda3/2020.02 which can be found in hpc-env/8.3 (if you want to use a different version/environment you can search with module av Anaconda3 or module spider Anaconda3):
[carl]$ module load hpc-env/8.3 [carl]$ module load Anaconda/2020.02
The next step is to create a new environment for ipyrad with the command:
[carl]$ conda create --name ipyrad Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /user/abcd1234/.conda/envs/ipyrad Proceed ([y]/n)? Preparing transaction: done Verifying transaction: done Executing transaction: done
The name for the environment can be freely chosen and it will be created after you have confirmed to proceed with pressing (y and)enter. You may see a warning about an outdated conda which you can safely ignore (or, if you wish, you can switch to a newer module of Anaconda3 if available).
The new environment can now be activated. We recommend using the command(*):
[carl]$ source activate ipyrad (ipyrad) [carl]$
You will notice the change of the command-line prompt to indicate the active environment. Packages that are now installed with conda install will be installed in this environment and not interfere with other software installations.
(*) The alternative conda activate requires you to use the command conda init bash first which modifies your .bashrc and more or less forces you to always use the same version of Anaconda3.
Now you can install ipyrad along with a the package mpi4py for parallel computing?
(ipyrad) [carl]$ conda install ipyrad -c bioconda (ipyrad) [carl]$ conda install mpi4py -c conda-forge
These commands will take a moment to complete but after that ipyrad is ready to use. And next time you log in or in a job script you only need the commands
[carl]$ module load hpc-env/8.3 [carl]$ module load Anaconda/2020.02 [carl]$ source activate ipyrad
to get started. If you want to leave the environment you can always type
(ipyrad) [carl]$ conda deactivate
which should return you to the normal command-line prompt.
Using ipyrad on CARL
Preparations and First Tests
Following the Introductory tutorial to the CLI you can start by downloading some test data and creating a parameter file in a new directory under $WORK:
(ipyrad) [carl]$ mkdir $WORK/ipyrad_test (ipyrad) [carl]$ cd $WORK/ipyrad_test (ipyrad) [carl]$ curl -LkO https://eaton-lab.org/data/ipsimdata.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11.8M 100 11.8M 0 0 8514k 0 0:00:01 0:00:01 --:--:-- 8508k (ipyrad) [carl]$ tar -xzf ipsimdata.tar.gz (ipyrad) [carl]$ ipyrad -n iptest New file 'params-iptest.txt' created in /gss/work/lees4820/ipyrad
The resulting file params-iptest.txt has to be opened in a text editor to add the locations of the raw non-demultiplexed fastq file and the barcodes file. With the test data the first couple of lines should look like this:
(ipyrad) [carl]$ $ head params-iptest.txt
------- ipyrad params file (v.0.9.53)-------------------------------------------
iptest ## [0] [assembly_name]: Assembly name. Used to name output directories for assembly steps
/gss/work/abcd1234/ipyrad ## [1] [project_dir]: Project dir (made in curdir if not present)
/gss/work/abcd1234/ipyrad/ipsimdata/rad_example_R1_.fastq.gz ## [2] [raw_fastq_path]: Location of raw non-demultiplexed fastq files
/gss/work/abcd1234/ipyrad/ipsimdata/rad_example_barcodes.txt ## [3] [barcodes_path]: Location of barcodes file
## [4] [sorted_fastq_path]: Location of demultiplexed/sorted fastq files
denovo ## [5] [assembly_method]: Assembly method (denovo, reference)
We recommend to use absolute file names including the full path to the file which allows you to move the parameter file to other locations (e.g. a job specific directory).
A simple test can be performed with this command:
(ipyrad) [carl]$ ipyrad -p params-iptest.txt -s 1 -c 8 ------------------------------------------------------------- ipyrad [v.0.9.53] Interactive assembly and analysis of RAD-seq data ------------------------------------------------------------- Parallel connection | hpcl004: 8 cores Step 1: Demultiplexing fastq data to Samples
The program performs the first step of a work flow (-s 1) using a total of 8 cores (-c 8) on the login node. We can now remove the newly created data files and directories with
(ipyrad) [carl]$ rm -r iptest_fastqs/ iptest.json
to avoid error message in the next steps.
Job Script for Using Multiple Cores on a Single Compute Node
With the preparations from the previous step, we can now use a job script to run ipyrad on the compute nodes. The first example uses a single compute node with multiple cores. The job script could look like this:
#!/bin/bash
#SBATCH --partition carl.p
#SBATCH --nodes 1
#SBATCH --tasks-per-node 1
#SBATCH --cpus-per-task 12
#SBATCH --time 0-01:00:00
#SBATCH --mem-per-cpu 5000
#SBATCH --job-name ipyrad
#SBATCH --output ipyrad_output_%j.txt
## assembly name
assembly_name="iptest"
## load modules and activate conda environment
module load hpc-env/8.3
module load Anaconda3/2020.02
source activate ipyrad
## change into the directory where your params file resides
cd $WORK/ipyrad
## create, prepare and change to a job specific dir
jobdir="ipyrad_${SLURM_JOB_ID}"
params="params-${assembly_name}.txt"
mkdir $jobdir
sed "s#$(pwd) #$(pwd)/$jobdir#" $params > $jobdir/$params
cd $jobdir
## setting the number of available cores
cores=1
if [ -z $SLURM_CPUS_PER_TASK ]
then
cores=${SLURM_NTASKS}
else
cores=$((SLURM_NTASKS*SLURM_CPUS_PER_TASK))
fi
## call ipyrad on your params file and perform 7 steps from the workflow
cmd="ipyrad -p $params -s 1234567 -c $cores"
echo "== starting ipyrad on $(hostname) at $(date) =="
echo "== command: $cmd"
eval $cmd
retval=$?
if [ $retval -ne 0 ]
then
echo "Warning: exit code for command $cmd is non-zero (=$retval)"
fi
echo "== completed ipyrad on $(hostname) at $(date) =="
exit $retval
Some explanations:
- the job script requests a single task with a total of 12 CPU cores (--cpus-per-task 12). Depending on the partition, the number of cores can be chosen between 1 and the maximum number of cores available. For carl.p this is 24. The the number of cores selected is automatically passed in the command-line near the end of the script.
- other resources can be requested as needed, here e.g. 5000 MB of RAM.
- the script assumes a base directory ($WORK/ipyrad where the file params-iptest.txt and the job script are located. A job-specific subdirectory is created and the params-iptest.txt is copied there (by the sed-command which also changes the project dir). If you do not want a job-specific directory you can comment-out the lines that contain jobdir (with or without the $).
When you save the job script as ipyrad_job.sh you can submit a job with
[carl]$ sbatch ipyrad_job.sh
Note that you do need to activate the conda environment for this, it will be (and has to be) done within the job.
Job Script for Using Multiple Compute Nodes
To run ipyrad on multiple compute nodes you will have to install the package mpi4py in your conda environment as explained above. Then, the job script from the previous example only needs a few modifications. The changed lines are shown below:
... #SBATCH --nodes 2 #SBATCH --tasks-per-node 24 ###SBATCH --cpus-per-task 12 ... cmd="ipyrad -p $params -s 1234567 -c $cores --MPI" ...
So, instead of requesting cpus per task you are now requesting nodes and tasks per node (each task will use a single core with --cpus-per-task commented out or set to 1). The --MPI tells ipyrad to use MPI to distribute the processes.
