Snakemake on NeSI

A short tutorial on using the workflow engine Snakemake on the NeSI HPC cluster

This is an introductory tutorial for working with Snakemake on NeSI high-performance computer cluster. This tutorial assumes that you have a basic knowledge of Snakemake and Bioconda to develop your bioinformatics workflows.

Note

Get a quick introduction to conda and Snakemake at https://reproducibility.sschmeier.com.

Here, you will learn how to configure the NeSI computing cluster and your workflow to run all jobs that your workflow needs to run with the Slurm system.

Currently, Dr. Sebastian Schmeier is teaching this material at Massey University in Auckland, New Zealand. More information about other bioinformatics material, tutorials, and our research can be found on the webpages of the Schmeier Group (https://sschmeier.com).

Note

A PDF-version of this tutorial can be downloaded here.

Contents

• 1. Slurm primer
  • 1.1. Slurm partitions
  • 1.2. Submitting a job with sbatch
• 2. Configuring NeSI
  • 2.1. Loading a module
  • 2.2. Installing Snakemake
• 3. Snakemake setup
  • 3.1. Getting your workflow and data onto NeSI
  • 3.2. An example workflow
    • 3.2.1. Download the workflow
    • 3.2.2. Inspect and setup the workflow
    • 3.2.3. Slurm and NeSI specific setup
• 4. Singularity
  • 4.1. Using existing Docker/Singularity images
  • 4.2. Creating a Singularity image
  • 4.3. Using one image for the whole workflow
  • 4.4. Execute Snakemake in Singularity-mode