Skip to content

Tutorial 3: A phylogenomic workflow with Snakemake

This tutorial builds a workflow that takes a published set of alignments to two phylogenetic trees with one command. SEGUL does the alignment work, and other software infers the trees. You will:

  1. Install Snakemake.
  2. Describe the software of each step.
  3. Write the workflow, one rule at a time.
  4. Run it and check the results.
  5. Change a setting and rerun only the steps it affects.
SoftwareRole in the workflow
SnakemakeRuns the steps in order, in parallel where it can, and only when their input changed.
SEGULSummarizes, filters, trims, and concatenates the alignments.
IQ-TREEInfers a maximum likelihood tree from the supermatrix, and one gene tree per locus.
ASTRALEstimates a species tree from the gene trees.

You install only Snakemake. It installs the other three from Bioconda the first time the workflow runs.

  • A computer with Linux or macOS. On Windows, use WSL.
  • Conda 24.7.1 or newer. Miniforge is a good way to get it. Check with conda --version.
  • About 1 GB of free storage, and an internet connection for the first run.
  • Basic use of a terminal. You do not need to know Snakemake or Python.

With 8 cores, the whole workflow takes about 10 minutes.

The data are the same shrew UCE alignments as in Tutorial 1, from Esselstyn et al. 2021: 4,040 NEXUS alignments of 102 taxa, in a 14.6 MB archive on Zenodo. The workflow downloads it.

StepRuleSoftwareOutput
Download and unpack the alignmentsdownload, unpackcurl, unzipdata/loci
Summarize the raw alignmentssummary_rawSEGULresults/01-summary-raw
Keep loci with enough taxafilterSEGULresults/02-filtered
Trim sites with missing datatrimSEGULresults/03-trimmed
Summarize the clean alignmentssummary_cleanSEGULresults/04-summary-clean
Concatenate the lociconcatSEGULresults/05-supermatrix
Infer the concatenated treeconcat_treeIQ-TREEresults/06-concat-tree
Infer a tree per locusgene_treeIQ-TREEresults/07-gene-trees
Estimate the species treecollect_gene_trees, species_treeASTRALresults/08-species-tree

Create a conda environment with Snakemake and activate it:

Terminal window
conda create --name snakemake --channel conda-forge --channel bioconda snakemake-minimal conda
Terminal window
conda activate snakemake

Check the installation. This tutorial was tested with Snakemake 9.27:

Terminal window
snakemake --version

A Snakemake project is a folder with a Snakefile, a configuration file, and one file per software environment:

segul-phylo/
├── Snakefile
├── config.yaml
└── envs/
├── segul.yaml
├── iqtree.yaml
└── aster.yaml
Terminal window
mkdir -p segul-phylo/envs
Terminal window
cd segul-phylo

Create the files in the next sections with any text editor.

Each file in envs lists the conda channels and one package with its version. Fixed versions give the same results on another computer or a year from now.

channels:
- conda-forge
- bioconda
dependencies:
- segul=0.25.0
channels:
- conda-forge
- bioconda
dependencies:
- iqtree=3.1.4

The aster package contains ASTRAL:

channels:
- conda-forge
- bioconda
dependencies:
- aster=1.25

Settings you may want to change go in config.yaml, so you never edit the commands themselves:

# Alignments to download: a zip archive of NEXUS files, one per locus.
dataset_url: "https://zenodo.org/records/6459213/files/esselstyn_et_al_2021_uce_aln.zip?download=1"
# Name of the supermatrix and the trees.
prefix: shrews
# Keep loci with at least this proportion of the taxa.
min_taxa: 0.75
# Remove sites where more than this proportion of the taxa is missing.
max_missing: 0.5
# Substitution model for IQ-TREE.
model: GTR+G
# Extra IQ-TREE options. `--fast` runs a quick tree search, which suits a
# first look. Set it to "" for a full search.
iqtree_options: "--fast"

A Snakefile is a list of rules. A rule names its input files, its output files, and the shell command that makes one from the other. Snakemake reads the rules backward from the files you ask for and works out which commands to run.

Add each block below to the Snakefile, in order.

The first lines load the settings. The rule all lists the final files of the workflow: the two summaries and the two trees.

configfile: "config.yaml"
PREFIX = config["prefix"]
TRIMMED = "results/03-trimmed/trimmed_alignments"
rule all:
input:
"results/01-summary-raw/raw_alignment_summary.txt",
"results/04-summary-clean/clean_alignment_summary.txt",
f"results/06-concat-tree/{PREFIX}.treefile",
f"results/08-species-tree/{PREFIX}.tre",
rule download:
output:
"data/alignments.zip",
params:
url=config["dataset_url"],
shell:
"curl --location --silent --show-error --output {output} '{params.url}'"
rule unpack:
input:
"data/alignments.zip",
output:
directory("data/loci"),
shell:
"unzip -q -j {input} '*.nex' -d {output}"

directory() marks an output that is a folder. unzip -j leaves out the folder inside the archive, so the alignments land in data/loci.

rule summary_raw:
input:
"data/loci",
output:
"results/01-summary-raw/raw_alignment_summary.txt",
"results/01-summary-raw/raw_locus_summary.csv",
"results/01-summary-raw/raw_taxon_summary.csv",
log:
"logs/summary-raw.log",
conda:
"envs/segul.yaml"
shell:
"segul align summary --dir {input} --input-format nexus "
"--output results/01-summary-raw --prefix raw --force --log {log}"
rule filter:
input:
"data/loci",
output:
directory("results/02-filtered"),
params:
min_taxa=config["min_taxa"],
log:
"logs/filter.log",
conda:
"envs/segul.yaml"
shell:
# SEGUL adds the threshold to the folder name, such as `_75p`.
"segul align filter --dir {input} --input-format nexus "
"--percent {params.min_taxa} --output {output} --log {log} && "
"mv {output}_*p {output}"

Four things make SEGUL fit a workflow:

  • conda: names the environment file, so the rule runs with that SEGUL version.
  • --log {log} sends SEGUL's log file to the logs folder. Without it, every step writes segul.log in the project folder.
  • --force lets SEGUL write into a folder that already exists. Snakemake creates the folder of a rule's output files before it runs the command.
  • params: passes a setting from config.yaml to the command, here --percent {params.min_taxa}.

segul align filter adds the threshold to the name of its output folder, such as 02-filtered_75p. The mv command gives it back the name that the rule declares.

checkpoint trim:
input:
"results/02-filtered",
output:
directory("results/03-trimmed"),
params:
max_missing=config["max_missing"],
log:
"logs/trim.log",
conda:
"envs/segul.yaml"
shell:
"segul align trim --dir {input} --input-format nexus "
"--missing-data {params.max_missing} --output {output} --log {log}"
def trimmed_dir(wildcards):
"""The trimmed alignments, once the trim checkpoint has run."""
checkpoints.trim.get()
return TRIMMED
rule summary_clean:
input:
trimmed_dir,
output:
"results/04-summary-clean/clean_alignment_summary.txt",
"results/04-summary-clean/clean_locus_summary.csv",
"results/04-summary-clean/clean_taxon_summary.csv",
log:
"logs/summary-clean.log",
conda:
"envs/segul.yaml"
shell:
"segul align summary --dir {input} --input-format nexus "
"--output results/04-summary-clean --prefix clean --force --log {log}"
rule concat:
input:
trimmed_dir,
output:
alignment=f"results/05-supermatrix/{PREFIX}.phy",
partition=f"results/05-supermatrix/{PREFIX}_partition.txt",
params:
prefix=PREFIX,
log:
"logs/concat.log",
conda:
"envs/segul.yaml"
shell:
"segul align concat --dir {input} --input-format nexus "
"--output-format phylip --partition-format raxml "
"--output results/05-supermatrix --prefix {params.prefix} --force --log {log}"

trim is a checkpoint, not a rule. Snakemake cannot know which loci pass the filter before it runs, so it cannot plan one gene tree per locus yet. A checkpoint tells Snakemake to look at the files again after this step and extend its plan.

segul align trim writes the alignments to a trimmed_alignments folder inside its output folder, next to trimming_summary.csv. The function trimmed_dir waits for the checkpoint, then gives that folder to the rules that need it.

rule concat_tree:
input:
f"results/05-supermatrix/{PREFIX}.phy",
output:
f"results/06-concat-tree/{PREFIX}.treefile",
params:
prefix=f"results/06-concat-tree/{PREFIX}",
model=config["model"],
extra=config["iqtree_options"],
threads: 8
conda:
"envs/iqtree.yaml"
shell:
"iqtree -s {input} -m {params.model} {params.extra} -T {threads} "
"--prefix {params.prefix} --redo --quiet"

threads: 8 lets IQ-TREE use up to 8 cores. --redo lets IQ-TREE overwrite the files of an earlier run. Without it, IQ-TREE stops when Snakemake reruns the rule.

rule gene_tree:
input:
TRIMMED + "/{locus}.nex",
output:
"results/07-gene-trees/{locus}.treefile",
params:
prefix="results/07-gene-trees/{locus}",
model=config["model"],
extra=config["iqtree_options"],
conda:
"envs/iqtree.yaml"
shell:
"iqtree -s {input} -m {params.model} {params.extra} -T 1 "
"--prefix {params.prefix} --redo --quiet"
def gene_trees(wildcards):
"""One gene tree per alignment that the trim checkpoint wrote."""
checkpoints.trim.get()
loci = glob_wildcards(TRIMMED + "/{locus}.nex").locus
return expand("results/07-gene-trees/{locus}.treefile", locus=loci)
rule collect_gene_trees:
input:
gene_trees,
output:
"results/08-species-tree/gene-trees.tre",
run:
with open(output[0], "w") as out:
for tree in input:
with open(tree) as file:
out.write(file.read())
rule species_tree:
input:
"results/08-species-tree/gene-trees.tre",
output:
f"results/08-species-tree/{PREFIX}.tre",
threads: 8
log:
"logs/species-tree.log",
conda:
"envs/aster.yaml"
shell:
"astral4 -t {threads} -i {input} -o {output} 2> {log}"

gene_tree has a wildcard, {locus}. It is one rule that Snakemake runs once per locus, several at a time. The function gene_trees lists the trimmed alignments after the checkpoint and asks for one tree for each. collect_gene_trees joins the trees into one file, the input that ASTRAL needs.

A dry run prints what Snakemake would do without running it:

Terminal window
snakemake --dry-run --cores 8

The job table at the end should list 11 jobs:

job count
------------------ -------
download 1
unpack 1
filter 1
summary_raw 1
trim 1
collect_gene_trees 1
concat 1
summary_clean 1
concat_tree 1
species_tree 1
all 1
total 11

The gene trees are missing, because Snakemake adds them after the trim checkpoint. An error here usually means a typo or a wrong indent in the Snakefile. Snakemake names the line.

Terminal window
snakemake --cores 8 --sdm conda
  • --cores 8 is the number of cores Snakemake may use. Jobs run side by side up to this limit.
  • --sdm conda creates the environments in envs and runs each rule in its own. This takes a few minutes on the first run only.

After the trim checkpoint, the plan grows to 994 steps: one gene tree for each of the 983 loci that passed the filter. The run ends with:

994 of 994 steps (100%) done
results/
├── 01-summary-raw/ raw_alignment_summary.txt, locus and taxon tables
├── 02-filtered/ 983 alignments
├── 03-trimmed/ trimmed_alignments/, trimming_summary.csv
├── 04-summary-clean/ clean_alignment_summary.txt, locus and taxon tables
├── 05-supermatrix/ shrews.phy, shrews_partition.txt
├── 06-concat-tree/ shrews.treefile
├── 07-gene-trees/ 983 gene trees
└── 08-species-tree/ gene-trees.tre, shrews.tre

Compare the two summaries:

Terminal window
head -n 7 results/01-summary-raw/raw_alignment_summary.txt results/04-summary-clean/clean_alignment_summary.txt
RawClean
Taxa102102
Loci4,040983
Sites5,398,947756,449
Missing data49.43%13.55%

These match Tutorial 1, which runs the same steps in SEGUI. To browse a summary in the terminal, run segul view summary on its alignment-report.json file. See viewing summaries.

The two trees are in Newick format, each with 102 taxa:

  • results/06-concat-tree/shrews.treefile: the maximum likelihood tree of the supermatrix.
  • results/08-species-tree/shrews.tre: the ASTRAL species tree, with support values on its branches.

Open them in a tree viewer such as FigTree or iTOL.

Snakemake records how each file was made. Change a setting, and it reruns only the steps that depend on it.

  1. In config.yaml, change min_taxa to 0.8.

  2. Check the plan:

    Terminal window
    snakemake --dry-run --cores 8 --sdm conda

    Snakemake lists filter and every step after it. It leaves out download, unpack, and summary_raw, whose results still hold.

  3. Run the workflow again:

    Terminal window
    snakemake --cores 8 --sdm conda

The stricter filter keeps 308 loci, so this run is faster.

  • Use your own data. Point dataset_url to your archive, or put your alignments in data/loci and remove the download and unpack rules. Change --input-format if they are not NEXUS.
  • Add a step. A new rule needs only its input, output, and command. Every SEGUL command works the same way. See the CLI guides.
  • Use a partitioned model. concat already writes shrews_partition.txt. Add it to the concat_tree input and pass it to IQ-TREE with -p.
  • Run on a cluster. Snakemake can submit each job to a scheduler such as Slurm. See the Snakemake documentation.
  • Share the workflow. The project folder is all someone needs to repeat the analysis. Keep it under version control, without data, results, and .snakemake.

This material is based upon work supported by the National Science Foundation under grant numbers: DEB-1754393 and DEB-2244754. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.