Skip to content

Tutorial 5: From reads to contigs with Python

This tutorial writes one Python script that takes the raw reads of a sample to assembled contigs. PySEGUL checks the data at each step, and Python runs the other programs in between. You will:

  1. Install fastp, SPAdes, and PySEGUL.
  2. Download a sequencing run and check its quality.
  3. Clean the reads with fastp and compare them with the raw reads.
  4. Assemble the clean reads with SPAdes.
  5. Summarize the contigs.
SoftwareRole in the script
PySEGULSummarizes the reads and the contigs, and loads the reports as Python dictionaries.
fastpTrims adapters and low quality ends, and drops poor reads.
SPAdesAssembles the reads into contigs.

Tutorial 4 covers PySEGUL on its own. This one adds the pattern for any other program: run it with subprocess, then check its output with PySEGUL.

  • A computer with Linux or macOS. On Windows, use WSL, since SPAdes does not run on Windows.
  • Conda. Miniforge is a good way to get it.
  • About 1 GB of free storage and 8 GB of memory.
  • Basic Python: functions, lists, and dictionaries.

With 8 cores, the script takes about 5 minutes after the download. The download is 238 MB and can take 10 minutes or more.

The data are the raw reads of Tutorial 2: run SRR6053059, a UCE target enrichment library of a short-tailed gymnure, Hylomys suillus, from Esselstyn et al. 2017. It has 1,761,846 read pairs of 125 bp. The script downloads the two FASTQ files from the European Nucleotide Archive, which keeps the quality scores from the sequencer.

Create a conda environment with Python and the two programs, then activate it:

Terminal window
conda create --name reads --channel conda-forge --channel bioconda python=3.12 fastp spades
Terminal window
conda activate reads

Install PySEGUL in the same environment:

Terminal window
pip install pysegul

Check that everything is in place:

Terminal window
fastp --version
Terminal window
spades.py --version

This tutorial was tested with PySEGUL 0.5.0, fastp 1.4.0, and SPAdes 4.0.0.

Create a folder for the project:

Terminal window
mkdir segul-reads
Terminal window
cd segul-reads

Create a file named reads_to_contigs.py and add each block below, in order.

#!/usr/bin/env python
"""Check raw reads, clean them with fastp, and assemble them with SPAdes."""
import shutil
import subprocess
import sys
import urllib.request
from pathlib import Path
import pysegul
RUN = "SRR6053059"
READ_URL = "https://ftp.sra.ebi.ac.uk/vol1/fastq/SRR605/009/SRR6053059"
DATA_DIR = Path("data")
OUTPUT_DIR = Path("results")
LOG_DIR = Path("logs")
THREADS = 8

Set THREADS to the number of cores you want fastp and SPAdes to use.

def check_programs() -> None:
"""Stop early if fastp or SPAdes is not installed."""
for program in ("fastp", "spades.py"):
if shutil.which(program) is None:
sys.exit(f"{program} not found. Activate the environment that has it.")
def run_program(command: list[str], log_name: str) -> None:
"""Run another program, with its messages saved to a log file."""
LOG_DIR.mkdir(exist_ok=True)
print(f"Running {command[0]}...")
with open(LOG_DIR / log_name, "w") as log:
subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, check=True)

These two functions are all the script needs to work with other programs:

  • check_programs looks for each program before anything runs. A missing program then stops the script at once, not after ten minutes of work.
  • run_program runs a command given as a list, one item per word. check=True stops the script if the program fails, and the program's messages go to a file in logs.
def download_reads() -> list[Path]:
"""Download both read files, unless they are already there."""
DATA_DIR.mkdir(exist_ok=True)
reads = [DATA_DIR / f"{RUN}_{pair}.fastq.gz" for pair in (1, 2)]
for read in reads:
if not read.exists():
print(f"Downloading {read.name}...")
urllib.request.urlretrieve(f"{READ_URL}/{read.name}", read)
return reads

The function skips the files that are already in data. If you did Tutorial 2 on a computer, copy the two files there.

def summarize_reads(reads: list[Path], name: str) -> dict:
"""Summarize read files in complete mode and load the report."""
summary = pysegul.ReadSummary(
input_fmt="auto",
mode="complete",
output_path=str(OUTPUT_DIR / f"reads-{name}"),
output_prefix=name,
plot=pysegul.PlotOptions(format="svg"),
)
outputs = summary.from_files([str(read) for read in reads])
return pysegul.load_read_report(outputs["report"])
def print_checks(report: dict) -> None:
"""Print the quality checks of each read file."""
for sample in report["samples"]:
print(f"{sample['sample']}: {sample['qc_status']}")
for check in sample["qc"]:
value = f"{check['value']:.1f}{check['unit']}"
print(f" {check['name']:16}{value:>8} {check['status']}")

ReadSummary in complete mode writes a JSON report, which load_read_report loads as a dictionary. Each sample in it has:

KeyContent
metricsStatistics such as read_count, q30_pct, and adapter_reads_pct
qcThe quality checks, each with a name, value, and status
qc_statusThe worst status among the checks: Pass, Warn, or Fail

These are the checks that the SEGUI read report shows. summarize_reads takes a name, so the script can use it twice: on the raw reads and on the clean reads.

def clean_reads(reads: list[Path]) -> list[Path]:
"""Trim adapters and low quality ends with fastp."""
output_dir = OUTPUT_DIR / "01-clean-reads"
output_dir.mkdir(parents=True)
cleaned = [output_dir / read.name for read in reads]
run_program(
[
"fastp",
"--in1", str(reads[0]),
"--in2", str(reads[1]),
"--out1", str(cleaned[0]),
"--out2", str(cleaned[1]),
"--detect_adapter_for_pe",
"--cut_right",
"--thread", str(THREADS),
"--json", str(output_dir / "fastp.json"),
"--html", str(output_dir / "fastp.html"),
],
"fastp.log",
)
return cleaned

The options tell fastp to find the adapters of paired-end reads (--detect_adapter_for_pe) and to cut each read where its quality drops (--cut_right). fastp also drops reads that are poor or too short. It writes its own report to fastp.html.

def assemble(reads: list[Path]) -> Path:
"""Assemble the reads into contigs with SPAdes."""
output_dir = OUTPUT_DIR / "02-assembly"
run_program(
[
"spades.py",
"-1", str(reads[0]),
"-2", str(reads[1]),
"-o", str(output_dir),
"--threads", str(THREADS),
],
"spades.log",
)
return output_dir / "contigs.fasta"
def summarize_contigs(contigs: Path) -> dict:
"""Summarize the assembly and load the report."""
summary = pysegul.ContigSummary(
input_fmt="fasta",
output_dir=str(OUTPUT_DIR / "03-contig-summary"),
output_prefix=RUN,
plot=pysegul.PlotOptions(format="svg"),
)
outputs = summary.from_files([str(contigs)])
return pysegul.load_contig_report(outputs["report"])

assemble returns the path of contigs.fasta, which goes straight to ContigSummary. The contig report has one entry per assembly in samples.

def print_read_comparison(raw: dict, clean: dict) -> None:
"""Print the main read statistics before and after cleaning."""
rows = [
("Reads", "read_count", ",.0f"),
("Mean length", "mean_read_length", ".1f"),
("Q30 bases (%)", "q30_pct", ".1f"),
("Low quality reads", "low_quality_reads", ",.0f"),
("Adapters (%)", "adapter_reads_pct", ".2f"),
]
for before, after in zip(raw["samples"], clean["samples"]):
print(f"{before['sample']:22}{'Raw':>12}{'Clean':>12}")
for label, key, fmt in rows:
old, new = before["metrics"][key], after["metrics"][key]
print(f" {label:20}{old:>12{fmt}}{new:>12{fmt}}")
def print_contig_summary(report: dict) -> None:
"""Print the main statistics of the assembly."""
stats = report["samples"][0]
print(f"Contigs: {stats['contigs']:,}")
print(f"Bases: {stats['bases']:,}")
print(f"N50: {stats['n50']:,} bp")
print(f"Longest contig: {stats['max']:,} bp")
for level in stats["long_contigs"]:
print(f"Contigs longer than {level['length']:,} bp: {level['contigs']:,}")
def main() -> None:
check_programs()
# PySEGUL and SPAdes do not overwrite files, so start from an empty folder.
shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
reads = download_reads()
raw = summarize_reads(reads, "raw")
print_checks(raw)
cleaned = clean_reads(reads)
clean = summarize_reads(cleaned, "clean")
print_read_comparison(raw, clean)
contigs = assemble(cleaned)
report = summarize_contigs(contigs)
print_contig_summary(report)
if __name__ == "__main__":
main()

main reads like the plan of the analysis: check, clean, check again, assemble, summarize.

Terminal window
python reads_to_contigs.py

PySEGUL prints the log of each step. Among it, the script prints three results.

SRR6053059_1.fastq: Fail
Q30 bases 85.7% Pass
Mean quality 33.3 Pass
N content 0.0% Pass
Duplication 76.0% Fail
Adapters 0.4% Pass
Poly-G 0.1% Pass
Overrepresented 0.2% Warn
SRR6053059_2.fastq: Fail
Q30 bases 83.2% Pass
Mean quality 32.6 Pass
N content 0.1% Pass
Duplication 73.7% Fail
Adapters 0.4% Pass
Poly-G 0.1% Pass
Overrepresented 0.2% Warn

These match the report in Tutorial 2, which explains each check. The duplication is expected for a target enrichment library.

SRR6053059_1.fastq Raw Clean
Reads 1,761,846 1,560,875
Mean length 125.0 114.5
Q30 bases (%) 85.7 94.7
Low quality reads 82,824 0
Adapters (%) 0.43 0.00
SRR6053059_2.fastq Raw Clean
Reads 1,761,846 1,560,875
Mean length 125.0 112.4
Q30 bases (%) 83.2 94.4
Low quality reads 118,992 0
Adapters (%) 0.43 0.00

fastp kept 89% of the read pairs. The clean reads are shorter, because their poor ends are gone, and more than 94% of their bases reach Q30. No read has a mean quality below 20, and the adapters are gone. Both files still have the same number of reads, which the assembler needs.

Contigs: 10,653
Bases: 2,610,251
N50: 283 bp
Longest contig: 3,334 bp
Contigs longer than 750 bp: 40
Contigs longer than 1,000 bp: 12
Contigs longer than 1,500 bp: 2

Most contigs are short, which is common for a target enrichment library: reads pile up on the targeted loci and thin out away from them. The next step of a UCE study matches the contigs to the probes and keeps the ones that hit.

The results folder holds:

results/
├── reads-raw/ summary table, report, and plots/
├── 01-clean-reads/ two FASTQ files, fastp.html, fastp.json
├── reads-clean/ summary table, report, and plots/
├── 02-assembly/ contigs.fasta and other SPAdes files
└── 03-contig-summary/ summary table, report, and plots/

To browse a read report, open its read-report.json file in SEGUI or with segul view summary.

  • Process many samples. Put the steps of main in a loop over your samples. ReadSummary and ContigSummary also take a whole folder with from_dir, and their reports then have one entry per file.
  • Act on the checks. The report is a dictionary, so the script can decide: skip the assembly of a sample whose clean reads fail a check, or list the samples that need a second look.
  • Change the programs. run_program works with any command. Swap SPAdes for another assembler, or add a step after the assembly.
  • Scale up. For many samples and steps, a workflow manager reruns only what changed and runs jobs in parallel. See Tutorial 3.

This material is based upon work supported by the National Science Foundation under grant numbers: DEB-1754393 and DEB-2244754. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.