This tutorial writes one Python script that takes the raw reads of a sample to assembled contigs. PySEGUL checks the data at each step, and Python runs the other programs in between. You will:
- Install fastp, SPAdes, and PySEGUL.
- Download a sequencing run and check its quality.
- Clean the reads with fastp and compare them with the raw reads.
- Assemble the clean reads with SPAdes.
- Summarize the contigs.
The software
Section titled “The software”| Software | Role in the script |
|---|---|
| PySEGUL | Summarizes the reads and the contigs, and loads the reports as Python dictionaries. |
| fastp | Trims adapters and low quality ends, and drops poor reads. |
| SPAdes | Assembles the reads into contigs. |
Tutorial 4 covers PySEGUL on its own. This one adds the pattern for any other program: run it with subprocess, then check its output with PySEGUL.
Before you start
Section titled “Before you start”- A computer with Linux or macOS. On Windows, use WSL, since SPAdes does not run on Windows.
- Conda. Miniforge is a good way to get it.
- About 1 GB of free storage and 8 GB of memory.
- Basic Python: functions, lists, and dictionaries.
With 8 cores, the script takes about 5 minutes after the download. The download is 238 MB and can take 10 minutes or more.
The dataset
Section titled “The dataset”The data are the raw reads of Tutorial 2: run SRR6053059, a UCE target enrichment library of a short-tailed gymnure, Hylomys suillus, from Esselstyn et al. 2017. It has 1,761,846 read pairs of 125 bp. The script downloads the two FASTQ files from the European Nucleotide Archive, which keeps the quality scores from the sequencer.
1. Set up the project
Section titled “1. Set up the project”Create a conda environment with Python and the two programs, then activate it:
conda create --name reads --channel conda-forge --channel bioconda python=3.12 fastp spadesconda activate readsInstall PySEGUL in the same environment:
pip install pysegulCheck that everything is in place:
fastp --versionspades.py --versionThis tutorial was tested with PySEGUL 0.5.0, fastp 1.4.0, and SPAdes 4.0.0.
Create a folder for the project:
mkdir segul-readscd segul-reads2. Start the script
Section titled “2. Start the script”Create a file named reads_to_contigs.py and add each block below, in order.
#!/usr/bin/env python"""Check raw reads, clean them with fastp, and assemble them with SPAdes."""import shutilimport subprocessimport sysimport urllib.requestfrom pathlib import Path
import pysegul
RUN = "SRR6053059"READ_URL = "https://ftp.sra.ebi.ac.uk/vol1/fastq/SRR605/009/SRR6053059"DATA_DIR = Path("data")OUTPUT_DIR = Path("results")LOG_DIR = Path("logs")THREADS = 8Set THREADS to the number of cores you want fastp and SPAdes to use.
3. Run other programs from Python
Section titled “3. Run other programs from Python”def check_programs() -> None: """Stop early if fastp or SPAdes is not installed.""" for program in ("fastp", "spades.py"): if shutil.which(program) is None: sys.exit(f"{program} not found. Activate the environment that has it.")
def run_program(command: list[str], log_name: str) -> None: """Run another program, with its messages saved to a log file.""" LOG_DIR.mkdir(exist_ok=True) print(f"Running {command[0]}...") with open(LOG_DIR / log_name, "w") as log: subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, check=True)These two functions are all the script needs to work with other programs:
check_programslooks for each program before anything runs. A missing program then stops the script at once, not after ten minutes of work.run_programruns a command given as a list, one item per word.check=Truestops the script if the program fails, and the program's messages go to a file inlogs.
4. Download the reads
Section titled “4. Download the reads”def download_reads() -> list[Path]: """Download both read files, unless they are already there.""" DATA_DIR.mkdir(exist_ok=True) reads = [DATA_DIR / f"{RUN}_{pair}.fastq.gz" for pair in (1, 2)] for read in reads: if not read.exists(): print(f"Downloading {read.name}...") urllib.request.urlretrieve(f"{READ_URL}/{read.name}", read) return readsThe function skips the files that are already in data. If you did Tutorial 2 on a computer, copy the two files there.
5. Check the reads
Section titled “5. Check the reads”def summarize_reads(reads: list[Path], name: str) -> dict: """Summarize read files in complete mode and load the report.""" summary = pysegul.ReadSummary( input_fmt="auto", mode="complete", output_path=str(OUTPUT_DIR / f"reads-{name}"), output_prefix=name, plot=pysegul.PlotOptions(format="svg"), ) outputs = summary.from_files([str(read) for read in reads]) return pysegul.load_read_report(outputs["report"])
def print_checks(report: dict) -> None: """Print the quality checks of each read file.""" for sample in report["samples"]: print(f"{sample['sample']}: {sample['qc_status']}") for check in sample["qc"]: value = f"{check['value']:.1f}{check['unit']}" print(f" {check['name']:16}{value:>8} {check['status']}")ReadSummary in complete mode writes a JSON report, which load_read_report loads as a dictionary. Each sample in it has:
| Key | Content |
|---|---|
metrics | Statistics such as read_count, q30_pct, and adapter_reads_pct |
qc | The quality checks, each with a name, value, and status |
qc_status | The worst status among the checks: Pass, Warn, or Fail |
These are the checks that the SEGUI read report shows. summarize_reads takes a name, so the script can use it twice: on the raw reads and on the clean reads.
6. Clean the reads with fastp
Section titled “6. Clean the reads with fastp”def clean_reads(reads: list[Path]) -> list[Path]: """Trim adapters and low quality ends with fastp.""" output_dir = OUTPUT_DIR / "01-clean-reads" output_dir.mkdir(parents=True) cleaned = [output_dir / read.name for read in reads] run_program( [ "fastp", "--in1", str(reads[0]), "--in2", str(reads[1]), "--out1", str(cleaned[0]), "--out2", str(cleaned[1]), "--detect_adapter_for_pe", "--cut_right", "--thread", str(THREADS), "--json", str(output_dir / "fastp.json"), "--html", str(output_dir / "fastp.html"), ], "fastp.log", ) return cleanedThe options tell fastp to find the adapters of paired-end reads (--detect_adapter_for_pe) and to cut each read where its quality drops (--cut_right). fastp also drops reads that are poor or too short. It writes its own report to fastp.html.
7. Assemble and summarize the contigs
Section titled “7. Assemble and summarize the contigs”def assemble(reads: list[Path]) -> Path: """Assemble the reads into contigs with SPAdes.""" output_dir = OUTPUT_DIR / "02-assembly" run_program( [ "spades.py", "-1", str(reads[0]), "-2", str(reads[1]), "-o", str(output_dir), "--threads", str(THREADS), ], "spades.log", ) return output_dir / "contigs.fasta"
def summarize_contigs(contigs: Path) -> dict: """Summarize the assembly and load the report.""" summary = pysegul.ContigSummary( input_fmt="fasta", output_dir=str(OUTPUT_DIR / "03-contig-summary"), output_prefix=RUN, plot=pysegul.PlotOptions(format="svg"), ) outputs = summary.from_files([str(contigs)]) return pysegul.load_contig_report(outputs["report"])assemble returns the path of contigs.fasta, which goes straight to ContigSummary. The contig report has one entry per assembly in samples.
8. Put the steps together
Section titled “8. Put the steps together”def print_read_comparison(raw: dict, clean: dict) -> None: """Print the main read statistics before and after cleaning.""" rows = [ ("Reads", "read_count", ",.0f"), ("Mean length", "mean_read_length", ".1f"), ("Q30 bases (%)", "q30_pct", ".1f"), ("Low quality reads", "low_quality_reads", ",.0f"), ("Adapters (%)", "adapter_reads_pct", ".2f"), ] for before, after in zip(raw["samples"], clean["samples"]): print(f"{before['sample']:22}{'Raw':>12}{'Clean':>12}") for label, key, fmt in rows: old, new = before["metrics"][key], after["metrics"][key] print(f" {label:20}{old:>12{fmt}}{new:>12{fmt}}")
def print_contig_summary(report: dict) -> None: """Print the main statistics of the assembly.""" stats = report["samples"][0] print(f"Contigs: {stats['contigs']:,}") print(f"Bases: {stats['bases']:,}") print(f"N50: {stats['n50']:,} bp") print(f"Longest contig: {stats['max']:,} bp") for level in stats["long_contigs"]: print(f"Contigs longer than {level['length']:,} bp: {level['contigs']:,}")
def main() -> None: check_programs() # PySEGUL and SPAdes do not overwrite files, so start from an empty folder. shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
reads = download_reads() raw = summarize_reads(reads, "raw") print_checks(raw)
cleaned = clean_reads(reads) clean = summarize_reads(cleaned, "clean") print_read_comparison(raw, clean)
contigs = assemble(cleaned) report = summarize_contigs(contigs) print_contig_summary(report)
if __name__ == "__main__": main()main reads like the plan of the analysis: check, clean, check again, assemble, summarize.
9. Run the script
Section titled “9. Run the script”python reads_to_contigs.pyPySEGUL prints the log of each step. Among it, the script prints three results.
The quality checks of the raw reads
Section titled “The quality checks of the raw reads”SRR6053059_1.fastq: Fail Q30 bases 85.7% Pass Mean quality 33.3 Pass N content 0.0% Pass Duplication 76.0% Fail Adapters 0.4% Pass Poly-G 0.1% Pass Overrepresented 0.2% WarnSRR6053059_2.fastq: Fail Q30 bases 83.2% Pass Mean quality 32.6 Pass N content 0.1% Pass Duplication 73.7% Fail Adapters 0.4% Pass Poly-G 0.1% Pass Overrepresented 0.2% WarnThese match the report in Tutorial 2, which explains each check. The duplication is expected for a target enrichment library.
The reads before and after fastp
Section titled “The reads before and after fastp”SRR6053059_1.fastq Raw Clean Reads 1,761,846 1,560,875 Mean length 125.0 114.5 Q30 bases (%) 85.7 94.7 Low quality reads 82,824 0 Adapters (%) 0.43 0.00SRR6053059_2.fastq Raw Clean Reads 1,761,846 1,560,875 Mean length 125.0 112.4 Q30 bases (%) 83.2 94.4 Low quality reads 118,992 0 Adapters (%) 0.43 0.00fastp kept 89% of the read pairs. The clean reads are shorter, because their poor ends are gone, and more than 94% of their bases reach Q30. No read has a mean quality below 20, and the adapters are gone. Both files still have the same number of reads, which the assembler needs.
The assembly
Section titled “The assembly”Contigs: 10,653Bases: 2,610,251N50: 283 bpLongest contig: 3,334 bpContigs longer than 750 bp: 40Contigs longer than 1,000 bp: 12Contigs longer than 1,500 bp: 2Most contigs are short, which is common for a target enrichment library: reads pile up on the targeted loci and thin out away from them. The next step of a UCE study matches the contigs to the probes and keeps the ones that hit.
The results folder holds:
results/├── reads-raw/ summary table, report, and plots/├── 01-clean-reads/ two FASTQ files, fastp.html, fastp.json├── reads-clean/ summary table, report, and plots/├── 02-assembly/ contigs.fasta and other SPAdes files└── 03-contig-summary/ summary table, report, and plots/To browse a read report, open its read-report.json file in SEGUI or with segul view summary.
Where to go next
Section titled “Where to go next”- Process many samples. Put the steps of
mainin a loop over your samples.ReadSummaryandContigSummaryalso take a whole folder withfrom_dir, and their reports then have one entry per file. - Act on the checks. The report is a dictionary, so the script can decide: skip the assembly of a sample whose clean reads fail a check, or list the samples that need a second look.
- Change the programs.
run_programworks with any command. Swap SPAdes for another assembler, or add a step after the assembly. - Scale up. For many samples and steps, a workflow manager reruns only what changed and runs jobs in parallel. See Tutorial 3.