AFI Terra Pipeline

1. Overview

The afi_terra pipeline detects and identifies Rickettsiales organisms — primarily Orientia, Rickettsia, and related genera — from 16S rRNA metagenomic sequencing data. It runs on Terra, the Broad Institute's cloud execution platform, using WDL 1.0 workflows backed by Docker images.

Target organisms

The pipeline is designed to detect genera in the order Rickettsiales. For Orientia and Rickettsia specifically, a confirmatory 16S rRNA alignment step provides higher specificity. All other genera detected by the classifier (e.g., Leptospira, Burkholderia, Anaplasma) are reported from Centrifuger read counts alone.

Two workflows

For clinical use, run the batch workflow. It processes all samples in a run together, automatically computing a per-run NTC background, and producing a consolidated run_summary.tsv. The Terra Sheet Builder desktop app (Section 8) generates the required sample sheet TSV without manual editing.

Two sample modes

A single batch run can contain both modes simultaneously.

2. Architecture and Data Flow

Processing steps

Each sample passes through up to nine steps. Steps 1–5 run in Phase 1; steps 6–8 run in Phase 2 after per-run NTC backgrounds are computed; step 9 is the final batch-level gather.

Step 1  Human read dehosting        NCBI SRA Scrubber (optional, default on)
Step 2  QC / adapter trimming       fastp v0.23.4
Step 3  Taxonomic classification    Centrifuger -> kreport -> genus counts TSV
Step 4  16S alignment               minimap2 -> BAM
Step 5  Alignment metrics           extract_rick16s_metrics.py -> align_metrics.tsv
        ---- batch gather: BuildNTCBackground (NTC/NC samples only; per run_id) ----
        ---- batch gather: MatchNTCBackground (assigns each sample its run's NTC) ----
Step 6  NTC-aware interpretation    call_taxa.py -> calls.tsv + taxa_evidence.tsv
Step 7a Validation summary          compare_expected (validation mode only)
Step 7b Routine summary             summarize_routine (routine mode only)
Step 8  Run summary                 run_summary.tsv (batch only)

Batch workflow phases

Phase 1 scatter — runs steps 1–5 for all samples in parallel.

BuildNTCBackground — collects all NTC/NC metrics and computes one ntc_background_<run_id>.tsv per distinct run_id.

MatchNTCBackground — maps each sample to the NTC background for its run_id. This task fails if any run_id has no NTC/NC sample.

Phase 2 scatter — runs steps 6–7 for all samples using matched NTC backgrounds.

BuildRunSummary — merges all per-sample summaries into a single run_summary.tsv.

3. Reference Files and Databases

16S Rickettsiales panel

Centrifuger database

The database (~67 GiB compressed) is stored in GCS as a TAR.GZ archive. Provide two inputs:

4. Importing Workflows into Terra

Option 1: Dockstore (recommended)

1. In Dockstore, link the GitHub repository PHemarajata/afi_terra.
2. Create or refresh a workflow version from the desired Git tag or branch.
3. In Terra, navigate to Workflows and click Find a Workflow.
4. Select Dockstore as the source and import via TRS. Terra resolves all imports automatically.

Option 2: Direct ZIP upload

If Dockstore is not available, create a ZIP preserving relative directory paths:

zip -r afi_terra_wdl.zip \
  wdl/AFI_16S_Batch.wdl \
  wdl/AFI_16S_Main.wdl \
  wdl/tasks/ \
  NCBI_scrub_PE/tasks/quality_control/read_filtering/task_ncbi_scrub.wdl

5. Running the Single-Sample Workflow

Required inputs

Mode and type inputs

Launching in Terra

1. Navigate to Workflows and select AFI_16S_Main.
2. Choose Run workflow with inputs defined by file paths.
3. Upload or paste your input JSON, then click Run Analysis.

6. Running the Batch Workflow

The batch workflow (AFI_16S_Batch) processes one or more sequencing runs in a single Terra submission. It scatters each sample through processing, automatically builds a per-run NTC background, and produces a consolidated run summary.

Per-sample arrays

Sample types in a batch run

7. The Two-Pass Pattern

The batch workflow handles NTC backgrounds entirely automatically — no manual intervention is needed. The two-pass procedure below applies only to the single-sample workflow or exceptional reprocessing scenarios.

Step 1: First pass — placeholder NTC background

Supply wdl/inputs/ntc_background.placeholder.tsv (all-zero thresholds) as ntc_background. Collect the align_metrics output for each NTC/NC sample from Terra.

Step 2: Build run-specific NTC background

python3 scripts/build_ntc_background_from_metrics.py \
  --ntc-metrics path/to/NTC_S11.align_metrics.tsv \
  --ntc-metrics path/to/NTC_S12.align_metrics.tsv \
  --out wdl/inputs/run3.ntc_background.tsv

Step 3: Upload and re-run

gsutil cp wdl/inputs/run3.ntc_background.tsv gs://YOUR_BUCKET/ref/run3_ntc_background.tsv

Re-submit the sample with ntc_background pointing to the real file. The second pass produces NTC-corrected calls.

8. Terra Sheet Builder (GUI)

Getting the application

Pre-built binaries are built automatically by GitHub Actions. Download the artifact for your platform from Actions → Build Terra Sheet Builder on the repository page:

Run from source (requires Python 3.10+):

pip install PySide6
python3 tools/terra_sheet_builder/terra_sheet_builder.py

Step-by-Step Walkthrough

The following walkthrough covers generating a Terra metadata sheet for the AFI_16S_Batch workflow for a two-run batch.

1

Specify the number of runs in this batch

On the first screen, enter how many sequencing runs you want to process together. In this example, there are two runs to process. Use the spinner to set the count.

2

Click Continue

After setting the run count, click Continue to advance to the run-naming step.

3

Name each run

In Step 2, enter a unique name for each run. Names are used as the run_id value for all samples in that run. No spaces are allowed — use underscores or hyphens (e.g., run_2024_11). In this example, runs are named Run 1 and Run 2. Click Next when done.

4

Select FASTQ files for each run

In this step, add FASTQ files to each run. You can import from pre-generated TSV files or browse to the folder where R1 and R2 FASTQ files are located. Click Browse Folder to select a directory — the app auto-discovers all R1/R2 pairs using the _R1_/_R2_ naming convention.

Three methods to add samples:

• Browse folder — auto-discovers all R1/R2 FASTQ pairs in a directory
• Add files… — multi-file dialog; app auto-pairs R1+R2 by name pattern
• Import from TSV… — global import with run_id and sample_id columns

5

Browse to the FASTQ folder and click Open

Navigate to the folder containing the paired read files. Select the folder and click Open. The app will scan for all valid R1/R2 FASTQ pairs automatically.

6

Files are sorted into the metadata table automatically

The app sorts discovered files into the metadata table based on the specified runs. Repeat the folder selection for the remaining runs to be processed.

7

Verify sample names and file names, then click Next

Tables with sample names and file names are populated automatically. Review the table to confirm all entries are correct. Once ready, click Next to proceed to sample metadata entry.

8

Review the sample metadata page

This page requires additional information. Note that the Analysis Date field has already been pre-filled automatically with today's date.

9

Enter the Terra table name and operator initials

Fill in the Terra metadata table name (lowercase alphanumeric, ≤32 chars, must start with a letter) and the operator's initials. These are recorded in the analysis_comments column of the output TSV for traceability.

10

Designate the NTC (negative control) for each run

Below the header fields, the sample table lists all samples for both runs. Each run must have at least one NTC and one positive control. In the sample_type column, find the NTC sample for Run 1 and select NTC from the dropdown. The sample_type for clinical samples remains clinical.

11

Confirm the NTC designation

The NTC sample for Run 1 is now designated. The sample_type dropdown shows NTC for that row. Continue to designate the positive control for Run 1.

12

Begin changing the positive control sample type

For the positive control sample in Run 1, click the sample_type dropdown to change it from clinical to the appropriate control type.

13

Open the sample type dropdown

Click the dropdown control for the positive control row to open the list of available sample types.

14

Select the positive control type

Change the sample_type to the type of control used. In this example, PC_MIX8 is selected — a positive control containing DNA from 8 bacterial species.

sample_type	mode (auto-filled)	expected_taxa (auto-filled)
PC_MIX8	validation	Bacillus;Listeria;Staphylococcus;Enterococcus;Limosilactobacillus;Salmonella;Escherichia;Pseudomonas
clinical	routine	(empty)
NTC / NC	routine	(empty)

15

Expected taxa auto-populate for PC_MIX8

Once PC_MIX8 is selected, the expected_taxa field auto-fills with the 8 bacterial species expected from that control. This tells the validation workflow which taxa should be identified.

16

Validation catches missing controls before export

If an operator forgets to designate the NTC and positive control for Run 2 and clicks Validate, the Export TSV button remains greyed out. The app has detected missing required controls.

17

Review and fix the validation error

The app generates a descriptive error message stating that Run 2 has no NTC and no positive control specified. Click OK to acknowledge. Correct the missing designations for Run 2 before proceeding.

Validation checks performed:

• No duplicate sample_id values across all runs
• Each run has at least one NTC or NC sample
• Each run has at least one positive control (PC_MIX8, PC_SINGLE, MIXED4, or PC)
• All validation-mode rows have a non-empty expected_taxa field
• Table name matches ^[a-z][a-z0-9_]{0,31}$

18

Validate — all checks must pass before export

After correcting all issues, click Validate again. If all checks pass, the Export TSV button becomes clickable. The TSV files are now ready to be uploaded to Terra.bio.

19

Export TSV and save the file

Click Export TSV and save the file to a location you can easily locate. The file will be named using the table name you entered in the header fields.

20

Verify the exported file in Microsoft Excel

Open the exported TSV in Microsoft Excel to confirm the format is correct and compatible with Terra.bio. Check that the entity:<table_name>_id column is the first column, all sample fields are present, and the analysis_comments column appears as the last column.

Importing the TSV into Terra

1. In your Terra workspace, go to Data → Import Data → Upload TSV.
2. Select the exported TSV file.
3. Terra will create or update a sample set entity table with the name you provided.

The exported file uses entity:<table_name>_id as the first column (required by Terra) followed by all sample fields. An analysis_comments column appended as the last column contains <table_name> | <date> | <initials> for traceability.

9. Helper Scripts

All helper scripts are in the scripts/ directory. They run locally (not in Terra) and are used to prepare inputs, post-process outputs, and manage Docker images.

build_batch_inputs_json.py

Converts a sample sheet TSV or AFI mapping TSV into a Terra-ready batch input JSON.

python3 scripts/build_batch_inputs_json.py \
  --sample-sheet wdl/inputs/batch_samples.template.tsv \
  --out-json wdl/inputs/batch_run1.json \
  --rickettsiales-panel gs://YOUR_BUCKET/ref/rickettsiales_panel_16S.clean.fa \
  --centrifuger-db centrifuger_bact_arch_plus_rickettsiales

build_ntc_background_from_metrics.py

Builds a run-specific NTC background TSV from one or more NTC align_metrics.tsv output files.

python3 scripts/build_ntc_background_from_metrics.py \
  --ntc-metrics path/to/NTC1.align_metrics.tsv \
  --ntc-metrics path/to/NTC2.align_metrics.tsv \
  --out wdl/inputs/run_ntc_background.tsv

make_terra_import_sheet.py

CLI companion to the Terra Sheet Builder GUI. Generates or validates a Terra-compatible sample import TSV and optionally writes an annotated Excel workbook.

python3 scripts/make_terra_import_sheet.py --template --output my_run.tsv
python3 scripts/make_terra_import_sheet.py --input samples.csv --output terra_import.tsv --excel

10. Output Files Reference

Per-sample outputs

Batch-only outputs

11. Interpreting Calls and Summaries

Call values — alignment source (Orientia and Rickettsia)

Call values — Centrifuger source (all other genera)

PC8 validity

For PC_MIX8 samples, a pass/fail flag is computed. At least 6 of the 8 expected taxa must be detected for a passing result. In run_summary.tsv, run_pc8_valid carries this value for every row, allowing immediate run-level QC assessment.

12. Detection Thresholds

Configurable thresholds (Tier 1 "Confirmed" and Centrifuger "Detected")

Hardcoded thresholds (Tier 2 "Probable" and equivocal calls)

The Tier 2 order-level Rickettsiales rescue and the equivocal-call thresholds are defined in scripts/call_taxa.py. These are not currently exposed as workflow inputs but are documented here for transparency.

13. V4 Decontamination Filter (post-pipeline)

The filter is implemented as a standalone Python script (afi_decontamination_filter_v4.py) that ingests .calls.tsv rows with positive calls (Detected / Confirmed / Probable), applies the tiered rules below, and emits a report of which detections were kept and which were removed.

Tier A — High-confidence kit / skin / water contaminants

The following 11 genera are removed on detection in clinical and study samples. Membership is documented in five or more independent low-biomass contamination reviews and confirmed in this dataset's NTC profiles.

Tier A exception — Burkholderia species-level safeguard

The genus Burkholderia is a Tier-A-equivalent kit contaminant (the genus is dominated in low-biomass samples by B. cepacia complex species), but it also contains B. pseudomallei, the etiologic agent of melioidosis — a "cannot-miss" diagnosis endemic in Thailand. For every sample with a Burkholderia genus detection, the filter:

1. Opens the Centrifuger kreport (<sample>.centrifuger.kreport.tsv) and parses rows at the species rank (S).
2. Sums reads assigned to Burkholderia pseudomallei (NCBI taxonomy ID 28450) at species rank.
3. Retrieves the run's NTC species-level B. pseudomallei read count for comparison.
4. Preserves the detection only if both: (i) species reads ≥ 500, AND (ii) species reads > the run-NTC species max.
5. When preserved, the retained record stores the species-level read count, not the genus total.

Tier B — NTC-only organisms

Nine genera observed only in NTC samples across all sequencing runs are removed globally: Cereibacter, Thioclava, Bdellovibrio, Saltatorellus, Pseudogemmobacter, Minisyncoccus, Rhodoluna, Microbacterium, Arcanobacterium.

Tier 1 — Ultra-low-abundance environmental noise

Sixteen genera with median per-sample abundance < 0.5% across the dataset are removed: Shigella, Metapseudomonas, Stutzerimonas, Capsulimonas, Chamaesiphon, Chloroflexus, Flavihumibacter, Hymenobacter, Limnoglobus, Methylovirgula, Microvirga, Pelagovum, Pseudonocardia, Rufibacter, Salmonella, Spirosoma.

Tier 2 — Marginal organisms (conditional removal)

Twenty-seven genera with median 0.5–2.0% abundance are retained only if detected in ≥ 2 samples AND each detection is ≥ 1.0% abundance. Below those thresholds, they are removed as likely contaminants.

Positive-control spike-in bypass

For samples designated as positive controls, Tier A removal is bypassed for any organism that is a documented spike-in for that sample. Without this bypass, the P-aeru_S5_L001 PC would fail because Pseudomonas is in Tier A — but Pseudomonas aeruginosa is precisely the organism the PC is designed to detect. The bypass restores PC interpretability without weakening Tier A removal in clinical samples.

Under the bypass, PC concordance is defined as all expected spike-in organisms must be detected AND retained by the filter.

Running the filter

python3 afi_decontamination_filter_v4.py

The script reads .calls.tsv and Centrifuger kreport files from a configured run directory and writes DECONTAMINATION-FILTER-REPORT-V4.txt with a per-sample / per-detection breakdown of filter actions. A companion appendix generator (generate_appendices.py) produces Markdown and Excel tables with biomass distribution, NTC cross-checks, and confidence labels.

14. Analytical Validation Summary

Validation panel composition

Sample-level analytical performance

The 72.1% sample-level analytical performance is the appropriate headline figure for regulatory documentation and matches the legacy APHL bioinformatic validation report for the same panel.

AFI study cohort (n=86)

• 71 / 86 samples (82.6%) carried ≥ 1 positive call (Centrifuger Detected, Tier 1 Confirmed, or Tier 2 Probable).
• 15 / 86 samples (17.4%) had zero detected taxa — consistent with pre-sequencing failure (DNA extraction, library prep, or sequencing depth).
• V4 removed 70 / 217 detections (32.3%): predominantly Cutibacterium, Staphylococcus, Brevundimonas, Acinetobacter, Corynebacterium.
• 147 detections retained across the cohort.

Key cohort findings

Additional candidate detections (require orthogonal confirmation): Leptospira in one sample at 22.78% (with a same-run NTC contamination caveat — the NTC carried 78,691 Leptospira reads, ~10× the study sample); Brucella in three samples at near-noise abundance (mean 0.98%, max 1.96%); Streptococcus in five samples (V1–V3 cannot resolve species).

Companion documents (manuscript-ready package)

Methodological revision log

The following corrections were identified during a critical review and are reflected throughout this documentation and the manuscript drafts:

15. Docker Images

Building and pushing the AFI core image

bash scripts/build_push_afi_core_image.sh phemarajata614 0.4.1 linux/amd64

16. Troubleshooting