目录
- coreprofilerMerge branch 'cgmlstorg' into 'main'5个月前
- docsupdate and improve project documentation6个月前
- tasksreplace flake8 by ruff (Lint)2年前
- testsfix cgmlstorg schema update5个月前
- tmp_blast_dbadd blast index and get scheme tool1年前
- .gitignoreReset to main2年前
- .gitlab-ci.ymlFix test: add environment.yml just to have blast because i can't have it in pyproject.toml11个月前
- LICENSEUpdate 2 files3个月前
- README.mdUpdate 2 files3个月前
- environment.ymlFix test: add environment.yml just to have blast because i can't have it in pyproject.toml11个月前
- poetry.lockadd authentication support for BIGSdb PubMLST8个月前
- pyproject.tomlNew release5个月前
邀请码
CoreProfiler: a robust and integrable cgMLST software
Introduction
Core genome Multilocus Sequence Typing (cgMLST) is a genotyping method used in microbiology to compare the genetic distances of bacterial strains. It achieves this by comparing the alleles of the core genome in genomes to a reference database of official alleles. The output is a cgMLST profile. CoreProfiler is a software that calls official alleles (“autotag”) and detects new ones (“scannew”).
Installation
Requirement
Conda installation
CoreProfiler is available on conda via bioconda.
Installation from sources
Clone the GitLab repository
Move inside the created folder
Install dependencies
Usage
Check the installation
Basic usage
The identification of alleles on the genome, often referred to as ‘allele calling,’ requires an allele database (or ‘scheme’).
Scheme downloading
A scheme contains the list of official alleles curated by its reference platform (such as BIGSdb for Klebsiella pneumoniae or Listeria, Enterobase for E. coli, PubMLST for Staphylococcus aureus, …) and cgMLST.org for several other schemes. Each locus has its FASTA file containing the list of alleles, with sequence id’s
>[locus]_[allele-number].Token Management (OAuth1)
BIGSdb and PubMLST platforms require OAuth1 authentication to access and download the most up-to-date schemes.
While authentication is not strictly mandatory, skipping it may result in downloading outdated schemes.
This workflow involves two types of tokens:
1. Getting request tokens
The first step is to create an account on pubMLST or BigsDB Pasteur and request a consumer key and consumer secret
(see the respective documentation for instructions on how to obtain them).
The second step is to request an access key and access secret using your consumer key and consumer secret provided by the platform for a specific schema.
-k / -ks → OAuth1 consumer key and secret. -a / -as → OAuth1 access token and secret.
Authentication (used for PubMLST and BIGSdb) is optional but recommended to ensure the latest version of the schema is downloaded
The list of available scheme can be obtained with the following command.
You will have a list with the
nameof the scheme, a shortdescription, the number of loci in the scheme (locus_count) and the name of theplatformfrom which the scheme originatesScheme updating
To update a scheme, run
-k / -ks → OAuth1 consumer key and secret. -a / -as → OAuth1 access token and secret.
Authentication (used for PubMLST and BIGSdb) is optional but recommended to ensure the latest version of the schema is downloaded
This code reads the last local update of the species scheme from
data/scheme_url_new.json, and replaces all locus files if the last remote locus update is newer than the last local database update.Available species schemes and their API requests URL are stored in
data/scheme_url_new.json.Allele identification
The main command
coreprofiler allele_callingconsist of 2 steps:Autotag (exact matches on reference database)
It runs
BLASTnwith 100% identity, no gaps, and parameters optimized for speed (-task megablastand-dust no; genome vs. BLAST db). The output is parsed, only results with 100% coverage are kept.Scannew (detection of new alleles)
It selects the loci for which no exact match was found in step 1. Creates a BLAST database with those loci files, and runs a looser
BLASTn(-task blastn, genome vs. BLAST db). The best hit is kept for each locus and classified as either anew allele, anincomplete alleleor amissing alleleaccording to the matching default values:[locus]_[sequence_hash]X-CoreProfiler takes as input:
Running allele calling
You can test the tool by running:
Doing so will run
makeblastdbon the scheme and create a BLAST database on which to run the genome against. This process may take a bit longer and does not need to be re-run unless the scheme has been updated. The path to a BLAST database can also be provided with the-dbargument.Example: (once the command above has been run and a BLAST database has been created already)
Detection of new alleles
Core-genome MLST brings out at almost every genome alleles that do not belong to the reference scheme. The aim of CoreProfiler is not to describe new “official” alleles, as that may enter in conflict with the platforms providing official schemes. It is however able to describe locally (temporary) new alleles. A genome is considered to carry a new allele if, for a given locus, the following conditions are met:
no exact match (100% identity & 100% coverage) with the locus alleles
one allele of this locus matched on the genome with:
The new allele sequences is the query subject that matched best with one allele (position trimming).
Parameters
general
-q, --query: [required] Path(s) to the genome assemblie(s) (fasta files).-sf, --scheme_dir: [required] Path to the directory containing the scheme files.-out: [required] Path to the output file (tsv file).-db, --blast_db_path: Path to the BLAST database.-n, --num_threads: Number of threads to run BLAST functions on. Default: 4.-V, --version: CoreProfiler version.-v, --verbose: Verbose mode.autotag
--autotag_word_size: Word size for the autotag BLASTn. Default: 31.scannew
--outfa: Path to output fasta file with new alleles sequences if detected.--min_id_new_allele: Minimum identity percentage to consider new alleles. Default: 90.--min_cov_new_allele: Minimum coverage percentage to consider new alleles. Default: 90.--min_cov_incomplete: Minimum coverage percentage to consider an allele incomplete (if--detailedoption). Default 70.-d, --detailed: Return detailed information on incomplete alleles: “incomplete, …”. Else, only new alleles or missing.--profiles_w_tmp_alleles: A JSON file containing info about files with temporary alleles.--num_alleles_per_locus: TSV file containing the number of alleles per locus of a given scheme.-cds: Extract new alleles within CDS.Platform integration
CoreProfiler has 2 modes:
allele_calling(see above) anddb(to download & updates databases).Scheme databases can be downloaded with
coreprofiler db downloadand updated withcoreprofiler db update [species]. When a database is updated, update stored profiles withcoreprofiler platform profiles_update [args], and re-runcoreprofiler db get_num_alleles [args]. To limit redondant processes between two database updates, consider runningcoreprofiler db makeblastdb [args]andcoreprofiler db get_num_alleles [args], and providing their output file when needed.Handling temporary/new alleles
When a new allele is detected, the sequence is extracted and the allele identifier is a MD5-hash of the sequence. This way, one new allele described in parallelized runs is given the same id in output profiles (and from one batch run to another). On a platform that aims to store query genomes and update cgMLST profiles, the
--profiles_w_tmp_allelesparameter should be provided to write the profiles containing temporary new alleles in a json file.The nucleic sequences of alleles described in the json file generated by the
--profiles_w_tmp_allelesparameter can be obtained with the--outfaparameter.For example, the following command:
Will generate the following output files:
test_kp.tsv:Output file
test_kp_tmp_alleles.json:Output file
test_kp_tmp_alleles.fasta:Optimizing performance with precomputed files
The BLAST database can be provided to avoid repeating expensive processes (
--blast_db_path).CoreProfiler runs BLAST functions on the query genomes vs. a BLAST database of the species scheme. When the user does not provide a BLAST database, one is created. Building such BLAST database takes some time, and does not need to be repeated unless the scheme has been updated. So, between two updates of a species scheme, consider providing a BLAST database as input. Create one running
coreprofiler db makeblastdb -s [scheme_path] -n [blastdb_name] -p [blastdb_path].'blastdb_name'is used to name output files, so provide a name compatible with file naming. To provide it as input, add the argument-db [blastdb_path/blastdb_name.fasta]tocoreprofiler allele_calling.Testing
Unittary testing
Unittary tests are written with pytest and can be run with the simple
pytestcommand. To test the API and HTML requests, runpytest -m requests.Accuracy verification
CoreProfiler is being thoroughly tested against reference platforms.
References
Authors
Contributors