DSRC

DSRC is a toolkit designed for efficient high-performance compression of sequencing reads stored in FASTQ format, where it’s main features are:

Effective multithreaded compression of FASTQ files.
Full support for Illumina, ABI SOLiD, and 454/Ion Torrent dataset formats with non-standard (AGCTN) IUPAC base values.
Support for lossy quality values compression using Illumina binning scheme.
Support for lossy IDs compression keeping only key fields selected by user.
Pipes support for easy integration with current pipelines.
Python and C++ libraries allowing to integrate DSRC archives in own applications.
Availability for Linux, Mac OSX and Windows 64-bit operating systems.
Open source C++ code under GNU GPL 2 license.

Building

Build prerequisites

Linux

DSRC binaries and C++ library can be compiled in two ways, depending on the selection of multithreading support library - for each a different makefile file is provided. In the first case, boost::threads library will be used, which is needed to be present on the build system. In the second - g++ compiler with c++11 support (version >= 4.8).

By default, binaries and libraries are compiled using g++, however compiling using Clang or Intel icpc should also succeeed without any problems.

Mac OSX

On Mac OSX Clang compiler will be used with c++11 support, so make sure to have Clang in version >= 3.3 installed.

Windows

To compile DSRC under Windows OS, Microsoft Visual Studio 2010 or 2012 is required. DSRC binaries and C++ library can be compiled in two ways, depending on the selection of multithreading support library - for each a different VS solution file is provided. When compiling using VS2010 the boost::threads library will be used to provide multithreading support, so make sure to have boost::threads library installed and boost library paths properly configured in Visual Studio. In case of using VS2012 c++11 standard implementation will be used to provide threading support.

There should be also no problems when compiling DSRC using MinGW-32-x64 with provided Makefile files.

Python library

To build DSRC Python library, boost::python library in development version and boost::build tool bjam are need to be present on the system. Next, in the Jamroot configuration file in py directory a local boost installation directory needs to be specified:

# To compile DSRC Python module please specify your boost installation directory below
#
use-project boost 
    : /absolute/path/to/boost/directory/ ;

Python library will be built using a default compilation toolset available on the build platform (auto selected by bjam), however in order to specify a different one append

<toolset>name

to the compilation flags as exmplained in the Jamroot file

# Specify toolset according to your platform manually in case of compilation problems in form: '<toolset>gcc'
# Available toolsets:
#    - Windows: msvc-*
#    - Linux: gcc, clang
#    - Mac OSX: darwin, gcc
    : <variant>release <address-model>64 <link>shared <runtime-link>shared <debug-symbols>off <inlining>full <optimization>speed <warnings>on <cxxflags>"-O2 -m64 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DUSE_BOOST_THREAD" ;

Building on Linux

Binary

To compile DSRC using boost::threads with static linking, in the main directory type:

make bin

To compile DSRC using g++ >= 4.8 with c++11 standard and dynamic linking:

make -f Makefile.c++11 bin

The resulting dsrc binary will be placed in bin subdirectory.

C++ library

To compile C++ DSRC library using boost::threads:

make lib

To compile DSRC using g++ >= 4.8 with c++11:

make -f Makefile.c++11 lib

The resulting libdsrc.a library will be placed in lib subdirectory.

Python library

To compile DSRC Python library:

make pylib

The resulting pydsrc.so library will be available in py subdirectory.

Building on Mac OSX

Binary

To compile DSRC binary, in the main directory type:

make -f Makefile.osx bin

The resulting dsrc binary will be placed in bin subdirectory.

C++ library

To compile DSRC C++ library:

make -f Makefile.osx lib

The resulting libdsrc.a library will be placed in lib subdirectory.

Python library

To compile DSRC Python library:

make -f Makefile.osx pylib

The resulting pydsrc.so library will be available in py subdirectory.

Building on Windows 64-bit

Binary

To compile DSRC using Visual Studio 2010 with boost::threads for multithreading support use the dsrc20-vs2k10.sln solution file. However, to compile DSRC using Visual Studio 2012 with c++11 threads use the dsrc20-vs2k12.sln.

To compile DSRC executable, select Release|x64 configuration and build.

The resulting dsrc.exe executable will be placed in bin subdirectory.

C++ library

To compile DSRC library, select Release Lib|x64 configuration and build.

The resulting dsrc.lib library will be placed in lib subdirectory.

Python library

To compile DSRC Python library in the py subdirectory type:

bjam

The resulting pydsrc.pyd library will be available in py subdirectory.

Usage

DSRC can be run from the command prompt:

dsrc <c|d> [options] <input_file_name> <output_file_name>

in one of two modes:

c — compression,
d — decompression.

Available options

Compression options

-d<n> — DNA compression mode: 0–3, default: 0
-q<n> — Quality compression mode: 0–2, default: 0
-f<1,...> — keep only those fields no. in ID field string, default: (keep all)
-b<n> — FASTQ input buffer size in MB, default: 8
-m<n> — Automated compression mode (one of the three preset combination of other pa- rameters): 0–2
-o<n> — Quality offset, 0 for auto selection, default: 0
-l — use Quality lossy mode (Illumina binning scheme), default: false
-c — calculate and check CRC32 checksum calculation per block (slows the compression about twice), default: false

Automated compression modes

-m0 — fast mode, equivalent to: -d0 -q0 -b8
-m1 — medium mode, equivalent to: -d2 -q2 -b64
-m2 — best mode, equivalent to: -d3 -q2 -b256

Options for both compression and decompression

-t<n> — processing threads number, default: max available hardware threads
-s — use stdin/stdout for reading/writing raw FASTQ files data (stderr is used for info/warning messages)

Usage examples

Compress SRR001471.fastq file saving DSRC archive to SRR001471.dsrc:

dsrc c SRR001471.fastq SRR001471.dsrc

Compress file in the fast mode with CRC32 checking and using 4 threads:

dsrc c -m0 -c -t4 SRR001471.fastq SRR001471.dsrc

Compress file using DNA and Quality compression level 2 and using 512 MB buffer:

dsrc c -d2 -q2 -b512 SRR001471.fastq SRR001471.dsrc

Compress file in the best mode with lossy Quality mode and preserving only 1–4 fields from record IDs:

dsrc c -m2 -l -f1,2,3,4 SRR001471.fastq SRR001471.dsrc

Compress in the best mode reading raw FASTQ data from stdin:

cat SRR001471.fastq | dsrc c -m2 -s SRR001471.dsrc

Decompress SRR001471.dsrc archive saving output FASTQ file to SRR001471.out.fastq:

dsrc d SRR001471.dsrc SRR001471.out.fastq

Decompress archive using 4 threads and streaming raw FASTQ data to stdout:

dsrc d -t4 -s SRR001471.dsrc > SRR001471.out.fastq

Citing

Roguski, L., Deorowicz, S. (2014) DSRC 2: Industry-oriented compression of FASTQ files, Bioinformatics, 30(15):2213–2215.

Deorowicz, S., Grabowski, Sz. (2011) Compression of DNA sequences in FASTQ format, Bioinformatics, 27(6):860–862.