Universal Source Separation (USS) with Weakly labelled Data
This is the PyTorch implementation of the Universal Source Separation with Weakly labelled Data [1]. The USS system can automatically detect and separate sound classes from a real recording. The USS system can separate up to hundreds of sound classes sound classes in a hierarchical ontology structure. The USS system is trained on the weakly labelled AudioSet dataset only. Here is a demo:
Download the AudioSet dataset from the internet. The total size of AudioSet is around 1.1 TB. For reproducibility, our downloaded dataset can be accessed at: link: https://pan.baidu.com/s/13WnzI1XDSvqXZQTS-Kqujg, password: 0vc2. Users may only download the balanced set (10.36 Gb, 1% of the full set) to train a baseline system.
The downloaded data looks like:
audioset
├── audios
│ ├── balanced_train_segments
│ | └── ... (~20550 wavs, the number can be different from time to time)
│ ├── eval_segments
│ | └── ... (~18887 wavs)
│ └── unbalanced_train_segments
│ ├── unbalanced_train_segments_part00
│ | └── ... (~46940 wavs)
│ ...
│ └── unbalanced_train_segments_part40
│ └── ... (~39137 wavs)
└── metadata
├── balanced_train_segments.csv
├── class_labels_indices.csv
├── eval_segments.csv
├── qa_true_counts.csv
└── unbalanced_train_segments.csv
Notice there can be missing files on YouTube, so the numebr of files downloaded by users can be different from time to time. Our downloaded version contains 20550 / 22160 of the balaned training subset, 1913637 / 2041789 of the unbalanced training subset, and 18887 / 20371 of the evaluation subset.
4.1 Pack waveforms into hdf5 files
Audio files in a subdirectory will be packed into an hdf5 file. There will be 1 balanced train + 41 unbalanced train + 1 evaluation hdf5 files in total.
Create 100 2-second mixture and source pairs to evaluate the separation result of each sound class. There are in total 52,700 2-second pairs for 527 sound classes. The validation data is used to calculate the loss during training.
@article{kong2023universal,
title={Universal Source Separation with Weakly Labelled Data},
author={Kong, Qiuqiang and Chen, Ke and Liu, Haohe and Du, Xingjian and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo and Plumbley, Mark D},
journal={arXiv preprint arXiv:2305.07447},
year={2023}
}
FAQ
If from uss.config import CLASSES_NUM occur, then execute:
Universal Source Separation (USS) with Weakly labelled Data
This is the PyTorch implementation of the Universal Source Separation with Weakly labelled Data [1]. The USS system can automatically detect and separate sound classes from a real recording. The USS system can separate up to hundreds of sound classes sound classes in a hierarchical ontology structure. The USS system is trained on the weakly labelled AudioSet dataset only. Here is a demo:
1. Installation
Prepare environment (optional)
Install the package
2. Separate Your Favorite Audio
2.1 Download test audio (optional)
2.2 Detect and separate sound classes in hierarchical levels (default)
2.3 The same as default
2.4 Separate by class IDs
2.5 Separate by queries
Download query audios (optional)
Do separation
3. Git Clone the Repo and do Inference
Users could also git clone this repo and run the inference in the repo. This will let users to have more flexibility to modify the inference code.
3.1 Set up environment
3.2 Inference
Download our pretrained checkpoint:
Then perform the inference:
4. Train the USS system from scratch
4.0 Download dataset
Download the AudioSet dataset from the internet. The total size of AudioSet is around 1.1 TB. For reproducibility, our downloaded dataset can be accessed at: link: https://pan.baidu.com/s/13WnzI1XDSvqXZQTS-Kqujg, password: 0vc2. Users may only download the balanced set (10.36 Gb, 1% of the full set) to train a baseline system.
The downloaded data looks like:
audioset ├── audios │ ├── balanced_train_segments │ | └── ... (~20550 wavs, the number can be different from time to time) │ ├── eval_segments │ | └── ... (~18887 wavs) │ └── unbalanced_train_segments │ ├── unbalanced_train_segments_part00 │ | └── ... (~46940 wavs) │ ... │ └── unbalanced_train_segments_part40 │ └── ... (~39137 wavs) └── metadata ├── balanced_train_segments.csv ├── class_labels_indices.csv ├── eval_segments.csv ├── qa_true_counts.csv └── unbalanced_train_segments.csvNotice there can be missing files on YouTube, so the numebr of files downloaded by users can be different from time to time. Our downloaded version contains 20550 / 22160 of the balaned training subset, 1913637 / 2041789 of the unbalanced training subset, and 18887 / 20371 of the evaluation subset.
4.1 Pack waveforms into hdf5 files
Audio files in a subdirectory will be packed into an hdf5 file. There will be 1 balanced train + 41 unbalanced train + 1 evaluation hdf5 files in total.
The packed hdf5 files looks like:
workspaces/uss └── hdf5s └── waveforms (1.1 TB) ├── balanced_train.h5 ├── eval.h5 └── unbalanced_train ├── unbalanced_train_part00.h5 ... └── unbalanced_train_part40.h54.2 Create indexes for balanced training
Pack indexes into hdf5 files for balanced training.
The packed indexes files look like:
workspaces/uss └── hdf5s └── indexes (3.0 GB) ├── balanced_train.h5 ├── eval.h5 └── unbalanced_train ├── unbalanced_train_part00.h5 ... └── unbalanced_train_part40.h54.3 Create validation data
Create 100 2-second mixture and source pairs to evaluate the separation result of each sound class. There are in total 52,700 2-second pairs for 527 sound classes. The validation data is used to calculate the loss during training.
The validation data look like:
workspaces/uss └── evaluation └── audioset ├── 2s_segments_balanced_train.csv ├── 2s_segments_test.csv ├── 2s_segments_balanced_train │ ├── class_id=0 │ │ └── ... (100 mixture + 100 clean) │ │... │ └── class_id=526 │ └── ... (100 mixture + 100 clean) └── 2s_segments_test ├── class_id=0 │ └── ... (100 mixture + 100 clean) │... └── class_id=526 └── ... (100 mixture + 100 clean)4.4 Train
Train the universal source separation system.
Or simply execute:
5 Evaluation
We provide AudioSet, FSDKaggle2018, FSD50k, Slakh2100, MUSDB18, and Voicebank-Demand datasets to evaluate the USS system.
5.0 Data processing
Users can skip this stage and directly download the processed evaluation data listed in the table below.
In case the users wish to process the evaluation data by themselves, please run the following scripts.
5.1 Calculate the query embeddings of datasets.
Users may run the following script line by line.
5.2 Separate and evaluate on datasets.
Users may run the following script line by line.
Citation
paper link
@article{kong2023universal, title={Universal Source Separation with Weakly Labelled Data}, author={Kong, Qiuqiang and Chen, Ke and Liu, Haohe and Du, Xingjian and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo and Plumbley, Mark D}, journal={arXiv preprint arXiv:2305.07447}, year={2023} }FAQ
If from uss.config import CLASSES_NUM occur, then execute: