efficient neural audio synthesis in the waveform domain

Last update: Dec 23, 2022

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

by Ben Hayes, Charalampos Saitis, György Fazekas

This repository is the official implementation of Neural Waveshaping Synthesis.

Model Architecture

Requirements

To install:

pip install -r requirements.txt
pip install -e .

We recommend installing in a virtual environment.

Data

We trained our checkpoints on the URMP dataset. Once downloaded, the dataset can be preprocessed using scripts/create_urmp_dataset.py. This will consolidate recordings of each instrument within the dataset and preprocess them according to the pipeline in the paper.

python scripts/create_urmp_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/urmp \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Alternatively, you can supply your own dataset and use the general create_dataset.py script:

python scripts/create_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/dataset \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Training

To train a model on the URMP dataset, use this command:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/urmp \
  --urmp \
  --instrument vn \  # select URMP instrument with abbreviated string
  --load-data-to-memory

Or to use a non-URMP dataset:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/data \
  --load-data-to-memory

efficient neural audio synthesis in the waveform domain

Related tags

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

Model Architecture

Requirements

Data

Training

Owner

Ben Hayes

Evolution Strategies in PyTorch

This is the source code for the experiments related to the paper Unsupervised Audio Source Separation Using Differentiable Parametric Source Models

Synthetic structured data generators

A python-image-classification web application project, written in Python and served through the Flask Microframework. This Project implements the VGG16 covolutional neural network, through Keras and Tensorflow wrappers, to make predictions on uploaded images.

Quantized models with python

TLXZoo - Pre-trained models based on TensorLayerX

CRF-RNN for Semantic Image Segmentation - PyTorch version

Bib-parser - Convenient script to parse .bib files with the ACM Digital Library like metadata

This is a template for the Non-autoregressive Deep Learning-Based TTS model (in PyTorch).

Official Implementation of "Learning Disentangled Behavior Embeddings"

A Re-implementation of the paper "A Deep Learning Framework for Character Motion Synthesis and Editing"

Pretrained Cost Model for Distributed Constraint Optimization Problems

The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Learning to See by Looking at Noise

Multi-label classification of retinal disorders

Simple Tensorflow implementation of "Adaptive Convolutions for Structure-Aware Style Transfer" (CVPR 2021)

Mixed Neural Likelihood Estimation for models of decision-making

TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning

ECAENet (TensorFlow and Keras)

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.