init v0.2.0

Author: keonlee9420

CITATION.cff

@@ -5,7 +5,7 @@ authors:
   given-names: "Keon"
   orcid: "https://orcid.org/0000-0001-9028-1018"
 title: "Comprehensive-Transformer-TTS"
-version: 0.1.1
+version: 0.2.0
 doi: 10.5281/zenodo.5526991
-date-released: 2021-09-25
+date-released: 2022-02-18
 url: "https://github.com/keonlee9420/Comprehensive-Transformer-TTS"

README.md

@@ -9,6 +9,10 @@
 - [x] [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) (Kitaev et al., 2020)
 - [x] [Attention Is All You Need](https://arxiv.org/abs/1706.03762) (Vaswani et al., 2017)
 
+### Prosody Modelings (WIP)
+- [x] [DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021](https://arxiv.org/abs/2110.12612) (Liu et al., 2021)
+- [x] [Rich Prosody Diversity Modelling with Phone-level Mixture Density Network](https://arxiv.org/abs/2102.00851) (Du et al., 2021)
+
 ### Supervised Duration Modelings
 - [x] [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://arxiv.org/abs/2006.04558) (Ren et al., 2020)
 
@@ -28,18 +32,26 @@
 |Conformer|18903MiB / 24220MiB|7m 4s
 |Reformer|10293MiB / 24220MiB|10m 16s
 |Transformer|7909MiB / 24220MiB|4m 51s
+|Transformer_fs2|11571MiB / 24220MiB|4m 53s
 
 Toggle the type of building blocks by
 ```yaml
 # In the model.yaml
-block_type: "transformer" # ["transformer", "fastformer", "lstransformer", "conformer", "reformer"]
+block_type: "transformer_fs2" # ["transformer_fs2", "transformer", "fastformer", "lstransformer", "conformer", "reformer"]
+```
+
+Toggle the type of prosody modelings by
+```yaml
+# In the model.yaml
+prosody_modeling:
+  model_type: "none" # ["none", "du2021", "liu2021"]
 ```
 
 Toggle the type of duration modelings by
 ```yaml
 # In the model.yaml
 duration_modeling:
-  learn_alignment: True # for unsupervised modeling, False for supervised modeling
+  learn_alignment: True # True for unsupervised modeling, and False for supervised modeling
 ```
 
 # Quickstart
@@ -55,7 +67,7 @@ Also, `Dockerfile` is provided for `Docker` users.
 
 ## Inference
 
-You have to download the [pretrained models](https://drive.google.com/drive/folders/1xEOVbv3PLfGX8EgEkzg1014c9h8QMxQ-?usp=sharing) and put them in `output/ckpt/DATASET/`. The models are trained with unsupervised duration modeling under transformer building block.
+You have to download the [pretrained models](https://drive.google.com/drive/folders/1xEOVbv3PLfGX8EgEkzg1014c9h8QMxQ-?usp=sharing) and put them in `output/ckpt/DATASET/`. The models are trained under unsupervised duration modeling with "transformer_fs2" building block.
 
 For a **single-speaker TTS**, run
 ```
@@ -109,7 +121,7 @@ Any of both **single-speaker TTS** dataset (e.g., [Blizzard Challenge 2013](http
 
   For the forced alignment, [Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io/en/latest/) (MFA) is used to obtain the alignments between the utterances and the phoneme sequences.
   Pre-extracted alignments for the datasets are provided [here](https://drive.google.com/drive/folders/1fizpyOiQ1lG2UDaMlXnT3Ll4_j6Xwg7K?usp=sharing). 
-  You have to unzip the files in `preprocessed_data/DATASET/TextGrid/`. Alternately, you can [run the aligner by yourself](https://montreal-forced-aligner.readthedocs.io/en/latest/aligning.html).
+  You have to unzip the files in `preprocessed_data/DATASET/TextGrid/`. Alternately, you can [run the aligner by yourself](https://montreal-forced-aligner.readthedocs.io/en/latest/user_guide/workflows/index.html).
 
   After that, run the preprocessing script by
   ```
@@ -136,15 +148,22 @@ tensorboard --logdir output/log
 to serve TensorBoard on your localhost.
 The loss curves, synthesized mel-spectrograms, and audios are shown.
 
-![](./img/tensorboard_loss.png)
-![](./img/tensorboard_spec.png)
-![](./img/tensorboard_audio.png)
+## LJSpeech
+
+![](./img/tensorboard_loss_ljs.png)
+![](./img/tensorboard_spec_ljs.png)
+![](./img/tensorboard_audio_ljs.png)
+
+## VCTK
+
+![](./img/tensorboard_loss_vctk.png)
+![](./img/tensorboard_spec_vctk.png)
+![](./img/tensorboard_audio_vctk.png)
 
 # Notes
 
 - Both phoneme-level and frame-level variance are supported in both supervised and unsupervised duration modeling.
 - Note that there are no pre-extracted phoneme-level variance features in unsupervised duration modeling.
-- Convolutional embedding is used as [StyleSpeech](https://github.com/keonlee9420/StyleSpeech) for phoneme-level variance in unsupervised duration modeling. Otherwise, bucket-based embedding is used as [FastSpeech2](https://github.com/ming024/FastSpeech2).
 - Unsupervised duration modeling in phoneme-level will take longer time than frame-level since the additional computation of phoneme-level variance is activated at runtime.
 - Two options for embedding for the **multi-speaker TTS** setting: training speaker embedder from scratch or using a pre-trained [philipperemy's DeepSpeaker](https://github.com/philipperemy/deep-speaker) model (as [STYLER](https://github.com/keonlee9420/STYLER) did). You can toggle it by setting the config (between `'none'` and `'DeepSpeaker'`).
 - DeepSpeaker on VCTK dataset shows clear identification among speakers. The following figure shows the T-SNE plot of extracted speaker embedding.
@@ -155,6 +174,27 @@ The loss curves, synthesized mel-spectrograms, and audios are shown.
 
 - For vocoder, **HiFi-GAN** and **MelGAN** are supported.
 
+### Updates Log
+- Feb.18, 2022 (v0.2.0): Update data preprocessor and variance adaptor & losses following [keonlee9420's DiffSinger](https://github.com/keonlee9420/DiffSinger) / Add various prosody modeling methods
+  1. Prepare two different types of data pipeline in preprocessor to maximize unsupervised/supervised duration modelings
+  2. Adopt wavelet for pitch modeling & loss
+  3. Add fine-trained duration loss
+  4. Apply `var_start_steps` for better model convergence, especially under unsupervised duration modeling
+  5. Remove dependency of energy modeling on pitch variance
+  6. Add "transformer_fs2" building block, which is more close to the original FastSpeech2 paper
+  7. Add two types of prosody modeling methods
+  8. Loss camparison on validation set:
+    - LJSpeech - blue: v0.1.1 / green: v0.2.0
+    <p align="center">
+        <img src="./img/loss_comparison_ljs.png" width="80%">
+    </p>
+
+    - VCTK - skyblue: v0.1.1 / orange: v0.2.0
+    <p align="center">
+        <img src="./img/loss_comparison_vctk.png" width="80%">
+    </p>
+- Sep.21, 2021 (v0.1.1): Initialize with [ming024's FastSpeech2](https://github.com/ming024/FastSpeech2)
+
 # Citation
 
 Please cite this repository by the "[Cite this repository](https://github.blog/2021-08-19-enhanced-support-citations-github/)" of **About** section (top right of the main page).
@@ -166,4 +206,8 @@ Please cite this repository by the "[Cite this repository](https://github.blog/2
 - [lucidrains' long-short-transformer](https://github.com/lucidrains/long-short-transformer)
 - [sooftware's conformer](https://github.com/sooftware/conformer)
 - [lucidrains' reformer-pytorch](https://github.com/lucidrains/reformer-pytorch)
+- [sagelywizard's pytorch-mdn](https://github.com/sagelywizard/pytorch-mdn)
+- [keonlee9420's Robust_Fine_Grained_Prosody_Control](https://github.com/keonlee9420/Robust_Fine_Grained_Prosody_Control)
+- [keonlee9420's Cross-Speaker-Emotion-Transfer](https://github.com/keonlee9420/Cross-Speaker-Emotion-Transfer)
+- [keonlee9420's DiffSinger](https://github.com/keonlee9420/DiffSinger)
 - [NVIDIA's NeMo](https://github.com/NVIDIA/NeMo): Special thanks to [Onur Babacan](https://github.com/babua) and [Rafael Valle](https://github.com/rafaelvalle) for unsupervised duration modeling.

audio/stft.py

@@ -2,14 +2,21 @@
 import torch.nn.functional as F
 import numpy as np
 from scipy.signal import get_window
+import librosa
 from librosa.util import pad_center, tiny
 from librosa.filters import mel as librosa_mel_fn
+import pyloudnorm as pyln
 
 from audio.audio_processing import (
     dynamic_range_compression,
     dynamic_range_decompression,
     window_sumsquare,
 )
+from audio.tools import (
+    librosa_pad_lr,
+    amp_to_db,
+    normalize,
+)
 
 
 class STFT(torch.nn.Module):
@@ -176,3 +183,77 @@ def mel_spectrogram(self, y):
         energy = torch.norm(magnitudes, dim=1)
 
         return mel_output, energy
+
+
+class FastSpeechSTFT(torch.nn.Module):
+    def __init__(
+        self,
+        fft_size,
+        hop_size,
+        win_length,
+        num_mels,
+        sample_rate,
+        fmin,
+        fmax,
+        window='hann',
+        eps=1e-10,
+        loud_norm=False,
+        min_level_db=-100,
+    ):
+        super(FastSpeechSTFT, self).__init__()
+        self.fft_size = fft_size
+        self.hop_size = hop_size
+        self.win_length = win_length
+        self.num_mels = num_mels
+        self.sample_rate = sample_rate
+        self.fmin = fmin
+        self.fmax = fmax
+        self.window = window
+        self.eps = eps
+        self.loud_norm = loud_norm
+        self.min_level_db = min_level_db
+
+    def mel_spectrogram(self, wav, return_linear=False):
+        """Computes mel-spectrograms from a batch of waves
+        PARAMS
+        ------
+        wav: Variable(torch.FloatTensor) with shape (B, T) in range [-1, 1]
+
+        RETURNS
+        -------
+        mel_output: torch.FloatTensor of shape (B, n_mel_channels, T)
+        """
+        if self.loud_norm:
+            meter = pyln.Meter(self.sample_rate)  # create BS.1770 meter
+            loudness = meter.integrated_loudness(wav)
+            wav = pyln.normalize.loudness(wav, loudness, -22.0)
+            if np.abs(wav).max() > 1:
+                wav = wav / np.abs(wav).max()
+
+        # get amplitude spectrogram
+        x_stft = librosa.stft(wav, n_fft=self.fft_size, hop_length=self.hop_size,
+                            win_length=self.win_length, window=self.window, pad_mode="constant")
+        spc = np.abs(x_stft)  # (n_bins, T)
+
+        # get mel basis
+        fmin = 0 if self.fmin == -1 else self.fmin
+        fmax = sample_rate / 2 if self.fmax == -1 else self.fmax
+        mel_basis = librosa.filters.mel(self.sample_rate, self.fft_size, self.num_mels, self.fmin, self.fmax)
+        mel = mel_basis @ spc
+
+        # get log scaled mel
+        mel = np.log10(np.maximum(self.eps, mel))
+
+        l_pad, r_pad = librosa_pad_lr(wav, self.fft_size, self.hop_size, 1)
+        wav = np.pad(wav, (l_pad, r_pad), mode='constant', constant_values=0.0)
+        wav = wav[:mel.shape[1] * self.hop_size]
+
+        # get energy
+        energy = np.sqrt(np.exp(mel) ** 2).sum(-1)
+
+        if not return_linear:
+            return wav, mel, energy
+        else:
+            spc = amp_to_db(spc)
+            spc = normalize(spc, self.min_level_db)
+            return wav, mel, energy, spc

audio/tools.py

@@ -32,3 +32,24 @@ def inv_mel_spec(mel, out_filename, _stft, griffin_iters=60):
     audio = audio.cpu().numpy()
     audio_path = out_filename
     write(audio_path, _stft.sampling_rate, audio)
+
+
+def librosa_pad_lr(x, fsize, fshift, pad_sides=1):
+    '''compute right padding (final frame) or both sides padding (first and final frames)
+    '''
+    assert pad_sides in (1, 2)
+    # return int(fsize // 2)
+    pad = (x.shape[0] // fshift + 1) * fshift - x.shape[0]
+    if pad_sides == 1:
+        return 0, pad
+    else:
+        return pad // 2, pad // 2 + pad % 2
+
+
+# Conversions
+def amp_to_db(x):
+    return 20 * np.log10(np.maximum(1e-5, x))
+
+
+def normalize(S, min_level_db):
+    return (S - min_level_db) / -min_level_db

config/LJSpeech/model.yaml

@@ -1,9 +1,46 @@
-block_type: "transformer"
+block_type: "transformer_fs2" # ["transformer_fs2", "transformer", "fastformer", "lstransformer", "conformer", "reformer"]
 
 duration_modeling:
   learn_alignment: True
   aligner_temperature: 0.0005
 
+prosody_modeling:
+  model_type: "none" # ["none", "du2021", "liu2021"]
+
+  # Du et al., 2021
+  # This is only supported under supervised duration modeling (learn_alignment: False)
+  du2021:
+    extractor_kernel_size: 9
+    predictor_kernel_size: [9, 5]
+    predictor_num_gaussians: 20
+    predictor_dropout: 0.2
+
+  # Liu et al., 2021
+  # This is only tested under supervised duration modeling (learn_alignment: False)
+  liu2021:
+    bottleneck_size_u: 256
+    bottleneck_size_p: 4
+    ref_enc_filters: [32, 32, 64, 64, 128, 128]
+    ref_enc_size: [3, 3]
+    ref_enc_strides: [1, 2] # '1' is to keep the sequence length
+    ref_enc_pad: [1, 1]
+    ref_enc_gru_size: 32
+    ref_attention_dropout: 0.
+    token_num: 32
+    predictor_kernel_size: 3 # [9, 5] for non-parallel predictor / 3 for parallel predictor
+    predictor_dropout: 0.5
+
+transformer_fs2:
+  encoder_layer: 4
+  encoder_head: 2
+  encoder_hidden: 256
+  decoder_layer: 6
+  decoder_head: 2
+  decoder_hidden: 256
+  ffn_kernel_size: 9
+  encoder_dropout: 0.1
+  decoder_dropout: 0.1
+
 transformer:
   encoder_layer: 4
   encoder_head: 2
@@ -37,18 +74,27 @@ reformer:
 
 variance_predictor:
   filter_size: 256
-  kernel_size: 3
+  predictor_grad: 0.1
+  predictor_layers: 2
+  predictor_kernel: 5
+  cwt_hidden_size: 128
+  cwt_std_scale: 0.8
+  dur_predictor_layers: 2
+  dur_predictor_kernel: 3
   dropout: 0.5
+  ffn_padding: "SAME"
+  ffn_act: "gelu"
 
 variance_embedding:
-  kernel_size: 9
-  pitch_quantization: "linear" # support 'linear' or 'log', 'log' is allowed only if the pitch values are not normalized during preprocessing
+  use_pitch_embed: True
+  pitch_n_bins: 300
+  use_energy_embed: True
+  energy_n_bins: 256
   energy_quantization: "linear" # support 'linear' or 'log', 'log' is allowed only if the energy values are not normalized during preprocessing
-  n_bins: 256
 
 multi_speaker: False
 
-max_seq_len: 1000
+max_seq_len: 1000 # max sequence length of LJSpeech is 870
 
 vocoder:
   model: "HiFi-GAN" # support 'HiFi-GAN', 'MelGAN'

config/LJSpeech/preprocess.yaml

@@ -12,6 +12,7 @@ preprocessing:
     text_cleaners: ["english_cleaners"]
     language: "en"
   audio:
+    trim_top_db: 23
     sampling_rate: 22050
     max_wav_value: 32768.0
   stft:
@@ -23,8 +24,14 @@ preprocessing:
     mel_fmin: 0
     mel_fmax: 8000 # please set to 8000 for HiFi-GAN vocoder, set to null for MelGAN vocoder
   pitch:
-    feature: "phoneme_level" # support 'phoneme_level' or 'frame_level'
-    normalization: True
+    pitch_type: "cwt" # support 'frame', 'ph', 'cwt'
+    pitch_norm: "log" # support 'standard', 'log'
+    pitch_norm_eps: 0.000000001
+    pitch_ar: False
+    with_f0: True
+    with_f0cwt: True
+    use_uv: True
+    cwt_scales: -1
   energy:
     feature: "phoneme_level" # support 'phoneme_level' or 'frame_level'
     normalization: True

config/LJSpeech/train.yaml

@@ -17,13 +17,28 @@ optimizer:
   warm_up_step: 4000
   anneal_steps: [300000, 400000, 500000]
   anneal_rate: 0.3
+loss:
+  noise_loss: "l1"
+  dur_loss: "mse"
+  pitch_loss: "l1"
+  cwt_loss: "l1"
+  # cwt_add_f0_loss: false
+  lambda_f0: 1.0
+  lambda_uv: 1.0
+  lambda_ph_dur: 1.0
+  lambda_word_dur: 1.0
+  lambda_sent_dur: 1.0
 step:
   total_step: 900000
   log_step: 100
   synth_step: 1000
   val_step: 1000
   save_step: 25000
+  var_start_steps: 50000
 duration:
   binarization_start_steps: 6000
   binarization_loss_enable_steps: 18000
   binarization_loss_warmup_steps: 10000
+prosody:
+  gmm_mdn_beta: 0.02
+  prosody_loss_enable_steps: 100000