Introduction

In recent years, text-to-speech (TTS) has accomplished remarkable improvement with the emergence of various end-to end TTS models [1, 2, 3]. Through these advanced models, TTS expands its field from a model built with a professional voice actor to a personalized TTS. To make the TTS model personalize, there have been several attempts to finetune the pre-trained model with a target speaker data [5]. Even though the required quantity of the target speaker data for the fine-tuning is small, it is troublesome to collect such personal data and perform fine-tuning. The zero-shot TTS (zs-TTS) [6, 7, 8, 9, 10, 11], which uses a single utterance to clone the voice of the target speaker without additional fine-tuning, resolves these inconveniences. The most common approach in zs-TTS is to condition speaker embedding to typical end-to-end TTS architecture.

As perceiving speaker similarity is a complex and ambiguous problem, an insightful definition of speaker similarity should precede. Thus, we assume that the speaker similarity between two speeches can be viewed in two aspects. To clarify, we restrict the definition of two terms to explain these aspects. First, each speaker has a unique identity of voice regardless of the textual context of an utterance, and we will call this property as timbre. The other term is cadence, which is an inter-utterance variant component related to prosody, style, and variation of tone within the same speaker.

In this blog post, we introduce our works, a timbre-cadence speaker encoder (TiCa) that has a hierarchical structure to model timbre and cadence separately.

Also, the conventional zs-TTS models have shown unstable performance such as switching timbre within an utterance. Thus, we introduce an effective augmentation method called speaker mixing augmentation (SMAug) to make a robust zs-TTS model.

Proposed Method

1. Timbre-cadence Speaker Encoder

To separate timbre and cadence, we suggest a hierarchical speaker encoder as in Figure 2. As the speaker encoder encodes gradually from local to global information, the cadence embedding is first extracted using the attention pooling at the lower part of the speaker encoder. Then, the output of the attention pooling layer of the cadence embedding is subtracted to disentangle the timbre and cadence information. The timbre embedding is extracted after two convolution blocks followed by attention pooling. Finally, the timbre and cadence embeddings are concatenated and form the speaker embedding conditioned on the zero-shot TTS (zs-TTS) framework.

Furthermore, we utilize two supplementary losses for training the timbre-cadence speaker encoder (TiCa). First, to keep the timbre embedding consistent within the same speaker, we give L1-loss against the speaker ID-based speaker embedding acquired by an embedding table.

Second, we adopt the VICReg [12] to regularize the cadence embeddings within a batch to be distinct from each other and to decorrelate the variables of each cadence embedding. By using such losses, we can successfully restrict the timbre embedding to have a smaller discrepancy within the same speaker and enlarge the diversity of the cadence embedding among different utterances.

To train the TiCa stably, in the early stage of the training, we use the speaker ID embedding instead of the timbre embedding. Then, to reduce the mismatch of the training and inference, we train the zs-TTS model conditioning on the timbre embedding while freezing the speaker ID embedding table for certain iterations.

2. Speaker Mixing Augmentation

In typical zs-TTS models, a single vector speaker embedding is broadcasted into the phoneme sequence length or acoustic feature sequence length and then used as a condition on TTS model. However, since the speaker embedding is fixed throughout the whole utterance, it can weaken the role of the speaker embedding and result in synthesized speech with unstable speaker similarity during inference.

To overcome this problem, SMAug concatenates two short utterances from different speakers. Using SMAug, the model encounters two different speakers in one integrated utterance, which enhances the robustness of the zs-TTS model.

Data augmentation in typical TTS is limited since modification in target speech can result in severe performance degradation in naturalness. However, as the SMAug does not perturb the speech samples, it can augment the data while not harming the naturalness of the TTS models.

Furthermore, as the SMAug has an advantage in compatibility, it can be expanded to other applications easily.

Experiments

1. Model

To show the performance of our proposed method, we performed experiments by replacing speaker encoder modules in a fixed end-to-end model which is a non-attentive version of [1] (NALPCTron). The overall architecture of an acoustic model in NALPCTron is in Figure 4 and we employed the Bunched LPCNet [13] as a neural vocoder.

2. Experiment Setup

We evaluated the timbre-cadence speaker encoder (TiCa) with the conventional reference encoder-based speaker encoders. First, we utilized a vanilla reference encoder [14]-based speaker embedding (REF) which consists of convolution blocks and a recurrent pooling layer. Second, we applied Meta-StyleSpeech [15]-based speaker encoder (META) which is a self-attention based method. Third, we adopted speaker embedding from the pre-trained speaker verification (SV) model1 (EXTERN) [16]. Also, to show the effect of SMAug, we conduct some experiments without SMAug (TiCa-NoAug). All these models were trained with LibriTTS train-clean-360 set.

3. Evaluation Metrics

For an objective test, we evaluated the phoneme error rate (PER) of the speech samples with an automatic speech recognition (ASR) model to predict pronunciation accuracy. We employ ASR model which is finetuned over the XLSR-53 model in Wav2vec 2.0 model [17]. In addition, averaged speaker embedding cosine similarity (SECS) was performed to evaluate the speaker similarity between the ground truth (GT) samples and synthesized samples. For the SECS, we adopted the pre-trained ECAPA-TDNN model [18] which is one of the SOTA SV models. The SECS ranges from 0 to 1, and a higher score implies better speaker similarity.

For the subjective evaluations, we measured the mean opinion score (MOS) and comparative similarity MOS (CSMOS). MOS estimates the perceptual speech quality by testers in the range from 1 to 5 with an interval of 1, where 5 is the best. For CSMOS, testers listened to one reference speech and two synthesized utterances including TiCa, then were asked to choose which utterance is similar to the reference speech in terms of timbre and cadence with ranges from -3 (the comparative model is much worse than TiCa) to 3 (the comparative model is much better than TiCa) with an interval of 1.

4. Evaluation on Unseen Speakers (zero-shot TTS)

As shown in Table 1, TiCa showed superior performance in speech quality such as pronunciation accuracy and naturalness in terms of PER and MOS results. The PER of GT was worse than the others since some noisy speeches were contained in the GT while the synthesized speeches of the zs-TTS models were typically clean. However, when it comes to the MOS test, GT showed a higher score, because the testers were guided to focus on the perceptual naturalness of speech.

In terms of speaker similarity, the CSMOS results demonstrated that TiCa had better perceptual speaker similarity than the comparison TTS models. However, the overall tendency of SECS results was different from perceptual tests. As the human sense of speaker similarity is a combination of timbre and cadence, this can be a reason that the tendency of CSMOS and SECS was different. Especially, for EXTERN, it achieved the best SECS results but showed worse results than TiCa and META in the CSMOS test. This implies that EXTERN only focused on encoding the timbre of the speaker.

Comparing the objective results of TiCa and TiCa-NoAug, it proves the positive effect of SMAug in terms of pronunciation accuracy and speaker similarity.

Conclusions

In this post, we introduce a timbre-cadence speaker encoder (TiCa) as a novel technique for cloning a target speaker’s voice. This approach assumes that speaker embedding can be viewed as a combination of timbre and cadence. To model these components, TiCa extracts timbre and cadence with a hierarchical structure and some effective additional losses. Moreover, we introduce a simple but powerful speaker mixing augmentation (SMAug) that concatenates two utterances from the different speakers for robust zero-shot TTS. From the experimental results, it showed that our proposed methods outperform the conventional other reference encoder-based speaker encoders.

Link to the paper

https://www.isca-speech.org/archive/interspeech_2023/lee23f_interspeech.html

Audio Samples

https://srtts.github.io/tc-zstts