Interspeech is the world’s leading conference on the science and technology of speech recognition, speech synthesis, speaker recognition and speech and language processing. The conference plays a crucial role in setting new technology trends and standards, as well as providing direction for future research. In this blog series, we are introducing some of our research papers at INTERSPEECH 2024 and here is a list of them. #1. Relational Proxy Loss for Audio-Text based Keyword Spotting (Samsung Research) #2. NL-ITI: Probing optimization for improvement of LLM intervention method (Samsung R&D Institute Poland) #3. High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model (Samsung Research) #4. Speaker personalization for automatic speech recognition using Weight-Decomposed Low-Rank Adaptation (Samsung R&D Institute India-Bangalore) #5. Speech Boosting: developing an efficient on-device live speech enhancement (Samsung Research) #6. SummaryMixing makes speech technologies faster and cheaper (Samsung AI Center - Cambridge) #7. A Unified Approach to Multilingual Automatic Speech Recognition with Improved Language Identification for Indic Languages (Samsung R&D Institute India-Bangalore)

Background

Personalizing automated speech recognition (ASR) for voice assistant systems is often considered the holy grail, requiring meticulous attention to detail in model optimization. When dealing with limited speaker data, the selection of hyperparameters becomes paramount in fine-tuning large ASR models. One effective method for this optimization is low-rank adaptation (LoRA), which proves instrumental in enhancing the performance of large language models (LLMs). A variation of LoRA, Weight-Decomposed Low-Rank Adaptation (DoRA) also promises enhanced performance.

In our study, we employed LoRA and DoRA, to refine the state-of-the-art cascaded conformer transducer model for speaker personalization. This involved adding a small number of speaker-specific weights to the existing model and fine-tuning them accordingly. Experimental assessments show an average relative improvement of 20% in word error rate across speakers with limited data, showcasing its efficacy in addressing the challenge of personalizing ASR systems in real-world applications.

Proposed Approach

Two variations of low-rank adaptations, LoRA and DoRA, are proposed for speaker personalisation in the ASR systems. These approaches are depicted in Figure 1. We propose two approaches of PEFT for speaker personalisation; The first proposed approach (proposed-1) is with LoRA and the second proposed approach (proposed-2) is with DoRA.

2.1. LoRA-based speaker personalisation (Proposed 1)

In the LoRA method, we add a small set of parameters to the weight matrix. The weight matrix is modified in a layer with the help of a new set of weight matrices as shown in Figure 1 (LoRA training). For a pre-trained weight matrix W₀ϵR_d∗k, we constrain its update by representing the latter with a low-rank decomposition
W′ = W₀ + ΔW = W₀ + BA (1)
where BϵR_d∗r,AϵR_r∗k, and the rank r ≪ min(d, k). During training, W₀ is frozen and does not receive gradient updates, while A and B contain trainable parameters. Note both W₀ and ΔW = BA are multiplied with the same input, and their respective output vectors are summed coordinate-wise. For h = W_0x, our modified forward pass yields:
h = W′_x = W_0x + BAx (2)
We use a random Gaussian initialization for A and zero for B, so ΔW = BA is zero at the beginning of training.

2.2. DoRA-based speaker personalisation (Proposed 2)

In the DoRA method, we add an extra layer of a small set of parameters to the LoRA-based method. The weight matrix modified in a layer with the help of a new set of weight matrices as shown in Figure 1 (DoRA training) [19]. The weight decomposition of WϵR_d∗k can be formulated as:
W₀ = m ∗ V ||V ||_c = ||W||_c ∗ W ||W||_c (3)
where mϵR_1∗k is the magnitude vector, V ϵR_d∗k is the directional matrix, with ||.||_c being the vector-wise norm of a matrix across each column. This decomposition ensures that each column of V/||V ||_c remains a unit vector, and the corresponding scalar in m defines the magnitude of each vector. We initialize DoRA with pre-trained weight W₀ as outlined in Eqn 3 where m = ||W₀||_c and V = W₀ after initialization. We then keep V frozen andma trainable vector. The directional component is then updated through LoRA. DoRA can be formulated similar to Eqn 1 as:
W′ = m ∗ V + ΔV ||V + ΔV ||_c = m W₀ + BA ||W₀ + BA||_c (4)

Dataset used

WA portion of the LibriSpeech [23] test-other data was utilized to create the dataset for the adaption studies. Since testother contains somewhat noisy data and is more appropriate for real-world situations, we have chosen it over test-clean. The speaker-specific utterances are sorted in reverse order of the total number of utterances, and they are grouped together. Four presenters are selected from the top of the list. These four speakers’ audio data spans five to ten minutes, with 62 to 130 utterances. These speaker data sets are divided into test, valid, and train sets. Table 2 presents all of these specifics. The Table 1 provides the speaker id and total duration for each speaker. Testing of the model is performed on these speaker-specific test data(denoted as Spk X).

Experimental details

The LoRA and DoRA weights, as explained in Section 2, are applied to the attention layer of each of the 16 layers of the conformer blocks in the causal encoder. For each layer, the W0 matrix is of the size 256 x 256, since the encoder cell size is 256. The ΔW is decomposed into LoRA matrices. The total number of parameters(#Params) added in each LoRA experiment can be obtained using the equation 5; where N is the number of encoder layers, m is the number of matrices using LoRA weights in each encoder layer, r is LoRA rank and C is the encoder cell size(shown in Table 2).
#Params = N ∗ m ∗ (#Params of A,B) = N ∗ m ∗ (2 ∗ C ∗ r) (5)
Similarly for DoRA-based method, we can find the total number of parameters using the Equation 6; where N is the number of encoder layers, m is the number of matrices using DoRA weights in each encoder layer, r is DoRA rank and C I the encoder cell size(shown in Table 2)
#Params = N ∗ m ∗ (#Params of A,B, ||V ||) = N ∗ m ∗ (2 ∗ C ∗ r + C) (6)
There are two fine-tuned models considered when comparing with the proposed approach. In the first case, the entire model is fine-tuned using the speaker-specific data and is denoted as FT. In the second case, only the attention layer is fine-tuned(FTA) as mentioned in [7]. FTA has shown to perform really well with just fine-tuning the attention module.

Results and Discussion

To find the ideal value, a range of rank(r) values are explored with. Given that the encoder cell is 256 by 256, rank values to the power of 2 are selected for the experiments. Based on the original LoRA work [12], weight values for the Wq(query) and Wv(value) matrices of the attention layer are selected for this set of tests.

The experiments’ outcomes are displayed in Table 3. The model name is indicated in Table 3’s first column of table.

For each of the proposed methods, the number of trainable parameters in each of these tests is indicated in the first column; the WER for each speaker is displayed in the remaining columns. The average and relative WER for each speaker are displayed in the final two columns. Only the query and value matrices of the attention layer (2M parameters) are fine-tuned in the third model FTA:qv, in a manner similar to [7]. The suggested models are represented by the remaining rows; in order to determine the ideal rank, several rank values are tested. The number of parameters rises in tandem with the rank. For all rank values over 64, the number of trained parameters is more than 1M, or more than FTA:qv, . Thus, only rank values 1 through 32 are taken into account for comparison in order to ensure fairness.

This demonstrates that the baseline model, or Base, has an average WER of 14.75%. This model has no trainable parameters because it hasn’t been altered. The second model, denoted by the letter FT, is the fully adjusted model. Here, every one of the 27 million model parameters is optimized. The results demonstrate that this model performs the best. However, training 27M parameters on an edge device is difficult. For rank values between 4 and 32, we can observe that the suggested method performs better than FTA:qv. With an 22% relative improvement over the Base model, rank 8 outperforms the others, while the previous state-of-the-art method FTA:qv could only achieve a 14% relative improvement. Furthermore, the suggested method has only 139k trained parameters, whereas FTA:qv has 2M parameters. Our strategy is more effective when the suggested approach yields results with less than 10% trainable parameters of FTA:qv.

The LoRA method is able to achieve a relative improvement of 10% while the DoRA is capable of achieving 20% which is even better than the FTA:qv and very close to the fully finetuned model(FT). The experiments clearly shows the introduction of normalisation vector in DoRA helps in better performance. The relative WER results of DoRA is almost double to its LoRA counter parts keeping minimal difference in trainable parameters . This observation is similar to that of [19].

Conclusion

It had proven to be a challenging task to fine-tune ASR models that were customized for speakers. In this study, we have shown the effectiveness of low-rank adaptation-based approaches for model fine-tuning in a compute- and parameter-efficient way. With a very small amount of speaker data, the baseline model is improved upto 22% relative word error rate. The suggested approach works better than the prior cutting-edge technique found in [7]. The original model also can be maintained while using this strategy and with very little training infrastructure better accuracy can be achieved.