Interspeech is the world’s leading conference on the science and technology of speech recognition, speech synthesis, speaker recognition and speech and language processing. The conference plays a crucial role in setting new technology trends and standards, as well as providing direction for future research. In this blog series, we are introducing some of our research papers at INTERSPEECH 2024 and here is a list of them. #1. Relational Proxy Loss for Audio-Text based Keyword Spotting (Samsung Research) #2. NL-ITI: Probing optimization for improvement of LLM intervention method (Samsung R&D Institute Poland) #3. High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model (Samsung Research) #4. Speaker personalization for automatic speech recognition using Weight-Decomposed Low-Rank Adaptation (Samsung R&D Institute India-Bangalore) #5. Speech Boosting: developing an efficient on-device live speech enhancement (Samsung Research) #6. SummaryMixing makes speech technologies faster and cheaper (Samsung AI Center - Cambridge) #7. A Unified Approach to Multilingual Automatic Speech Recognition with Improved Language Identification for Indic Languages (Samsung R&D Institute India-Bangalore)

Introduction

Keyword spotting (KWS) aims at identifying pre-specified keywords in audio streams. Recently, KWS has attracted considerable research attention with the popularity of voice assistants that are triggered by keywords such as “Alexa,” “Hi Bixby,” or “Okay Google.” KWS techniques can be broadly classified into two categories: fixed KWS [1-3] and flexible (or user-defined) KWS [4-13]. Unlike fixed KWS where users are required to utter fixed keywords exclusively, flexible KWS allows users to enroll and speak arbitrary keywords. Specifically, in flexible KWS, user-defined or customized keywords can be enrolled either in the form of audio [4-7] or text [8-13]. Because the text-based keyword enrollment does not require multiple utterances of a target keyword, but can be easily achieved through text typing, the demand for text-enrolled flexible KWS is increasing.

Text-enrolled flexible KWS systems [8-13] typically utilize a text encoder during the enrollment phase and an acoustic encoder during the test phase, where both encoders are optimized employing deep metric learning (DML) [14] objectives such as contrastive loss [12], triplet-based loss [8], and proxy-based loss [9,10]. Particularly in text-enrolled flexible KWS, acoustic and text embeddings representing the same keyword must become closer, while those representing different keywords must not. Considering that the acceptance or rejection of a keyword is determined via measuring the similarity between audio and text inputs, we refer to this task as audio-text based KWS.

In this blog post, we focus on the DML loss function and propose Relational Proxy Loss (RPL). We leverage structural relations of acoustic embedding (AE) and text embedding (TE) motivated by Park et al.'s work [15]. We incorporate this concept in the context of DML by employing the RPL, which utilizes the relational information within AEs and TEs. Similar to [9], we treat TEs to proxies. The primary information of proxies representing the corresponding word classes is well-presented by the structural relations, so we can assume that the AEs belonging to the corresponding classes are expected to follow the same relations. Fig. 1 illustrates the distinction between (a) traditional DML-based approaches and (b) our proposed RPL. While conventional DML losses compare an AE and a TE by computing their similarity (point-to-point), the proposed RPL compares structural relations within the embeddings (structure-to-structure). Through the implementation of RPL, the structural relations of AEs are adjusted and brought closer to those of TEs, thereby enhancing the comparability between AE and TE. Specifically, we compute distance- and angle-wise relations for both AE and TE independently. We then combine the point-to-point and structure-to-structure approaches, yielding better overall system performance than either approach alone.

Proposed method

Motivated by [9], we treat TEs as proxies representing distinct word classes because each TE is generated for a specific word class. The primary goal of the Relational Proxy Loss (RPL) is to transfer structural information from TEs to AEs by exploiting the relationships among TEs. By applying the RPL, we train the acoustic encoder to adopt the same relational structure as that of the text encoder through a relational potential function ϕ that quantifies the relational energy of the given n-tuple.

First, we establish two types of potential functions: a distance-wise function ϕ_D using pairwise examples, and an angle-wise function ϕ_A using ternary examples. Based on ϕ_D and ϕ_A, we define distance-wise RPL (RPL-D) and angle-wise RPL (RPL-A), respectively. Both are visualized in Fig. 2(a) and (b), respectively. RPL-D can be formulated by using Huber loss [17] with a scaling factor 1 that aims to minimize the difference of the relational energies measured via ϕ between the text and acoustic encoders. Through this method, the acoustic encoder is guided towards focusing on the distance structures of AEs. An angle-wise potential function ϕ_A calculates the angle created by the three instances in the embedding space. To simplify computation, we opt for calculating the cosine value of the angle, which can be represented using an inner product, rather than directly obtaining the angle value. The RPL-A transfers relational knowledge about angles from the text encoder to the acoustic encoder, encouraging the acoustic encoder to focus on the angle structures of AEs. Finally, we introduce an extra kind of distance-based RPL that makes use of prototypes (i.e., class centroids). Adopting the concept from [18], we define the prototype of the AE for the class k. As shown in Fig. 2(c), the RPL-P transfers distance information between proxies (i.e., TEs) to the distance between the AE and its corresponding class centroid.

In the above, it is assumed that the classes of TEs are distinct from one another, as this is commonly observed within a mini-batch setting. When the classes coincide, the RPLs behave similarly to other types of losses. For example, RPL-D resembles the cross-entropy loss used for AE classification, since RPL-D pulls a_i and a_j closer together due to ϕ_D (t_i,t_j) being equal to zero. Also, RPL-P is akin to the Prototypical loss [18] if the class label y_i matches k and the value of ϕ_D (t_i,c_k^t) becomes zero. As such, the AE moves towards its corresponding centroid because of the term ϕ_D (a_i,c_k^a).

Experiments

1. Model

For the acoustic encoder, we adopted the ECAPA-TDNN [27] with 256 filters in the convolutional layers (2.2M parameters). After statistics pooling, a 512-dimensional AE was obtained. In the text encoder, we incorporated a pre-trained byte pair encoding (BPE) [28] tokenizer and a two-layer bi-LSTM with 512 hidden units (10M parameters). We used the implementation of a character-level tokenizer available at HuggingFace [29] and trained the model with a vocabulary of 100 tokens on an internal dataset consisting of 50M English texts. The tokenizer split the input text into a sequence of subword units, and then the resulting sequence was transformed into a 1024-dimensional vector sequence by a trainable look-up table. The 1024-dimensional vector sequence was fed into the following bi-LSTM. At each time step, the outputs of the bi-LSTM were concatenated and then passed through a global average pooling, followed by a fully connected layer, producing a 512-dimensional TE. The design choices for these architectures were determined based on the computational resources available for deployment on our target devices.

2. Experiment setup

We used the King-ASR-066 [19] dataset which contains recordings from 2.6k native English speakers for training. Each sample was recorded at a 16 kHz sampling rate in home/office environments. All utterances were divided into word-level segments by forced alignment using the Montreal Forced Aligner [20], and we made use of 4.6k hours of speech data. The training set comprised phrases containing one to five words, with a total of 211,676 classes. For data augmentation, we convolved the clean speech signals with synthetic room impulse responses (RIRs) from the OpenSLR dataset [21] and added various kinds of noise, such as babble, car, and music, at randomly selected signal-to-noise ratios (SNRs) between -3 and 25 dB. To evaluate the performance of all systems under consideration, we used the Wall Street Journal (WSJ) [22] corpus, acquired using the WSJ recipe available within the Kaldi toolkit [23]. To simulate real-world scenarios, we generated noisy and reverberant speech by convolving synthetic RIRs from the OpenSLR dataset and adding the MUSAN noise dataset [24] with SNRs varying from 5 to 25 dB. During evaluation, we randomly selected 1.3M positive and 13M negative pairs from the test set. We measured the model performance using two metrics: the Equal-Error-Rate (EER) [11-13] and the Average Precision (AP) [8,9,25,26], both of which are commonly used in KWS.

3. Results

We use ‘P2P’ and ‘S2S’ to represent ‘point-to-point’ and ‘structure-to-structure’ methods, respectively, as shown in Fig. 1. To ensure a fair comparison, all systems share the same experimental setup, including datasets and model architectures, but differ only in their training criteria.

First, we conduct an ablation study to verify the effectiveness of each RPL loss, as presented in Table 1. In this case, we designate P2P as the AdaMS method, combining the Adaptive Margin and Scale (AdaMS) into the AsyP loss. This serves as the baseline in the table, showing 71.9% of Average Precision (AP). We observe that three RPL losses (i.e., RPL-D, RPL-A, and RPL-P) improve the performance of the P2P method. In summary, Integrating all RPL losses leads to a noticeable improvement in the performance of the ‘P2P’ method, resulting in an AP of 79.4% with a relative improvement (RI) of 10.43% compared to the baseline. Additionally, we investigate the performance when only three RPL losses are employed without P2P. As a result, the performance declines, yielding 66.8% of AP. Thus, based on these findings, we can conclude that employing both P2P and S2S demonstrates the better performance than either approach alone, and using all four losses shows optimum performance among all systems presented in the table.

Table 2 shows the effects of auxiliary losses (AL) on the performance of our proposed system. Here, ‘RPL’ refers to our system using P2P with three RPL losses in Table 1. The initial loss is referred to as the prototype-centroid matching loss, L_pc. It is computed based on the centroids representing the average Euclidean distance between each prototype (i.e., TE) and its corresponding centroid of AE within a mini-batch. The losses discussed so far are all related to the relationship between AE and TE, regardless of whether it is P2P or S2S. However, we find that incorporating losses from the speech-enrolled KWS approach is beneficial for the text-enrolled KWS as well. Specifically, we introduce losses from [7], consisting of two auxiliary losses, namely L_mono and L_triplet, exclusively associated with the acoustic encoder.

In Table 3, we compare the performances of our proposed systems ‘RPL’ and ‘RPL+AL’ with those of other state-of-the-art methods. It is evident that RPL outperforms the baseline systems, and the addition of auxiliary losses leads to further improvements. ‘RPL+AL’ attains a relative improvement of 22.96% in terms of EER compared to PATN and 15.71% in terms of AP compared to AsyP, respectively.

Conclusions

We have introduced the Relational Proxy Loss (RPL) for audio--text based keyword spotting (KWS). Unlike previous works using deep metric learning that only focused on comparing acoustic embedding (AE) and text embedding (TE) in a point-to-point manner, RPL exploits the structural relations within AEs and within TEs with respect to distance and angle. Specifically, we introduced three variants of RPL, namely RPL-D, RPL-A, and RPL-P. Our experiments demonstrated that combining point-to-point and structure-to-structure approaches led to better performance. Further improvement was achieved by incorporating auxiliary losses. On the noisy and reverberant WSJ test set, the proposed RPL and RPL+AL outperformed existing techniques, including both speech enrolled KWS and text enrolled KWS methods. In the future, we will focus on audio-text alignment and try to integrate our RPL into an audio-text alignment framework that predominantly adopts a naive loss function.

Link to the paper

https://arxiv.org/pdf/2406.05314