IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) is an annual flagship conference organized by IEEE Signal Processing Society. And ICASSP is the world’s largest and most comprehensive technical conference focused on signal processing and its applications. It offers a comprehensive technical program presenting all the latest development in research and technology in the industry that attracts thousands of professionals. In this blog series, we are introducing our research papers at the ICASSP 2025 and here is a list of them. #1. Evaluation of Wearable Head BCG for PTT Measurement in Blood Pressure Intervention (Samsung Reseach America) #2. Better Exploiting Spatial Separability in Multichannel Speech Enhancement with an Align-and-Filter Network (AI Center - Mountain View) #3. Vision-Language Model Guided Semi-supervised Learning for No-Reference Video Quality Assessment (Samsung R&D Institute India-Bangalore) #4. Text-aware adapter for few-shot keyword spotting (AI Center - Seoul) #5. Single-Channel Distance-Based Source Separation for Mobile GPU in Outdoor and Indoor Environments (AI Center - Seoul) #6. Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning (Samsung R&D Institute United Kingdom) #7. Find Details in Long Videos: Tower-of-Thoughts and Self-Retrieval Augmented Generation for Video Understanding (Samsung R&D Institute China-Beijing) #8. Globally Normalizing the Transducer for Streaming Speech Recognition (AI Center - Cambridge)

Introduction

Keyword spotting (KWS) is a technique for detecting pre-defined keywords in audio streams. Unlike fixed KWS [1]–[3], which requires users to exclusively use specific keywords, flexible KWS allows users to utilize any custom keyword. Custom keywords can be enrolled in flexible KWS either through audio [4]–[6] or text [7]–[11]. Since text-based keyword enrollment does not require multiple utterances of a target keyword and can be achieved easily via text input, the demand for Text-enrolled Flexible KWS (TF-KWS) is growing.

TF-KWS systems typically use a text encoder for enrollment and an acoustic encoder for testing, both of which are optimized using deep metric learning (DML) [12] objectives such as contrastive loss [9], triplet-based loss [7], and proxy-based loss [8], [11]. Specifically for TF-KWS, the text embedding (TE) is learned to act as a representative vector for its corresponding keyword, thus attracting the acoustic embedding (AE) of the same keyword and repelling AEs of different keywords in the shared embedding space.

While TF-KWS models can support an unlimited number of keywords, their performance does not match that of keyword-specific models trained with abundant data for each keyword [13]. Thus, there remains potential to enhance performance for specific target keywords. Given the challenge of collecting large amounts of data for a particular keyword, this problem can be approached through few-shot learning [14]. To address this, we propose a few-shot transfer learning approach called text-aware adapter (TA-adapter). To the best of our knowledge, this is the first work on applying few-shot transfer learning to TF-KWS.

Our research aims to adapt a small portion of the pre-trained acoustic encoder to a target keyword using limited speech data, leveraging the TE extracted from the corresponding text encoder. The TA-adapter consists of three main components: text-conditioned feature modulation (TCFM), feature weight adapter (FW-adapter), TE classifier. Due to its modular design, the TA-adapter enables seamless restoration to the original pre-trained model, facilitating rapid adaption to various target keywords. In our experiments, we evaluate performance of our method on the Google Speech Commands (GSC) V2 dataset [15] under noisy and reverberant conditions.

Proposed method

For the pre-trained model, we trained the acoustic and text encoders using Relational Proxy Loss (RPL) [11]. We employed the ECAPA-TDNN architecture [28] as the acoustic encoder.

Text-conditioned feature modulation (TCFM)

First, for the TA-adapter, we adopt TCFM to transfer the target keyword information from the TE into the pre-trained acoustic encoder. By freezing the text encoder, we extract a TE for the target keyword, which serves as a representative vector for its corresponding keyword. Fig. 2 highlights the difference between AdaIN [23]-based conditioning and TCFM. TCFM conditions the keyword information from the TE by learning a weighted combination of basic activation functions based on the TE, requiring significantly fewer parameters. We employ six activation functions from the set of basic activation functions in [24], selected based on their validation performance: ELU, hard sigmoid, ReLU, softplus, swish, and tanh.

Feature weight adapter (FW-adapter)

In addition to feature modulation, we aim to refine the weighting and aggregation process of features within the acoustic encoder by adjusting attention weights and activation distributions. We hypothesize that essential keyword features have already been effectively learned by the pre-trained acoustic encoder using extensive training samples. Therefore, when transferring information about the target keyword with limited samples, it is sufficient to adjust the aggregation of low-level features into higher-level ones by emphasizing feature importance, which can be achieved through squeeze-and-excitation (SE) and batch normalization (BN). As shown in Fig. 1, we adapt only the BN and SE modules, highlighted in green. Our experiments demonstrate that TCFM and the FW-adapter work synergistically, complementing each other.

Text embedding classifier (TE classifier)

The TE classifier process is illustrated by the red line in Fig. 1 (labeled as ‘TE classifier’). When TCFM and FW-adapter are applied simultaneously, an AE is extracted and then passed through the final fully-connected (FC) layer. A sigmoid activation function is then applied to the output. The acoustic encoder is adapted using binary cross-entropy loss. Instead of learning a new weight vector from scratch, we fix the TE as the weight vector in the final classification layer. This is reasonable because the TE has already been trained as a representative vector for its corresponding keyword during the pre-training phase using DML.

Experiments

Experiment setup

The pre-training strategy followed the same approach as our previous work in [11], including the use of identical datasets, acoustic features, and model architectures. For the TA-adapter, we used the GSC V2 dataset [15] containing 35 keywords. Compared to the training set used for pre-training [33], 10 of these keywords were seen during pre-training, while 25 were unseen (as shown in Table 1). We evaluated the model under three low-resource scenarios: 5-shot, 10-shot, and 15-shot learning. For each keyword, we developed three separate models by randomly selecting 5, 10, and 15 samples, respectively, for each scenario. To develop a keyword-specific model, we fine-tuned the pre-trained model for each keyword individually. The official validation and test sets of GSC v2 were employed for model selection and evaluation. Table 1 shows the average counts of positive and negative samples for the corresponding keywords. To simulate real-world conditions, we generated noisy and reverberant speech by convolving synthetic room impulse responses (RIRs) from the OpenSLR dataset [34] and adding noise from the MUSAN dataset [35], with signal-to-noise ratios (SNRs) ranging from 5 to 25 dB.

Results

Table 2 presents an ablation study that evaluates the effectiveness of the FW-adapter and the TE classifier. For simplicity, the experiment focuses on a 15-shot scenario with five keywords from both seen and unseen keywords: 1) Seen keywords: ‘follow’, ‘happy’, ‘house’, ‘one’, and ‘seven’; 2) Unseen keywords: ‘cat’, ‘dog’, ‘eight’, ‘nine’, and ‘off’. The table reports the average AP (%) values for both seen and unseen keywords, as well as the average between them (‘Avg.’). ‘PT’ and ‘FT’ refer to the pre-trained and fully fine-tuned models without the adapter, respectively. ‘FT clf’ represents the model where the network is frozen, and only an additional classifier (clf) is finetuned. For ‘PT’, the score is obtained from the cosine similarity between AE and TE, indicating the use of TE classifier.

Interestingly, ‘FT (15-shot)’ performs worse (67.04%) than ‘PT’ (72.02%), as 15-shot samples are insufficient for fine-tuning all the parameters. However, when using all available samples for finetuning (FT (full-shot)), the model achieves remarkable performance (95.35%), although this is impractical due to the excessive cost of data collection. Table I shows the average number of training samples. Freezing the model and fine-tuning only an additional classifier (FT clf) yields poor performance (50.14%). Comparing ‘PT’ and ‘FT clf’, the TE classifier clearly boosts few-shot KWS performance.

We evaluate performance by individually or collectively adapting the BN layers in each group. Hereafter, we omit mentioning ‘(15- shot)’ in the method, but all methods continue to use 15-shot samples. Regardless of the adaptation location within the ECAPA-TDNN, BN adaptation consistently outperforms the pre-trained model for both seen and unseen keywords, validating our hypothesis that adjusting feature weights at each layer enables successful adaptation to a target keyword. Adopting BN adaptation across all groups (‘BN’) yields better results (79.40%) than fine-tuning individual groups, indicating that the gains from individual group adaptations are complementary and cumulative. Combining SE and BN adaptations yields further improvements. However, contrary to BN, applying SE adaptation to all groups (‘SE & BN’) performs worse compared to applying it individually to each group. We hypothesize that this could be attributed to the excessive number of additional parameters required for few-shot KWS. The best performance is an AP of 84.33%, achieved with ‘SE G3 & BN’, without adding any extra parameters.

Table 3 presents the ablation results on TCFM, where ‘FW-adapter’ corresponds to ‘SE G3 & BN’. Conditioning keyword information at the lower layers (i.e., G0 to G3) degrades performance. We suspect that directly modifying their features with limited samples impairs performance, as these early layers generate fundamental features that eventually form keyword-specific representations. In contrast, the FW-adapter merely adjusts feature aggregation, enabling effective adaptation at any layer and improving performance compared to the pre-trained model. Our results suggest that applying TCFM to higher layers is suitable for conditioning TE, consistent with AdaKWS [22], where KAMs are inserted just before the final classifier. Optimal performance is achieved when applying TCFM to G4 and G5, boosting the AP of TF-KWS (i.e., ‘PT’ in Table 2) from 72.02% to 87.22%. Although TCFM slightly increases the total number of parameters (by 3.2K, representing 0.14% of the original count), the performance gain is substantial.

Finally, Table 4 compares the performance of the TA-adapter with other baseline approaches using all 35 keywords. ‘TA-adapter’ corresponds to the best-performing model from Table 3. Across all scenarios, the TA-adatper consistently outperforms the baseline systems, including TF-KWS models (‘AdaMS’ and ‘RPL’) and few-shot learning approaches (‘2-class clf’, ‘3-class clf’, and ‘AdaKWS’). Notably, the performance gap widens under lower resource conditions.

Conclusions

This paper introduces the TA-adapter for addressing the few-shot transfer learning problem in TF-KWS. The TA-adapter utilizes the text embedding to condition keyword information into the acoustic encoder and to generate a keyword score. Also, BN layers and SE modules are adapted using only a few samples of the target keyword. Experimental results demonstrate that the TA-adapter effectively overcomes the challenges of few-shot KWS. Specifically, the TA-adapter boosts the pre-trained model’s average precision from 77.93% to 87.63% using just 5 target samples. Since the TA-adapter only modifies a small subset of the model, it enables seamless reversion to the original pre-trained model. In future work, we plan to leverage text-to-speech (TTS) technology to generate synthetic data, enabling zero-shot transfer learning for TF-KWS.

Link to the paper

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10890609

View More

Close