Introduction

Speech-to-text (STT) models are usually trained on wideband (WB) audio, but narrowband (NB) audio remains important for mobile and telephony use cases. Training separate models for WB/NB inputs and different tasks like ASR and speech translation (ST) is resource-intensive, especially on mobile devices. Multi-task learning (MTL) can help by sharing a single model across tasks, but traditional hard-parameter sharing often leads to task interference.

To address this, we propose Supervised Mixture of Experts (S-MoE) - a modular approach inspired by MoE but without the need for complex gating functions. Instead, it uses guiding tokens to route each task to its designated expert network, making the model efficient and scalable. (see Figure 1)

Figure 1. Two S-MoE layer embedded within a STT model. The S-MoE in the encoder directs inputs to expert networks based on the speech bandwidth (NB or WB), while the S-MoE in the decoder routes inputs according to the task (ASR or ST)

Key contributions:

Approach

We propose the Supervised Mixture of Experts (S-MoE) architecture, which modifies the traditional Mixture of Experts (MoE) by replacing the learnable gating network with a predefined gating function. This simplifies training by eliminating the need for gating network learning.

Figure 2. Embedding flow of S-MoE Transformer. During training/inference, a single feedforward block within the encoder is chosen, whereas both feedforward blocks within the decoder are utilized

Embedding Flow: Special tokens indicating the task ( or ) and target language ( or ) are inserted at the start of each sequence to guide the model during training and inference. (see Figure 2)

Experimental setup

Datasets: Training uses 40,000 hours of Korean speech from the AIHub corpus. ASR targets are transcripts; ST targets are either provided English translations or generated via an internal machine translation model. Both narrowband (NB, 8 kHz) and wideband (WB, 16 kHz) audio variants are used. NB data is created by downsampling and codec-based conversions applied to 15% of the training data for fine-tuning.

Evaluation uses two in-house test sets (1,000 daily conversation samples each) and the public Fleurs dataset, with NB/WB variants generated similarly to training.

Model: Transformer-based sequence-to-sequence with 12-layer encoder and 6-layer decoder. We use sinusoidal positional embeddings, GLU activations, 8 attention heads, FFN dimension 2048, embedding size 512, and 0.15 dropout. Input features are 80-dim log Mel filterbanks extracted from 16 kHz resampled audio. Tokenization uses byte-level byte-pair encoding (BBPE) with 40,000 vocab size.

Training runs for 10 epochs with batch size 3,200, learning rate decayed by cosine annealing.

Encoder-Decoder S-MoE model is initialized from Decoder S-MoE and fine-tuned for 5 epochs with the NB/WB fine-tuning subset.

Metrics: ASR performance is evaluated by Word Error Rate (WER). ST quality is measured using BLEU score with the sacrebleu library.

Experimetal results

Table 1. Impact of Decoder S-MoE on ASR/ST

Decoder S-MoE (DecS-MoE) consistently outperformed the baseline, showing:

→ Average 8.06% relative improvement in WER and 1.77% increase in BLEU across test sets.

→ Better results than DecFFNx2, proving that task-specific experts outperform just increasing model size.

→ Outperformed Whisper-large (much bigger model) with 21.98% relative WER reduction and 3.99% BLEU improvement, despite having only 1/10th of the parameters.

Table 2. Impact of Encoder-Decoder S-MoE on ASR/ST

Fine-tuned on mixed bandwidth data, EncDecS-MoE achieved:

→ 6.35% relative improvement in WER and 1.63% increase in BLEU on NB data.

→ 2.39% relative improvement in WER and 1.15% increase in BLEU on WB data.

→ Results show separate FFNs in the encoder effectively address challenges of mixed bandwidth data without increasing inference cost.

Extended S-MoE to both encoder and decoder to handle mixed bandwidth (NB/WB) inputs and multitask learning. Encoder uses two experts for NB and WB audio; decoder uses two for ASR and ST, totaling four experts. Although the total parameter count is larger, only relevant experts are activated during inference, keeping active parameters equal to the baseline.

Conclusion

This paper proposes the S-MoE architecture to reduce task interference in multi-task learning for speech-to-text applications. By replacing dynamic gating with guiding tokens, S-MoE improves training efficiency and performance while handling ASR and speech translation with mixed-bandwidth inputs. This makes it well-suited for resource-limited environments. Future work aims to extend S-MoE to more speech tasks and multilingual support for broader real-world use.

View More

Close