1. Introduction

Modern smartphones quietly run a surprising number of machine learning models. Voice assistants classify what you meant, messaging apps flag spam, keyboards sense sentiment, and organizer apps deduplicate entries. Each of these features has traditionally shipped with its own specialized model. Keeping this processing on the device is essential for privacy, network-independent latency, and energy efficiency, but it creates a problem that grows with every new feature: the combined storage and memory footprint of many task-specific models.

In our Interspeech 2026 paper, we asked a simple question with a practical payoff: can a single lightweight architecture solve multiple speech-adjacent classification tasks? Our answer is AnySimLite, a similarity encoder that runs in about 700 KB of storage and under 30 ms per inference on a flagship smartphone. Combined with a novel dataset transformation strategy, it matches or closely approaches state-of-the-art results on a diverse set of language classification tasks in a few-shot setting, even when compared against a 7-billion-parameter large language model (LLM) baseline.

In this post, we walk through the idea behind the model, the key design choices, and what the results mean for on-device AI.

2. Background: many tasks, one underlying question

A large share of on-device language features operate on the output of automatic speech recognition (ASR) or on other user text. Intent detection in a voice assistant, spam filtering, sentiment classification, topic tagging, and toxicity screening are, at their core, classification of a piece of text into known categories. We call these speech-adjacent (SA) classification tasks.

Text similarity, the task of deciding whether two pieces of text are “similar”, is a classic problem in natural language processing (NLP). It is usually interpreted lexically (shared tokens) or semantically (shared meaning). But similarity can be defined more subtly. Two product reviews can be “similar” because they express the same sentiment, even if they describe completely different experiences. Two emails can be “similar” simply because both are spam. We refer to this generalized notion as nuanced text similarity (NTS): similarity judged on a task-specific feature alignment rather than on overall lexical or semantic closeness.

This reframing is powerful because it turns classification into comparison. Instead of designing a new architecture for every task, we can keep a small set of pre-annotated example texts (exemplars) per class, and classify a new input by asking: which class’s exemplars is this input most similar to? With around 20 exemplars per class, each task is served by a compact encoder built on one common architecture, and the class knowledge lives in a handful of tiny embeddings (Figure 1).

Figure 1. Reducing diverse on-device classification tasks to one nuanced text similarity task (similarity scores shown are illustrative).

3. Methodology

3.1 A toy problem that forces the right design

To design the architecture without biasing it toward any single downstream task, we developed and validated it on a deliberately chosen toy problem: Event Title Similarity. The toy problem serves purely as a design testbed; the selected architecture is then trained separately for each downstream task, as Section 3.3 describes. Two event titles (for example, calendar entries) are similar if and only if they describe the same type of event and involve the same people. “Birthday party for John” is similar to neither “Meeting with John” (different event) nor “Sarah’s birthday party” (different person).

This problem has two properties that make it an excellent testbed. First, the similarity constraint is non-obvious: it is neither purely lexical nor purely semantic, exactly like the nuanced similarity we ultimately care about. Second, names of people are effectively open-vocabulary, so any architecture that succeeds must handle out-of-vocabulary (OOV) named entities, a persistent weakness of compact word-embedding models. We curated a dedicated dataset, TitleSimCurated, for this problem using hand-crafted templates together with a pretrained LLM under prompt engineering, spanning 14 everyday event categories.

3.2 Designing the architecture

Our first formulation concatenated the two titles into one string and trained a binary classifier. It worked moderately well, but it has a subtle flaw: text similarity is commutative, meaning the score for (A, B) must equal the score for (B, A), and a classifier over concatenated strings does not guarantee this. This pushed us to an encoder design: each text is independently encoded into an embedding, and similarity is the cosine similarity of the two embeddings. As a bonus, the encoder formulation slashes inference complexity: populating a database of n titles drops from O(n²) pairwise checks to O(n) encodings, and checking one new title becomes O(1).

The encoder itself combines two channels (Figure 2). A word channel (word embeddings feeding a bidirectional long short-term memory (BiLSTM) layer, followed by an attention layer) captures compositional and semantic structure. A character channel (character embeddings feeding a one-dimensional convolution (Conv1D) layer with global max pooling) catches spelling-level differences. The character channel is what lets a 0.42M-parameter model notice that “John” and “Sarah” are different people even when both names are outside its word vocabulary. The channel outputs are concatenated, projected through a dense layer, and L2-normalized into a compact 16-dimensional embedding.

Figure 2. Inside the AnySimLite encoder (left) and scoring a pair of texts with shared encoder weights (right).

Our ablation study on TitleSimCurated guided these choices. Attention on the word channel proved to be the single most valuable component, ahead of variants with character-side or cross-channel attention: identifying which word tokens matter is the key discriminative skill. The study also steered us away from two tempting alternatives. A Siamese network with triplet training depends on mining anchor and negative examples that are dissimilar but not too dissimilar, which requires hand-crafted domain knowledge for every task and does not suit a general-purpose recipe. Origin clustering, which treats each event-plus-participants combination as its own class, held up only when the test distribution mirrored the training distribution. Finally, applying knowledge distillation from a MiniLM-L12-v2 teacher [2] to the selected base model yielded our deployment variant: 0.72M parameters, and the best overall scores on the toy problem (90.8% accuracy, 90.75 F1).

3.3 Turning classification datasets into similarity datasets

Reducing a task to NTS requires transforming its labeled classification dataset into labeled pairs of documents. Random pairing turns out to be a trap: most randomly drawn dissimilar pairs are “too dissimilar”, and a model trained on them never learns the task-specific nuance (for sentiment analysis, for instance, it must learn that what matters is sentiment, not topic, length, or vocabulary).

Our transformation strategy (Figure 3) mines “hard” pairs instead. We embed all documents with a pretrained language model (PLM), cluster the embeddings with the density-based clustering algorithm DBSCAN [3], and then sample pairs both within and across clusters, drawing dissimilar pairs intra-cluster versus inter-cluster at a fixed 8:2 ratio. Intra-cluster pairs with different labels are precisely the valuable hard negatives: the two texts share surface characteristics (that is why they landed in one cluster) yet differ in the one nuance that defines the task. Their mirror image, inter-cluster pairs that nonetheless share a label, supplies hard positives. Compared with token-matching (which needs O(N²) comparisons) or term-frequency weighting schemes like TF-IDF (which need a large term-document matrix), the PLM-plus-DBSCAN route is also computationally economical.

Figure 3. Transforming a classification dataset into a labeled pair dataset for similarity training.

4. Results

4.1 One recipe, many tasks

We evaluated AnySimLite in a few-shot setting (20 exemplars per class) on nine datasets spanning six task families: text similarity (TitleSimCurated, Quora Question Pairs), sentiment classification (Sentiment-140, IMDB), intent detection (SNIPS, ATIS), spam detection (SMS Spam Collection), topic classification (AG News), and toxicity detection (Toxic Comment). Across these benchmarks, AnySimLite consistently reaches state-of-the-art (SOTA) results or stays SOTA-competitive while being, in all but one case, the smallest model in its comparison group (Figure 4):

Figure 4. AnySimLite versus the strongest baseline on each dataset. Blue bars report accuracy; the violet pair after the divider reports ROC-AUC for Toxic Comment. Labels under each dataset compare model sizes (AnySimLite versus baseline; n/r means the baseline size is not reported). Baselines are the best published results, except on our own TitleSimCurated dataset, where we evaluated the baselines ourselves.

4.2 What it costs on a smartphone

Benchmarks aside, the deployment numbers are the point of the exercise. On a Samsung Galaxy S25 Ultra, the 8-bit quantized deployment model occupies roughly 700 KB on disk and runs inference in under 30 ms. The few-shot exemplars are stored as precomputed 16-dimensional embeddings, occupying about 320 bytes per class in a float8 format. Each task carries its own compact model built on the common architecture, so supporting J tasks costs about J × 700 KB of storage plus these negligible exemplar embeddings. A dozen tasks together occupy well under 10 MB.

5. Discussion

The broader lesson we take from this work is that problem reduction can be a resource-optimization tool. Rather than compressing each of a dozen models individually, one can reduce a family of problems to a common core task and spend the entire optimization effort on one small, well-designed architecture for that core. Our results also add to the growing evidence that sub-million-parameter models remain genuinely competitive with much larger models on many classical NLP tasks. Across the benchmarks in our paper, the strongest baselines are between roughly 100 times (BERT-scale models) and over 100,000 times (a 175-billion-parameter GPT-3) larger than the AnySimLite instance they compete with, for accuracy gaps of a few points at most. That trade is often the difference between a feature shipping on-device and not shipping at all.

As with any compact model, there are trade-offs to consider. Tasks whose decision boundary depends on long-range document structure, such as full-length reviews, or on fine-grained multi-label distinctions, such as toxicity subtypes, show the largest gaps to large pretrained models. Our few-shot protocol also inherits sensitivity to exemplar quality. The natural next step is exploring how far this reduction recipe extends beyond classification.

6. Conclusion

AnySimLite demonstrates that a carefully designed sub-megabyte encoder, combining two channels with one attention layer and optionally distilled from a teacher model, can replace a fleet of task-specific on-device models for speech-adjacent classification. A single shared architecture, a principled hard-pair dataset transformation, and 20 exemplars per class are enough to be SOTA-competitive across multiple task families at a fraction of the memory cost. We hope this recipe helps bring richer language understanding to everyday devices.