Introduction

Large Language Models (LLMs) have revolutionized natural language processing, offering unprecedented capabilities. However, deploying these massive models on consumer devices – such as smartphones and tablets – presents a significant challenge due to strict memory and compute constraints.

To bring LLMs to the edge, developers increasingly rely on Parameter-Efficient Fine-Tuning (PEFT) techniques, specifically Low-Rank Adaptation (LoRA) [1]. By keeping the base model frozen and injecting small, task-specific adapter modules, devices can support multiple personalized tasks (e.g., summarizing emails, translating languages, or drafting messages) without storing multiple massive models.

But as on-device AI evolves, a new challenge emerges: how do we manage these adapters when new tasks arrive continuously over time, and device storage is strictly limited?

In our paper, we introduce a novel, data-free framework that enables on-device LLMs to continuously acquire new capabilities without exceeding their storage limits or forgetting past knowledge.

The Challenge: Continual Learning Under a Strict Storage Budget

Imagine a smartphone that receives new, customized AI capabilities – represented by LoRA adapters – via over-the-air updates over several months.

In a traditional setup, model merging requires all task-specific adapters and any calibration data to be present at the exact same time. However, in a realistic on-device scenario, this is impossible:

1. Sequential Arrival: Adapters arrive online, one after another, rather than all at once.

2. Strict Storage Constraints: The device has a limited storage budget capable of holding only a maximum number of $K$ adapters.

3. No Access to Data: Due to privacy regulations and limited storage, the device does not have access to the data used to train the original adapters, meaning retraining or any calibration is not possible.

When the storage budget is full and a new adapter arrives, the device faces a critical problem. It cannot save the new adapter, and it cannot simply discard an old one without suffering from catastrophic forgetting (losing the ability to perform the old task).

Figure 1. Online continual merging of adapters for on-device LLMs. Each adapter corresponds to a specific task $τ_t$ (e.g., a specific problem type in a selected language). The objective is to increase the LLM capabilities over time without storing all adapters, but rather using a budget of $K$ adapters (e.g., $K$=4 here). The key steps are: 1) The device downloads new adapters over time. 2) The system selects the most similar stored adapter to the new one. 3) The system updates the selected stored adapter by merging it with the new adapter.

Our Solution: The K-Merge Framework

To overcome the described challenge, we developed K-Merge, a lightweight algorithm that decides dynamically which adapters to merge and how to merge them, entirely on-device and without relying on any data.

K-Merge operates through a highly efficient two-stage pipeline:

1. Distance-Based Adapter Selection

When a new adapter arrives and the device's $K$-adapter storage limit is reached, K-Merge evaluates the incoming adapter against the currently stored adapters. It calculates the distance between them in the weight parameter space to identify the "nearest neighbour" – the stored adapter that is most structurally similar to the new one. Merging similar adapters minimizes interference and preserves the functionality of both tasks.

2. History-Aware Dynamic Merging

Once the optimal pair is identified, K-Merge fuses their weights. Standard merging methods often use a simple average, which can dilute the capabilities of older tasks if an adapter has already absorbed multiple updates. Instead, K-Merge utilizes a merge history counter. It scales the merging coefficients based on the number of past tasks each adapter represents. This history-aware weighting ensures that older, foundational capabilities are preserved rather than overwritten by the newest update.

Future-Proofing with K-Merge++

We also introduce an advanced variant, K-Merge++. Instead of waiting for the storage budget to become exhausted, K-Merge++ applies a similarity threshold. If an incoming adapter is highly similar to an already stored adapter, K-Merge+ merges them immediately – even if there are empty storage slots available. This proactive approach reserves precious storage space for future adapters that might be unique or dissimilar.

Figure 2. K-Merge++ outline. When the storage budget is not fulfilled and a new LoRA $L^{(t)}$ is downloaded, it is compared with the set of stored LoRAs $L^{(t-1)}$. If similarity is smaller than the threshold $s$, $L^{(t)}$ initializes a new cluster; otherwise, it gets merged with the closest stored LoRA. After the budget limit has been reached, new LoRAs can only be merged. History of merges $H$ is maintained to keep track of which LoRAs support what tasks.

Experimental Results

We evaluated the K-Merge framework against various state-of-the-art baselines across multiple language tasks. Because K-Merge requires no data, it runs with exceptional efficiency on edge devices.

Our testing revealed several key insights:

Overall we see that K-Merge allows us to retain the vast majority of single-task precision while maintaining a strict, lightweight storage footprint, making it ideal for real-world deployment.

Figure 3. Normalized aggregate score $S^{(γ)}$ of compared methods at variable storage budget $K$. Averaged over 3 random task orderings.

Conclusion

The transition from static, cloud-based AI to dynamic, on-device intelligence requires models that can learn and adapt continuously. K-Merge provides a practical, scalable solution to the storage and privacy bottlenecks of on-device LLMs. By intelligently selecting and merging task-specific adapters on the fly, our framework ensures that edge devices can grow smarter over time without running out of memory or forgetting what they have already learned.