1. Introduction

The rapid advancement of Large Language Model (LLM) capabilities has been accompanied by a significant increase in model parameter scale. For autoregressive LLMs, efficient inference is primarily constrained by memory bandwidth rather than raw computational capacity, as serving these models requires memory-intensive transfers of model parameters for every generated token. Consequently, model compression is no longer merely an optimization preference, but a strict prerequisite for deploying models on physical hardware instead of confining them to research environments.

While 4-bit quantization has become the industry standard, the research frontier has progressed to the more challenging 2-bit regime, where weight representation is restricted to a quarter of a byte. In this domain, existing methodologies present a critical trade-off: techniques that preserve model accuracy require complex, hardware-unfriendly architectures, whereas methods designed for high-throughput execution on physical chips suffer from significant degradation in accuracy.

Our paper introduces RaBiT (Residual-Aware Binarization Training), a quantization-aware training framework designed to resolve this trade-off. RaBiT delivers state-of-the-art 2-bit accuracy while maintaining the highly efficient, multiplication-light structure characteristic of binary models. In principle, this architecture facilitates fully matrix-multiplication-free (matmul-free) inference as hardware support matures.

This article provides a technical overview of the principles underlying RaBiT. We detail the engineering motivations, analyze the core architectural bottlenecks uncovered during development, explain the framework's methodology, present performance benchmarks, and discuss the practical deployment challenges and engineering lessons encountered during implementation.

2. Background

From post-training tricks to training-aware compression

Model quantization methodologies are broadly categorized into two paradigms. Post-Training Quantization (PTQ) compresses a fully trained, static model by mapping its weights to lower precision after the completion of the training phase. PTQ is computationally efficient and widely adopted for 3-bit to 4-bit configurations; indeed, algorithms such as GPTQ[1] and AWQ[2] have established themselves as industry standards for 4-bit precision. However, at ultra-low precisions such as 2-bit, the substantial quantization error and information loss cannot be sufficiently mitigated by advanced rounding heuristics alone. Alternatively, Quantization-Aware Training (QAT)[3,4,5] models the low-precision constraints during the fine-tuning process, allowing the model parameters to dynamically adapt to the reduced numerical representations. RaBiT is classified within the QAT framework, utilizing knowledge distillation[6] to transfer dark knowledge from a high-precision teacher model.

Two roads at the 2-bit frontier

At the 2-bit threshold, contemporary research is primarily divided into two architectural approaches. Vector Quantization (VQ) methods[7,8,9] achieve high representation accuracy by clustering and mapping groups of weights onto optimized, learned codebooks. However, this accuracy is gained at the expense of computational overhead, introducing lookup tables and complex coordinate rotations that run inefficiently on conventional hardware. Conversely, residual binarization[10,11,12,13] utilizes a cascading structure of binary (±1) layers. In theory, this allows the forward pass to be computed using highly efficient addition and subtraction operations, bypassing dense matrix multiplications entirely. While this structural paradigm offers significant potential for hardware-accelerated efficiency, achieving complete, matmul-free execution remains dependent on specialized hardware architectures. Furthermore, residual binarization models have historically suffered from training instability and degraded downstream performance.

This performance gap is fundamentally linked to the limitations of residual designs. Ideally, each successive residual path should systematically minimize the approximation error propagated by preceding layers. In practice, however, this objective is impeded by feature co-adaptation[14]. Rather than learning orthogonal or complementary features to divide the optimization task, parallel components tend to converge on redundant representations. Consequently, stacking additional residual layers fails to compensate for the quantization error, leading to underutilized parameter capacity.

3. Methodology

The failure mode: inter-path adaptation

In standard residual QAT, independent binary paths are updated simultaneously by a single, global gradient. Lacking structural constraints, this shared signal drives the paths to converge on redundant features-a phenomenon we term inter-path adaptation. This failure mode is characterized by path correlations drifting toward or exceeding zero, indicating that the secondary path does not compensate for the primary path's error. Mathematically, minimizing the training loss reveals that true error cancellation is contingent upon negative correlation, strictly requiring the secondary path to align with the residual of the first.

Figure 1. Standard QAT versus RaBiT's coupled training. Independent paths under a shared global gradient learn redundant features (weak error cancellation), whereas deriving the second path from the first path's residual forces genuine error compensation.

The fix: coupled training from one shared weight

To resolve this issue, RaBiT maintains a single shared full-precision weight from which binary paths are sequentially derived. The first path is defined as the sign of the shared weight, while the second path is constructed by binarizing the residual error of the first. Defining each path relative to its predecessor's residual mathematically guarantees an error-correcting hierarchy by design. Per-channel scaling factors remain learnable to modulate path magnitudes, and gradients are backpropagated through the sign operations via the Straight-Through Estimator (STE) [15].

A key advantage of this unified weight representation is that it reduces optimizer state memory (e.g., momentum) by approximately 50%, directly mitigating a critical memory bottleneck during LLM fine-tuning. At inference, the shared weight is discarded, and the frozen binary paths are executed fully in parallel.

Figure 2. Overview of the RaBiT training framework: binary paths are dynamically derived from a shared weight to enforce a residual hierarchy (left); at inference the paths execute in parallel - a structure amenable to matmul-free computation (center); gradients update both the learnable scales and the shared weight (right).

Getting the starting point right

Extreme quantization is notoriously sensitive to initialization - a bad start can trap the model in a poor solution it never escapes. RaBiT uses a two-part, function-aware initialization. First, an iterative residual decomposition repeatedly re-fits each path to the other's leftover error (rather than greedily fixing the first path once and moving on), which converges within roughly 20 iterations. Second, and more counterintuitively, we deliberately bias the decomposition toward functionally important channels – those with large activations or gradients – even though doing so increases the raw weight-reconstruction error. The reasoning: at 2 bits, matching the model's behavior matters far more than matching the exact weight values.

4. Results

We evaluated RaBiT across the Llama2, Llama3, Gemma3, and Qwen3 model families, measuring perplexity, zero-shot reasoning accuracy, and physical inference throughput. Our experimental results yield three primary insights:

Figure 3. End-to-end decoding throughput (tokens/second) for Llama2-7B on an RTX 4090 across generation lengths. RaBiT consistently outpaces other 2-bit methods by relieving the memory-bandwidth bottleneck and avoiding codebook/rotation overhead.

Ablation studies validate the individual contributions of RaBiT’s core design components to its overall performance gains. Replacing standard Quantization-Aware Training (QAT) with RaBiT’s coupled training framework yields the most significant performance improvement, reducing Llama2-7B WikiText-2 perplexity from 6.55 to 5.84, which is subsequently refined to 5.78 by the integration of function-aware initialization. Furthermore, experiments with the importance-scaled initialization reveal a critical disconnect between weight reconstruction fidelity and actual downstream performance: while this initialization increases absolute weight-reconstruction error, it simultaneously reduces initial distillation loss by approximately 81%. This striking divergence empirically demonstrates that preserving downstream functional behavior, rather than minimizing individual weight deviations, is the primary driver of performance in ultra-low (2-bit) quantization regimes.

5. Discussion

Core Architecture and Training Dynamics

The fundamental thesis of RaBiT is that residual binarization is not inherently flawed, but has historically been undermined by suboptimal training methodologies. While prior approaches attempted to mitigate path co-adaptation using restrictive heuristics – such as freezing one path – these methods constrained optimization and degraded final accuracy. By contrast, RaBiT structurally enforces a residual hierarchy, enabling joint parameter optimization while guaranteeing functional differentiation across paths. This design is validated by training dynamics: RaBiT maintains a stable, negative inter-path correlation, whereas standard Quantization-Aware Training (QAT) exhibits positive drift, demonstrating that co-adaptation is structurally prevented rather than merely discouraged.

Technical and Practical Development Challenges

Translating this architectural framework into a viable system required overcoming several critical engineering and theoretical bottlenecks. First, diagnosing co-adaptation required isolating and decomposing error terms to expose the lack of path anti-correlation, as the issue was not visible on standard loss curves. Second, to maintain computational tractability during training, a hybrid approach was adopted: binary cores are derived dynamically on the fly, while the scales are treated as learnable parameters, avoiding the prohibitive overhead of full, step-by-step SVD recomputation. Finally, ensuring convergence across diverse architectures (Llama, Gemma, and Qwen) required precise initialization and activation-range tuning to prevent model collapse, while realizing the 4.49× on-device decoding speedup required developing custom kernels optimized to reduce memory-bandwidth traffic without relying on codebooks or rotation transforms.

Key Architectural Takeaway

The primary transferable insight from this work is that in ultra-low bitwidth regimes, preserving downstream functional behavior must supersede exact numerical weight reproduction. This principle is demonstrated both by RaBiT's initialization – where accepting higher weight reconstruction error yielded a substantial reduction in behavioral distillation loss – and by its training paradigm, where structural constraints successfully guided the optimizer toward superior minima. Ultimately, these findings reinforce that introducing the correct inductive bias is more effective than relying on unconstrained, brute-force optimization.

6. Conclusion

RaBiT resolves the inherent co-adaptation limitations of stacked binary networks through a structurally principled architecture. By deriving successive binary paths from a single shared weight tensor – enabling each path to sequentially correct the quantization error of its predecessor-and initializing the system to preserve downstream functional behavior rather than exact weight fidelity, RaBiT achieves state-of-the-art 2-bit quantization accuracy. The framework matches the accuracy of computationally intensive Vector Quantization (VQ) methods while delivering a 4.49× end-to-end decoding speedup on physical hardware. This performance is achieved by mitigating memory-bandwidth bottlenecks and eliminating the overhead of runtime codebooks and rotation transforms, while maintaining an architecture poised for fully matmul-free execution as hardware support matures. Ultimately, RaBiT demonstrates that the performance trade-offs of extreme quantization are optimization challenges rather than intrinsic physical limits, providing a robust, practical foundation for deploying capable large language models on commodity hardware.