Kimi Delta vs. Gated DeltaNet: A trade-off between training speed and model loss

A comparative study of four linear attention architectures provides clear benchmarks on the trade-off between training throughput and final validation loss for founders building custom long-context models.

The Answer Up Front

For teams training their own long-context models (350M-3B parameters) who need to escape the quadratic cost of standard attention, this research provides a clear decision framework. If your primary goal is the best possible model performance for a given compute budget, the paper's findings point to Kimi Delta Attention with the Muon optimizer, which achieved the lowest final validation loss. If your main constraint is training time and you need maximum throughput, a pure Gated DeltaNet stack trained with AdamW is the fastest configuration benchmarked. Skip this if you primarily fine-tune existing models or work with contexts short enough that standard attention is not a bottleneck.

Methodology

This v0 review analyzes the findings presented in the academic paper "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing," published on Hugging Face on July 10, 2026. The paper itself is the source signal and the public artifact containing the benchmarks. Our analysis is based entirely on the data and claims within this paper; independent benchmarks are pending.

This review covers the paper's central comparison of four linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. The benchmarks focus on 350M-parameter models trained for 15B tokens, measuring training throughput and final validation loss. We also cover the comparison of AdamW and Muon optimizers and the introduction of a new Cross-Layer Value Routing (CLVR) mechanism.

What is not covered are independent performance benchmarks, as we have not replicated the study. The paper explicitly omits empirical inference-speed benchmarks, which are critical for production deployment. Downstream task evaluations in the paper are limited, so we do not assess real-world task performance extensively.

What it does

Standard self-attention, while powerful, has a computational cost that scales quadratically with sequence length (O(n²)). This makes training and inference on very long contexts prohibitively expensive. This paper evaluates four alternative architectures that use recurrent mechanisms to achieve linear scaling (O(n)).

The architectures under test

The study provides a unified framework for understanding four recent architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. By expressing them in a common recurrent-memory notation, the authors make their differences in memory decay, expressivity, and control mechanisms explicit. The core comparison uses 350M-parameter models, providing a direct look at how architectural choices affect outcomes at a common scale.

Optimizer and stack composition matter

Beyond the core architecture, the paper tests two other variables. First, it compares the standard AdamW optimizer against the newer Muon optimizer. The results show Muon consistently lowers the final validation loss for the architectures tested. Second, it evaluates "hybrid" models, which mix linear attention layers with standard softmax attention layers. These hybrids generally improve model performance (lower loss) but at the cost of reduced training throughput, presenting another trade-off for model developers.

A new routing mechanism

The paper also introduces and evaluates a new technique called Cross-Layer Value Routing (CLVR). This lightweight mechanism is designed to improve information flow between layers in DeltaNet-style models. The authors find it provides a modest improvement, lowering final validation loss for both DeltaNet and Gated DeltaNet in their tests.

What's interesting / what's not

The most valuable contribution here is the direct, quantified trade-off between speed and performance. The paper doesn't just say one model is faster; it shows how much faster (normalized training throughput) and at what cost (final validation loss). For a founder allocating a fixed budget for a model training run, this is actionable data. The choice between Gated DeltaNet for speed and Kimi Delta for quality is a concrete engineering decision, not an abstract preference.

The consistent performance lift from the Muon optimizer is also a significant finding. It suggests that optimizer choice is as critical as architectural choice, at least in this specific training regime. For teams looking for every possible performance gain, this is a clear directive to test Muon.

What's less impactful is the new CLVR mechanism. The paper is commendably transparent that it offers only a "modest improvement." While a positive contribution, it's an incremental refinement, not a fundamental breakthrough. The most significant gap in the paper is the lack of inference benchmarks. Training throughput is critical, but for any model intended for production, inference latency and cost at long sequence lengths are paramount. The authors acknowledge this omission, but it remains the largest open question for anyone considering these architectures for a live product.

Pricing

These are open research architectures, not commercial products. They are free to implement, with the only cost being the compute required for training and inference. (Pricing snapshot: July 10, 2026).

Verdict

This paper provides a clear guide for teams building their own long-context models. The choice depends directly on your primary constraint. If you are optimizing for the lowest possible training cost and fastest iteration time, the data supports using a pure Gated DeltaNet stack with the AdamW optimizer. It delivers the highest training throughput. If you are optimizing for the best possible model quality and have a fixed compute budget, Kimi Delta Attention paired with the Muon optimizer is the superior choice, achieving the lowest validation loss in the paper's 350M-parameter comparison. Hybrid stacks offer a configurable middle ground for teams willing to trade some throughput for better performance.

What we'd test next

A v2 of this review would require independent benchmarks. First on the list is inference speed. We would need to measure latency and throughput at various sequence lengths (e.g., 32k, 128k, 1M tokens) to understand the production viability of these models. Second, we would expand the downstream task evaluation beyond the paper's limited set to include suites like MMLU, HumanEval, and long-context-specific tasks like "needle in a haystack" retrieval. Finally, we would want to see if these trade-offs hold at larger model scales, such as 7B and 13B parameters, where scaling laws can shift.

The investor read

This paper signals the intense focus on moving beyond the quadratic bottleneck of standard self-attention for long-context AI. The market is rewarding efficiency, and architectures like the DeltaNet family, Mamba, and RWKV represent the frontier. While this is an academic paper, not a company, it provides a public-facing benchmark for a critical component of next-generation models. An investable company in this space would either productize one of these efficient architectures into a best-in-class foundation model for long-context tasks or provide the tooling that makes training and deploying them significantly easier than current methods. The key is translating architectural efficiency into a defensible product advantage, as the underlying research is rapidly becoming open and commoditized.

Pull quote: “If your primary goal is the best possible model performance for a given compute budget, the paper's findings point to Kimi Delta Attention with the Muon optimizer.”