Adrian Höhne's llama.cpp fork caches MoE experts instead of layers in VRAM

An evaluation of an experimental llama.cpp fork that optimizes Mixture of Experts models on 12GB GPUs by caching active experts in VRAM rather than offloading entire layers.

This experimental fork is for local LLM developers and indie founders running large Mixture of Experts (MoE) models on constrained hardware, specifically 12GB VRAM GPUs like the RTX 2060, 3060, or 4060. If you run dense models or have ample VRAM, skip this entirely. The bottom line is that by caching frequently used experts in VRAM instead of blindly offloading entire layers to the CPU, this fork offers a path to run models like Qwen 35B or Gemma 26B with a 100k context window at usable speeds on consumer-grade hardware.

Methodology

This review evaluates the experimental llama.cpp fork developed by Adrian Höhne, accessed via his public GitHub repository on May 22, 2026. Because this is a v0 review, it relies on the developer's published implementation details, command-line execution patterns, and initial self-reported benchmarks on Linux; independent verification on our own hardware test rigs is pending. The scope of this analysis covers the technical mechanism of the "experts first" hot-cache allocation, the profiling workflow, and the reported token-generation throughput on an NVIDIA RTX 2060 12GB GPU. It does not cover long-term stability, performance on Windows or macOS, compatibility with non-MoE architectures, or behavior under multi-user concurrent workloads.

What it does

Profiling expert activation patterns

Standard implementations of llama.cpp split MoE models across the VRAM and CPU boundary by entire layers. When VRAM is full, remaining layers offload to system memory. This fork introduces a profiling step using the --moe-layer-perf-out experts.json flag. This flag monitors and records which specific experts within the MoE layers are activated most frequently during a run. The developer observed that the initial layers of models like Qwen3.6-35B-A3B exhibit higher expert churn, making their specific experts more critical to keep in fast memory.

Dynamic hot-cache expert allocation

Instead of keeping inactive experts in VRAM just because they belong to an on-GPU layer, the fork uses a dynamic cache. By executing the server with the --moe-hot-cache argument pointing to the generated profile, the engine loads only the highly active experts into VRAM. Parameters like --moe-hot-cache-max-mib and --moe-hot-cache-auto-reserve-mib 1024 control the memory footprint, allowing users to reserve VRAM for large context windows (such as 100k) while keeping the most critical weights on the GPU.

Visualizing expert utilization

The fork integrates directly with the default Llama UI, adding a visual monitor that displays real-time expert usage. This interface allows developers to inspect the hit rate of the GPU cache during inference, providing immediate feedback on whether the current cache allocation matches the prompt's actual expert distribution.

What's interesting / what's not

Granular memory management is necessary

The technical approach is highly logical. In MoE models like Qwen3.6-35B-A3B, only a fraction of the total experts (typically 8 experts per token) are active at any given step. Offloading entire layers means wasting precious VRAM on inactive weights. Caching at the expert level rather than the layer level is a genuine architectural improvement for consumer hardware. The developer's reported jump from 19 tk/s under standard CPU-offloading to 26 tks with a 62% cache hit rate demonstrates that memory bandwidth bottlenecks can be bypassed without buying expensive enterprise GPUs.

The profiling overhead penalty

The main drawback is the workflow friction. The approach requires a two-step process: first running a slow, CPU-bound profiling pass to generate the JSON map, and then restarting the server with the hot-cache flags. If a user's prompt style or domain changes significantly, the cached profile may become stale, dropping the hit rate below the reported 42% break-even point where CPU-to-GPU transfer overhead negates the speed gains. Furthermore, the implementation is currently limited to Linux and requires manual compilation, making it unsuitable for non-technical users.

Pricing

As an open-source fork of llama.cpp, this software is entirely free to use under the original project's MIT license. This pricing status is accurate as of the repository observation date on May 22, 2026.

Verdict

We recommend this fork for developers running MoE models on 12GB GPUs who require large context windows for tasks like codebase analysis. The performance gains are real if you can maintain a cache hit rate above 42%. However, if your workflow involves highly diverse prompts that cause erratic expert activation, or if you are running on Windows, you should stick to the main upstream branch of llama.cpp until this memory-management strategy is fully upstreamed and automated.

What we'd test next

In a v2 evaluation, we would benchmark this fork on an RTX 3060 and an RTX 4060 to measure how the increased CUDA core count and memory bandwidth affect the 42% break-even threshold. We also want to test the stability of the expert cache under continuous context shifting during a 100k token coding session, verifying if the cache hit rate remains stable or degrades over long multi-turn conversations.

The investor read

This fork highlights a critical trend in local AI tooling: software-level memory optimization is delaying the hardware obsolescence of mid-range consumer GPUs. For venture investors, this signals that the addressable market for local, high-context MoE models is much larger than current hardware sales suggest. Startups building localized developer tooling do not necessarily need to mandate Mac Studio or dual-RTX 4090 setups for their users. If optimization techniques like Adrian Höhne's expert-level caching become standard in upstream runtimes, the barrier to entry for running 30B+ parameter models locally will drop significantly, shifting value from hardware providers to highly optimized local orchestration layers.

Pull quote: “The performance gains are real if you can maintain a cache hit rate above 42%.”