llama.cpp MTP Speculative Decoding Boosts Local LLM Inference on Consumer Hardware

We review the performance of llama.cpp's newly landed MTP speculative decoding, analyzing reported speedups for local LLM inference on Strix Halo and RTX 3090 GPUs.

The Answer Up Front

For developers and enthusiasts running large language models locally, llama.cpp's MTP speculative decoding is a significant performance upgrade. It's particularly beneficial for dense models like Qwen3.6 27B, offering substantial token generation speedups on consumer-grade hardware. Skip it if your primary models are Mixture-of-Experts (MoE) architectures, as their inherent efficiency reduces the relative gains. The bottom line is a compelling reason to update your llama.cpp build for faster, more responsive local LLM experiences.

Methodology

This v0 review draws on the founder C_Coffie's published claims on Reddit and linked blog posts, with independent benchmarks pending. Update cadence: re-tested when claims diverge from observed behavior.

• Tool: llama.cpp mainline, specifically PR #22673 (commit 4f13cb7), which landed MTP speculative decoding on May 16.
• Source Signal: The primary signal is a Reddit post by C_Coffie, linking to detailed blog posts and a GitHub repository containing raw YAML run data for reproducibility.
  • Reddit post: https://www.reddit.com/r/LocalLLaMA/comments/1tgxau6/llamacpp_mtp_support_landed_qwen36_27b_at_244_on/
  • Power limits writeup: https://calebcoffie.com/blog/how-much-do-power-limits-affect-llm-benchmark-tok-s
  • MTP writeup: https://calebcoffie.com/blog/benchmarking-llama-cpp-mtp-on-strix-halo
  • Raw YAML data: https://github.com/CCoffie/CalebCoffie.com/tree/main/content/benchmarks/runs
• What's Covered: This review covers C_Coffie's reported performance benchmarks for MTP speculative decoding, including specific token-per-second (tok/s) figures and speedup factors across different hardware, models, and quantization levels. Technical details on enabling MTP and its impact on various model types are also included.
• What's Not Covered: This review does not include independent performance verification by Founderr Pulse, analysis of long-term workflow integration, or edge-case testing beyond the reported single-stream chat scenarios.

What It Does

llama.cpp's MTP (Multi-Token Prediction) speculative decoding is an optimization technique designed to accelerate LLM inference by predicting multiple future tokens simultaneously. Instead of generating one token at a time, a smaller, faster draft model proposes a sequence of tokens, which the larger, more accurate target model then verifies in parallel. This significantly reduces the number of full forward passes required by the target model.

Reported Performance Gains

C_Coffie reports substantial speedups for the Qwen3.6 27B model. On a Strix Halo (Framework Desktop, ROCm 7.0.2), Q8_0 quantization saw a 2.44× speedup, increasing from 7.4 tok/s to 18.1 tok/s. Q4_K_M quantization on the same hardware achieved 1.81×, from 11.7 tok/s to 21.2 tok/s. For a single RTX 3090 (CUDA 12.9, driver 590.26) running Q4_K_M, the speedup was 1.54×, moving from 38.7 tok/s to 59.5 tok/s. A dual RTX 3090 setup, using layer-splitting for Q8_0, saw a 2.17× improvement, from 25.7 tok/s to 55.9 tok/s.

MoE Model Behavior

Notably, MoE models like Qwen3.6 35B-A3B (MoE) show comparatively smaller gains. C_Coffie reports a 1.40× speedup on Strix Halo and 1.24× on the RTX 3090. This is attributed to MoE models already having a cheaper per-token forward pass, as only a fraction of their parameters are active for each token. The benefit of saving N-1 forward passes is thus diminished.

Configuration and Output Integrity

Users can enable MTP with the command-line flags --spec-type draft-mtp --spec-draft-n-max N, where N is the maximum number of tokens to speculatively decode. C_Coffie explicitly states that the output generated with MTP is byte-identical to the baseline at the same seed and temperature, ensuring no compromise on output quality for the speed gains.

What's Interesting / What's Not

The most interesting aspect of this update is the significant, reported performance uplift for dense LLMs on readily available consumer hardware. A 2.44× speedup on a Strix Halo or 2.17× on an RTX 3090 rig for a 27B parameter model is not incremental; it fundamentally changes the responsiveness of local inference. C_Coffie's detailed methodology, including specific hardware configurations, software versions, and the provision of raw YAML run data on GitHub, lends considerable credibility to these claims. This level of transparency is crucial for the LocalLLaMA community and for developers seeking reproducible benchmarks. The analysis of how power limits affect tok/s on GPUs, detailed in a linked blog post, also provides valuable practical insight for anyone optimizing local setups.

What's less compelling, though well-explained, is the reduced benefit for MoE models. While understandable from a technical perspective, it means users primarily running models like Mixtral 8x7B might not see the same dramatic improvements. The need to tune the spec-draft-n-max parameter based on specific hardware and quantization levels also adds a layer of complexity for users aiming for optimal performance, requiring some experimentation rather than a universally optimal setting.

Pricing

llama.cpp is an open-source project, distributed under the MIT License, meaning it is free to use. There are no paid tiers or subscription costs associated with the software itself.

Verdict

llama.cpp's MTP speculative decoding is a must-adopt feature for anyone performing local LLM inference, particularly with dense models. The reported speedups, exceeding 2× in some configurations, translate directly to a more fluid and productive development experience. For those running Qwen3.6 27B or similar dense architectures on consumer GPUs or integrated NPUs like Strix Halo, this update offers a substantial, verifiable performance boost without compromising output quality. While MoE models see more modest gains, the overall impact on local LLM accessibility and performance is a clear win for the open-source community.

What We'd Test Next

Our next steps would involve independently replicating C_Coffie's benchmarks across a broader range of popular dense and MoE models, including Llama 3 8B/70B and Mixtral 8x7B, to confirm the reported speedups. We would also investigate the optimal spec-draft-n-max settings for various hardware and model combinations, generating a matrix of recommended configurations. Further testing would focus on the impact of MTP on different inference tasks, such as long-form content generation, coding assistance, and RAG workflows, to assess its real-world benefits beyond single-stream chat. Finally, we would benchmark the energy consumption implications of MTP to understand the efficiency trade-offs at higher performance levels.

The investor read

The continued optimization of llama.cpp for local inference, exemplified by MTP speculative decoding, signals a growing trend towards powerful LLMs becoming increasingly accessible on consumer hardware. This development reduces the reliance on expensive cloud APIs for certain inference tasks, potentially shifting tooling spend towards local compute and specialized hardware. While llama.cpp itself is an open-source project and not a direct investment target, its advancements are foundational for a myriad of startups building applications that leverage local LLM capabilities, from privacy-focused AI assistants to edge-device intelligence. Companies developing optimized hardware for local AI, or software layers that abstract llama.cpp for specific use cases, stand to benefit significantly from these performance gains, making them more investable.

Pull quote: “For developers and enthusiasts running large language models locally, llama.cpp's MTP speculative decoding is a significant performance upgrade.”