ik_llama.cpp 110.24 tok/s on 12GB VRAM; Outperforms merged llama.cpp MTP

This review examines ik_llama.cpp's Multi-Token Prediction (MTP) performance, specifically its reported 110.24 tok/s average on an RTX 4070 Super 12GB, and its technical implications for local LLM inference.

TL;DR

Best for: Indie founders and developers optimizing local LLM inference on consumer-grade GPUs with limited VRAM (e.g., 12GB RTX cards), especially for tasks benefiting from high token throughput. Skip if: You have ample VRAM (24GB+) where traditional llama.cpp might perform adequately, or if your workflow does not prioritize raw token generation speed over other factors like model compatibility. Bottom line: ik_llama.cpp offers a compelling, measurable performance uplift for MTP on constrained hardware, making it a strong contender for efficient local LLM deployment.

METHODOLOGY

v0 review draws on the founder's published claims at https://github.com/ikawrakow/ik_llama.cpp and a user's detailed performance report on Reddit (https://www.reddit.com/r/LocalLLaMA/comments/1tj2gya/try_ik_llamacpp_with_mtp_if_you_have_limited_vram/). Tool name: ik_llama.cpp (version not explicitly stated, but implies latest as of user's test). Date observed: Reddit post from 2026-05-21. The review covers: The reported 110.24 tok/s average, the specific hardware (RTX 4070 Super 12GB), model (Qwen3.6-35B-A3B-IQ4_XS-4.19bpw.gguf), and the full llama-server launch parameters provided by Reddit user janvitos. It also covers the comparison against llama.cpp's merged MTP performance. What's NOT covered: Independent performance benchmarks, long-term workflow integration, or edge case behavior beyond the mtp-bench.py scenarios. Update cadence: re-tested when claims diverge from observed behavior.

WHAT IT DOES

ik_llama.cpp is a fork of llama.cpp by ikawrakow that implements Multi-Token Prediction (MTP) with a focus on performance on limited VRAM. The project aims to provide faster local LLM inference, particularly for users with consumer-grade GPUs.

Optimized Multi-Token Prediction

The core feature highlighted is its MTP implementation. MTP allows the LLM to predict multiple tokens simultaneously, rather than one-by-one, significantly increasing generation speed. The Reddit user janvitos specifically notes that ik_llama.cpp's MTP implementation provides superior performance compared to the MTP merged into the main llama.cpp branch, which reportedly saw a performance drop from 75-80 tok/s to 65-70 tok/s.

Specific Hardware and Model Support

The tool is demonstrated to run effectively on an RTX 4070 Super with 12GB VRAM. The benchmark uses byteshape's Qwen3.6-35B-A3B-IQ4_XS-4.19bpw.gguf model, a quantized 35B parameter model, indicating ik_llama.cpp's capability to handle larger models on more constrained hardware through efficient memory management and MTP.

Configurable Parameters for Performance

The provided launch parameters (--fit, --fit-margin, --ctx-size, --cache-type-k, --cache-type-v, --multi-token-prediction, --draft-p-min, --draft-max, --no-mmap, --mlock, --threads, --temp) reveal a high degree of configurability. These options allow users to fine-tune memory usage, caching strategies, and MTP behavior to maximize throughput for specific hardware configurations and models. The --fit and --fit-margin 1664 parameters are particularly relevant for managing VRAM on 12GB cards, with the user noting to increase fit-margin to 1792 or 2048 if OOM errors occur.

WHAT'S INTERESTING / WHAT'S NOT

The most interesting aspect is the quantifiable performance uplift over the official llama.cpp MTP implementation. A reported average of 110.24 tok/s on a 12GB RTX 4070 Super is a significant improvement over the 65-70 tok/s observed with the merged llama.cpp MTP, and even more so compared to the 75-80 tok/s before the merge. This suggests ikawrakow has found a more efficient way to manage the MTP pipeline, especially concerning VRAM utilization. This is not merely an incremental gain; it represents a substantial difference for interactive use cases.

The detailed mtp-bench.py output, showing per-task token rates, adds credibility. Tasks like summarize hit 122.3 tok/s, while long_code_review is lower at 101.4 tok/s, illustrating the variability but consistently high performance. The high aggregate accept rate (0.8749) for draft tokens is also noteworthy, indicating that the MTP mechanism is not sacrificing quality for speed.

What's not interesting, or rather, what's expected, is the need for careful parameter tuning. While the provided llama-server parameters are a great starting point, achieving optimal performance on different hardware or with other models will still require experimentation. The mention of using the iGPU for display to free up 100% of the discrete GPU's VRAM is a practical tip for power users, but also highlights the tight VRAM constraints that ik_llama.cpp is designed to address.

What's missing from the founder's pitch (or the Reddit post, which serves as the primary signal here) is a deeper technical explanation of why ik_llama.cpp outperforms the main llama.cpp branch on MTP. Without this, it's harder to predict how these performance gains will generalize to future llama.cpp updates or different hardware.

PRICING

ik_llama.cpp is an open-source project, available for free. There are no paid tiers or commercial offerings associated with the tool itself. Pricing snapshot date: 2026-05-21.

VERDICT

For indie founders and developers working with local LLMs on consumer-grade GPUs, ik_llama.cpp is the recommended choice for maximizing inference speed with Multi-Token Prediction. The reported 110.24 tok/s average on an RTX 4070 Super 12GB, significantly outperforming the official llama.cpp MTP, provides a clear, measurable advantage. This tool directly addresses the challenge of running large language models efficiently on hardware with limited VRAM, making high-throughput local inference a more accessible reality. Its detailed configuration options allow for fine-tuning to specific hardware, reinforcing its utility for performance-sensitive applications.

WHAT WE'D TEST NEXT

We would conduct independent benchmarks comparing ik_llama.cpp against the latest llama.cpp MTP implementation across a wider range of quantized models (e.g., Llama 3 8B, Mixtral 8x7B) and GPU configurations (e.g., RTX 3060 12GB, RTX 4090 24GB). Specifically, we would investigate the VRAM usage profiles under various context sizes and MTP parameters. A deeper dive into the architectural differences between ik_llama.cpp and llama.cpp's MTP would also be valuable to understand the source of the performance gains and their long-term sustainability. We would also test the impact of mlock and no-mmap on overall system stability and performance beyond raw token generation.

Pull quote: “A reported average of 110.24 tok/s on a 12GB RTX 4070 Super is a significant improvement over the 65-70 tok/s observed with the merged llama.cpp MTP, and even more so compared to the 75-80 tok/s before the merge.”