ByteShape CPU-5 quantizes Qwen 35B for 30% faster generation on 6GB VRAM

A head-to-head benchmark of ByteShape CPU-5 against Unsloth UD-IQ4_XS on a memory-constrained RTX 3060 laptop, demonstrating a significant text generation speedup during CPU offloading.

For indie developers and researchers running large local models on highly constrained hardware, ByteShape's CPU-5 quantization format is a superior alternative to traditional imatrix (IQ) quants. If you are running a 35B parameter model like Qwen3.6-35B-A3B on consumer laptops with only 6GB of VRAM, ByteShape delivers a substantial 30% speedup in text generation. Skip it if your hardware has enough VRAM to fit the entire model, as the prompt processing speed is slightly slower than Unsloth's alternative. The bottom line is that ByteShape makes local agentic workflows viable on older, low-VRAM hardware by optimizing CPU-bound generation.

Methodology

This review analyzes a head-to-head benchmark conducted by Reddit user OsmanthusBloom on May 22, 2026. The testing compared the ByteShape CPU-5 quantization (Q4_K_S-4.22bpw, 18.3 GB) against the Unsloth UD-IQ4_XS quantization (17.7 GB) using the Qwen3.6-35B-A3B model. The physical testbed was an Asus ROG Zephyrus G14 laptop (2021 model) equipped with an AMD Ryzen 7 5800HS CPU (8 cores, 16 threads), an NVIDIA RTX 3060 Laptop GPU with 6GB VRAM, and 24GB of DDR4 RAM running at 3200 MT/s. The software stack consisted of Linux Mint 22.2, llama.cpp version 9203 (commit 87589042c), and CUDA 12.0. The test configuration used a context size of 65,536 tokens, mmap disabled, mlock enabled, and a ubatch size of 2048. Independent verification of these exact speeds is pending our own lab replication; this analysis evaluates the user's documented configuration and the underlying architectural differences between the quants.

Optimizing constrained local execution

ByteShape provides custom quantization formats designed to run large language models on consumer-grade hardware. When deploying a 35B parameter model like Qwen3.6-35B-A3B, the model's weights exceed the 6GB VRAM limit of a mobile GPU like the RTX 3060. This forces llama.cpp to offload a massive portion of the model to the system's DDR4 RAM, executing calculations on the CPU.

Bypassing the imatrix bottleneck

The benchmark compares two distinct quantization strategies. The Unsloth UD-IQ4_XS quant uses an importance matrix (imatrix) to optimize quantization loss, resulting in a 17.7 GB file. ByteShape's CPU-5 quant (Q4_K_S-4.22bpw) is slightly larger at 18.3 GB but uses a regular Q-quantization scheme. While imatrix quants preserve high quality at lower bitrates, they are notoriously slow to compute on the CPU during the text generation phase. ByteShape's regular Q-quant bypasses this computational overhead, trading a minor increase in file size for a massive boost in execution speed.

Performance trade-offs in action

Under identical test conditions with a 10k token prompt and a 1.5k to 2k token generation run, the performance split is clear:

• Prompt Processing (PP): Unsloth achieved 585 tok/s, while ByteShape managed 564 tok/s (a 4% advantage for Unsloth).
• Text Generation (TG): Unsloth generated text at 25.4 tok/s, whereas ByteShape reached 33.1 tok/s (a 30% speedup for ByteShape).

The CPU offload reality

What makes this benchmark interesting is how it exposes the hidden costs of imatrix quants in hybrid CPU/GPU environments. In pure GPU setups, the computational overhead of imatrix (IQ) quants is negligible. However, when a developer is forced to offload to a Ryzen 7 5800HS CPU, those complex calculations drag down text generation speeds. ByteShape's decision to offer a "CPU-optimized" regular Q-quant (CPU-5) directly addresses this bottleneck. A 30% increase in generation speed (from 25.4 to 33.1 tok/s) is the difference between an interactive, usable local coding assistant and an agonizingly slow terminal.

The prompt processing penalty

What is less impressive is the 4% regression in prompt processing speed. For agentic workflows that constantly feed large context windows back into the model, prompt processing speed is highly critical. If your workflow involves sending massive, multi-thousand-token system prompts for single-token classifications, the Unsloth quant remains marginally faster. Furthermore, ByteShape's CPU-5 quant is 600MB larger than Unsloth's UD-IQ4_XS, which eats into the already tight 24GB system RAM overhead.

Pricing

Both the ByteShape CPU-5 and Unsloth UD-IQ4_XS quantization files are open-source and free to download via Hugging Face. The underlying execution engine, llama.cpp, is also open-source. This pricing snapshot is accurate as of May 2026.

Verdict

For developers running local agentic coding workflows on older laptops with 6GB of VRAM, the ByteShape CPU-5 quant is the superior choice. The 30% boost in text generation speed easily outweighs the minor 4% loss in prompt processing efficiency. However, if your system has a high-end GPU with 16GB or more VRAM, you should stick to Unsloth's imatrix quants or run unquantized models, as you will not experience the CPU-bound bottlenecks that ByteShape's format is designed to mitigate.

What we'd test next

In a future benchmark, we want to test the perplexity loss of ByteShape's CPU-5 quant against Unsloth's UD-IQ4_XS to verify if the speedup comes at the cost of reasoning quality. We also need to test ByteShape's "GPU-5" imatrix variant on the same hardware to see if the performance delta is entirely due to the imatrix calculation overhead or if ByteShape has introduced proprietary optimizations to the quantization layout itself.

The investor read

This benchmark highlights a critical shift in the local LLM tooling ecosystem: the optimization of the "long tail" of developer hardware. While venture capital heavily funds massive cloud-based API infrastructure, a massive demographic of indie builders and enterprise developers in highly secure environments rely on local execution. The fact that a 35B parameter model can run at an interactive 33 tok/s on a five-year-old consumer laptop (RTX 3060) signals that local hardware lifespans are being artificially extended by software-layer optimizations. For investors, this suggests that the addressable market for local AI developer tools is much larger and more resilient than cloud-only SaaS metrics imply. Companies like Unsloth and ByteShape are fighting for developer mindshare at the quantization layer, which acts as the gatekeeper for local model adoption.