Why MoE models beat dense weights on legacy Volta GPU clusters

An analysis of a custom 16-GPU local legal drafting pipeline, highlighting why llama.cpp and Mixture of Experts outpace dense models on older enterprise hardware.

For teams running local LLMs on legacy enterprise hardware like Nvidia V100s, dense models are too slow for production. Instead, the optimal stack pairs llama.cpp with Mixture of Experts (MoE) models. This setup bypasses the architectural limitations of older Tensor Cores while delivering up to 113 tokens per second. Skip dense models entirely on Volta hardware if your throughput floor is 40 tokens per second.

Methodology

This review analyzes the hardware configuration and performance benchmarks published by user TumbleweedNew6515 on Reddit. The source document outlines a custom 16-GPU setup running across two servers: a Threadripper Pro host with twelve V100-SXM2 32GB GPUs, and an EPYC 7302P host with four RTX 3090s and two V100-PCIe GPUs. The benchmarks measure decode throughput using Q8 GGUF quantization, a Q4 KV cache, and Flash Attention enabled on a single four-card board. The tests were run on real-world legal drafting prompts with several thousand tokens of context. Independent verification of these exact hardware configurations is pending. This analysis covers the reported throughput metrics, hardware topology constraints, and software stack transitions, but does not cover long-term reliability or comparative power efficiency.

What it does

Hardware topology constraints

The system relies on twelve V100-SXM2 32GB GPUs on a Threadripper Pro motherboard, supplemented by a second box running an EPYC 7302P processor with 512GB RAM, four RTX 3090s, and two V100-PCIe cards. The physical layout requires strict model isolation. Splitting a model across different NVLink boards causes throughput to drop because cross-board communication must hop over slower PCIe/NUMA links rather than NVLink.

Software stack transition

The pipeline uses llama.cpp instead of vLLM. While vLLM is highly regarded for modern architectures, it presents a dead end for MoE GGUFs on Volta (SM 7.0) hardware. FP8, AWQ, and Marlin kernels require SM 7.5 or higher, and the GPTQ kernels are broken on SM 7.0. Mainline llama.cpp resolves these issues and handles long-context parsing without mangling prompts.

Multi-model orchestration

Rather than relying on a single monolithic model, the system uses a sequential pipeline that distributes tasks across 16 resident GPUs. A small Qwen router directs tasks. Qwen3.6-35B-A3B handles workhorse drafting, Qwen3.5-122B-A10B manages heavy reasoning, a dedicated pair of GPUs runs a gate model, and another pair hosts an adversarial reviewer.

What's interesting / What's not

MoE performance advantages

The performance delta between MoE and dense models on older hardware is stark. The user's benchmarks show Gemma-4-26B-A4B achieving approximately 113 tokens per second, while Qwen3.5-122B-A10B hits roughly 50 tokens per second. In contrast, dense models of similar sizes fail to meet usable thresholds.

A dense 27-32B model only manages 20 to 28 tokens per second, falling below the user's 40 tokens per second floor, while a dense 128B model crawls at 9 tokens per second.

Hardware cost realities

While the performance of the MoE models on legacy SXM2 hardware is impressive, the setup is highly specialized. Building a secondary EPYC system with mixed RTX 3090 and V100-PCIe cards introduces significant complexity in model routing and power management. For most enterprise teams, the engineering overhead of maintaining custom NVLink topologies and managing mixed-generation GPU nodes will quickly eclipse the cost savings of buying depreciated enterprise hardware.

Pricing

Hardware pricing is highly variable based on secondary market availability. As of May 2026, V100-SXM2 32GB cards and RTX 3090s are sourced via used hardware markets. The software stack (llama.cpp, Ollama, PyMuPDF, Tesseract) is entirely open-source and free.

Verdict

For developers and self-hosters committed to legacy Volta hardware, the verdict is clear: run llama.cpp and focus exclusively on Mixture of Experts models. The architectural limitations of SM 7.0 make dense models over 20B parameters unusable for interactive workflows. If you require high-throughput local inference on a budget, MoE models like Qwen3.5-122B-A10B provide the only viable path to maintaining speeds above 50 tokens per second.

What we'd test next

We would benchmark the power draw of this 16-GPU setup under full load to calculate the true total cost of ownership against modern alternatives like a dual RTX 4090 or Mac Studio setup. Additionally, we want to measure the latency penalty of the sequential orchestration pipeline when handling multi-turn legal analysis.

The investor read

This setup signals a growing trend where highly sensitive verticals, such as legal services, choose to build complex, multi-GPU local clusters rather than sending data to third-party APIs. The reliance on legacy Volta (V100) and Ampere (RTX 3090) hardware highlights a secondary market that remains highly active for cost-conscious builders. However, the extreme engineering overhead required to orchestrate these mixed-GPU environments suggests that enterprise tooling startups focusing on simplified local orchestration and heterogeneous GPU clustering have a massive, underserved market. For investors, this underscores that local data privacy demands are driving hardware-level complexity down to the individual practitioner level.

Pull quote: “Splitting a model across different NVLink boards causes throughput to drop because cross-board communication must hop over slower PCIe/NUMA links rather than NVLink.”