vLLM on 4x Nvidia RTX A4000: Qwen 3.6 27B Q8 achieves 83 tokens/second

This review examines vLLM's inference performance on a 4x Nvidia RTX A4000 server, focusing on Qwen 3.6 models and specific vLLM configuration parameters. We analyze reported token generation and prefill speeds.

TL;DR Best for: Users with multi-GPU Ampere setups (like Nvidia RTX A4000s) seeking high-throughput local LLM inference, especially with Qwen 3.6 quantized models. Skip if: You require BF16 precision on similar memory-constrained hardware, or prioritize ease of setup over fine-tuned performance. Bottom line: vLLM delivers strong inference performance for Qwen 3.6 27B Q8 on a 4x A4000 setup, achieving 83 tokens/second, but requires careful configuration.

Methodology

This v0 review draws on the founder's published claims at the provided Reddit URL; independent benchmarks pending. Update cadence: re-tested when claims diverge from observed behavior.

The signal for this review is a Reddit post by user Alternative_Ad4267, published on 2026-05-29. The post details the user's experience migrating from llama.cpp to vLLM for local LLM inference on a specific hardware configuration. The tool under review is vLLM, with its version implied by the vllm serve command line arguments provided.

This review covers the reported performance metrics, including tokens per second for both generation and prefill, on a server equipped with 4 Nvidia RTX A4000 GPUs, each with 16GB of VRAM (64GB total). The system runs Cuda 13.2 on Fedora 43, leveraging Ampere architecture GPUs. The specific models tested are Qwen 3.6 27B GPTQ 8bit, Qwen 3.6 35B A3B FP8, and Qwen 3.6 27B FP8. The review also analyzes the detailed vllm serve command, which specifies various configuration parameters like tensor-parallel-size, gpu-memory-utilization, max-model-len, and attention-backend flashinfer.

What is NOT covered in this v0 review includes independent verification of the reported performance numbers, long-term stability of the setup, resource utilization beyond VRAM, a comprehensive comparative analysis against other inference engines (beyond the user's brief mention), or an investigation into edge cases.

What it does: High-throughput inference

vLLM is an open-source library designed for high-throughput and low-latency LLM inference. The user's configuration demonstrates its capability to distribute model inference across multiple GPUs using tensor-parallel-size 4, effectively utilizing the 4 Nvidia RTX A4000 cards. For the Qwen 3.6 27B GPTQ 8bit model, the user reports achieving up to 83 tokens per second on generation. This performance is attributed to vLLM's optimized serving architecture.

Advanced scheduling and caching

The vllm serve command reveals several parameters aimed at optimizing resource use and performance. The gpu-memory-utilization 0.90 setting maximizes VRAM usage. Features like enable-chunked-prefill and enable-prefix-caching are enabled, which are key to managing prompt processing and reducing redundant computation for common prefixes. The attention-backend flashinfer flag indicates the use of a highly optimized attention mechanism, crucial for maximizing throughput on Ampere architecture GPUs.

Qwen-specific optimizations

The configuration includes specific parsers for Qwen models: reasoning-parser qwen3 and tool-call-parser qwen3_coder. These indicate vLLM's support for model-specific functionalities, including automatic tool choice (enable-auto-tool-choice). The user also specifies a custom chat template (--chat-template /home/user/qwen3.6/chat_template.jinja) to fix reported behavior issues with Qwen's default template, highlighting the flexibility of vLLM's serving interface.

What's interesting / What's not

The most interesting aspect of this signal is the detailed, reported performance data on a specific, multi-GPU setup. Achieving 83 tokens per second on generation for a Qwen 3.6 27B Q8 model on 4 Nvidia RTX A4000 GPUs is a strong indicator of vLLM's efficiency for local inference. The explicit vllm serve command is a valuable artifact, providing a reproducible starting point for others with similar hardware. The high prefill generation speeds, up to 9k tokens per second, with a peak of 19k tokens per second when Qwen Code performs automatic context compression, underscore vLLM's ability to handle large input contexts efficiently. The use of flashinfer and prefix caching are clear technical choices that contribute to these numbers.

What is less compelling is the lack of direct, benchmarked comparisons against llama.cpp, which the user states they switched from. While the user expresses satisfaction with vLLM's speed, quantitative data for the previous setup would have provided a clearer performance delta. The "huge peak of 19k tokens per second on prefill when Qwen Code does automatic context compress" is intriguing but lacks specific conditions or a reproducible test case for that peak, making it difficult to assess its general applicability. Furthermore, the review focuses heavily on VRAM and GPU performance, but omits details on CPU, system RAM, or storage, which can also influence overall inference throughput and latency in a production environment. The user's assertion that if Q8 cannot make it, BF16 will not either, on a MacBook Pro M5 Max, is a generalization without supporting data from that specific machine.

Pricing

vLLM is an open-source library, available for free under the Apache 2.0 License. Costs are associated with underlying hardware and cloud infrastructure, not the software itself. Pricing snapshot date: 2026-05-29.

Verdict

vLLM is a highly effective choice for local LLM inference, particularly for users with multi-GPU setups like the 4x Nvidia RTX A4000 configuration detailed here. The reported 83 tokens per second for Qwen 3.6 27B Q8 demonstrates its capability to deliver high throughput on consumer-grade Ampere GPUs. Its advanced scheduling, caching, and tensor parallelism features are critical for maximizing hardware utilization. While requiring careful configuration, the detailed vllm serve command provided in the source offers a solid blueprint for achieving strong performance. For those prioritizing raw inference speed and willing to fine-tune their setup, vLLM is a compelling option.

What we'd test next

Our next steps would involve independently replicating the reported benchmarks for Qwen 3.6 27B Q8 on a similar 4x Nvidia RTX A4000 setup. We would conduct a direct comparative benchmark against llama.cpp using the exact same model and hardware to quantify the performance difference. Further testing would include measuring end-to-end latency under varying batch sizes and concurrent request loads, which is crucial for real-world application deployments. We would also investigate the performance impact of different gpu-memory-utilization values and attempt to reproduce the "19k tokens per second on prefill" peak under controlled conditions, documenting the specific context compression scenarios that trigger it.

Pull quote: “Achieving 83 tokens per second on generation for a Qwen 3.6 27B Q8 model on 4 Nvidia RTX A4000 GPUs is a strong indicator of vLLM's efficiency for local inference.”