vLLM GPU cost benchmarks show hardware winners shift by latency SLA

Qaiser Mehdi's public vLLM benchmark reveals how RTX 5090, H100, A100, and RTX 3090 unit economics flip based on target p95 latency thresholds rather than raw hourly rates.

The answer up front

If your application requires p95 response times under three seconds, rent an H100. Despite its high hourly cost ($3.557/hour on vast.ai), the H100 achieves high batch saturation while maintaining low latency, delivering $0.162 per 1M output tokens under a 3-second SLA. If your SLA tolerates sub-8-second responses, the RTX 5090 takes the efficiency crown at $0.050 per 1M output tokens. Skip the RTX 3090 for production workloads unless your latency tolerance exceeds 18 seconds, and avoid comparing GPUs purely at static concurrency levels.

Methodology

This review analyzes public benchmark data published by Qaiser Mehdi in November 2026. The test suite evaluates Qwen2.5-7B-Instruct served on vLLM version 0.27.1 across four cloud GPU instances rented on vast.ai: an RTX 3090 ($0.259/hour), RTX 5090 ($0.522/hour), A100 80GB PCIe ($1.017/hour), and H100 80GB SXM ($3.557/hour).

The benchmark protocol sent synthetic payloads with 512-token prompts and 256-token output generations across concurrency levels ranging from 1 to 256 concurrent requests, stepping through 60-second measurement windows. Token counts and request counts were validated against vLLM's internal /metrics endpoint alongside client-side telemetry.

v0 review caveat: This analysis draws on Qaiser Mehdi's published claims and raw metrics at https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/corpus; independent benchmarks under dynamic traffic patterns are pending. Update cadence: re-tested when vLLM engine flags or underlying hardware performance diverge from observed behavior.

Hardware performance across latency SLAs

High throughput flips unit economics

Comparing GPU hardware solely on hourly rental rates produces misleading financial models. At low concurrency levels, consumer-grade hardware appears cheaper, but under high concurrent loads, enterprise accelerators sustain faster response times.

When enforcing a strict sub-2-second p95 latency boundary, the H100 SXM is the only tested GPU capable of meeting the target, costing $0.441 per 1M output tokens. Under a 3-second SLA, the H100 delivers $0.162 per 1M tokens, outperforming the RTX 5090 ($0.382 per 1M tokens) by 2.4x because the RTX 5090 must restrict batch concurrency to four requests to maintain response speed.

Consumer cards dominate relaxed SLAs

When latency bounds loosen to sub-4-second or sub-8-second SLAs, the RTX 5090 shifts into its optimal throughput window. At a sub-4-second p95 threshold, the RTX 5090 drops to $0.120 per 1M output tokens, beating the H100's $0.162 rate. At sub-8-second thresholds, the RTX 5090 reaches $0.050 per 1M output tokens, cutting costs in half compared to the H100 ($0.099) and A100 ($0.106).

What's interesting vs what's not

What's interesting

The core value of Mehdi's dataset is dismantling static-load benchmarks. Comparing cards at a fixed 64-request concurrency makes the H100 look inefficient at $0.162 per 1M tokens versus the RTX 5090 at $0.050 per 1M tokens. However, that comparison ignores that the H100 delivers a 2.6-second median latency while the 5090 hits 5.4 seconds.

Furthermore, price sensitivity varies drastically by provider pricing. At alternative cloud rates ($0.78/hour for RTX 5090 and $2.65/hour for H100 SXM), the H100 matches the 5090 on sub-8-second SLAs ($0.074 vs $0.075) and beats it by 4.7x on sub-3-second targets ($0.121 vs $0.571).

What's missing from the pitch

The dataset assumes 100% GPU utilization during the measured window. Real production workloads experience variable traffic spikes, idle periods, and cold starts that alter unit economics. Additionally, consumer cards like the RTX 5090 and 3090 were capped at 64 concurrent requests in the benchmark, whereas enterprise cards were pushed to 256, leaving high-concurrency efficiency for consumer silicon under-tested.

Pricing structure

Pricing is derived from spot rentals on vast.ai as of November 2026:

• RTX 3090 (24GB): $0.259 per hour
• RTX 5090 (32GB): $0.522 per hour
• A100 PCIe (80GB): $1.017 per hour
• H100 SXM (80GB): $3.557 per hour

Public cloud instances without spot discounts alter these equations significantly, making Mehdi's custom pricing input tool at https://qaisermehdi3-coder.github.io/qvunex/ essential for evaluation.

Verdict

Do not select GPU inference hardware based on raw cost per hour or static token generation speeds. For user-facing interactive applications requiring sub-3-second p95 latencies, the H100 SXM provides the lowest token cost due to memory bandwidth advantages under heavy batching. For asynchronous background jobs or less time-sensitive chat workflows tolerating 5 to 8 second latencies, the RTX 5090 offers superior return on compute spend.

What we'd test next

We plan to evaluate these cards under non-synthetic, variable arrival rates (Poisson distribution traffic) rather than fixed batch sizes. Future iterations will test larger model sizes (32B and 70B parameter models requiring tensor parallelism), measure cold-start latency, and evaluate KV-cache memory exhaustion limits under long-context scenarios (8k+ tokens).

The investor read

Mehdi's benchmark highlights a structural transition in AI infrastructure spend: hardware selection is decoupling from raw GPU per-hour pricing and anchoring to latency SLAs. As consumer-grade cards like the RTX 5090 offer competitive throughput for background jobs, inference providers face pressure to offer SLA-tiered pricing models rather than flat GPU rental rates. For venture investors evaluating LLM infrastructure, companies providing dynamic routing across heterogeneous GPU clusters (directing strict SLA traffic to H100/B200 and batch traffic to consumer silicon) are positioning themselves to capture margin efficiency as inference workloads scale.

Pull quote: “Comparing GPU hardware solely on hourly rental rates produces misleading financial models.”