Local LLM Inference: DGX Station vs. SuperMicro for 122B AWQ Models

This review evaluates hardware options for a production-class local LLM inference server under $150K, focusing on NVIDIA DGX Station and SuperMicro configurations for 122B AWQ models and 300 users.

The Answer Up Front

For Porespellar's specific use case (122b AWQ models, 256k context, 300 users, $150K budget), a custom SuperMicro rack server with 4x NVIDIA RTX 6000 Ada Generation GPUs is the most pragmatic choice. The DGX Station is difficult to acquire and likely exceeds the budget for comparable performance. While not matching 4x H100s directly, the RTX 6000 Ada offers a strong balance of VRAM and inference performance within the stated financial and availability constraints. Skip if you require H100-level FP16 performance or have a longer procurement timeline.

Methodology

v0 review draws on the founder's published claims at https://www.reddit.com/r/LocalLLaMA/comments/1tr7b0n/if_you_had_150k_for_building_a_productionclass/; independent benchmarks pending. Update cadence: re-tested when claims diverge from observed behavior. This review covers hardware recommendations for a production-class local LLM inference server, specifically addressing Porespellar's stated budget of $150K and target workload: serving 122b AWQ models with 256k context and a small embedding model to 300 users via vLLM. The analysis considers the proposed NVIDIA DGX Station and a SuperMicro rack server with 4x NVIDIA RTX 6000 Ada Generation GPUs, comparing them against the user's existing 4x H100 setup. What's not covered includes independent performance benchmarks, long-term operational workflows, specific power consumption, cooling requirements, or edge cases beyond the stated model and context size.

What It Does

The inference challenge

Porespellar requires a failover server for a production LLM inference workload, targeting 300 users with 122b AWQ models at 256k context, using vLLM, plus a small embedding model. The budget is $150K, aiming for performance comparable to an existing 4x NVIDIA H100 setup.

NVIDIA DGX Station

The DGX Station is a workstation-class system designed for AI development and deployment, often featuring multiple high-end NVIDIA GPUs (e.g., A100s or H100s in newer generations) in a compact, office-friendly form factor. Porespellar considered this for its integrated design and potential for high performance.

SuperMicro with RTX 6000 Ada

The alternative proposed is a custom SuperMicro rack server configured with 4x NVIDIA RTX 6000 Ada Generation GPUs. Each RTX 6000 Ada card features 48GB of GDDR6 ECC VRAM, a significant capacity for large models, and is built on the Ada Lovelace architecture. This configuration aims to balance VRAM capacity, inference throughput, and cost.

What's Interesting / What's Not

H100 performance vs. budget reality

Porespellar's existing 4x H100 setup represents a high-water mark for current inference performance, particularly for FP16 workloads. The claim that "H100s are reaching the end of their product cycle" is not accurate; H100s are current-generation, high-demand, and expensive. The challenge is replicating this performance within a $150K budget.

RTX 6000 Ada as a strong contender

The NVIDIA RTX 6000 Ada Generation is a compelling option for inference. Its 48GB VRAM per card allows for loading large models like 122b AWQ with 256k context, especially with Tensor Parallelism (TP=2) as Porespellar describes. While it won't match the raw FP16 throughput of an H100, its FP8 and INT8 capabilities, combined with its VRAM, make it highly efficient for quantized inference. The key here is the quantized 122b AWQ model, which benefits significantly from the RTX 6000 Ada's architecture.

DGX Station availability and cost

The DGX Station, while appealing for its integrated nature, is notoriously difficult to acquire quickly and often carries a premium price tag that would likely push a 4-GPU H100 or even A100 configuration well beyond the $150K budget. Finding them "available for purchase yet" is a common hurdle. A custom SuperMicro build offers greater flexibility in component selection and potentially better cost efficiency.

The "best bang for the buck" dilemma

The current hardware market, particularly for high-end NVIDIA GPUs, is characterized by high demand and elevated prices. Achieving "best bang for the buck" requires careful optimization between VRAM, compute, and interconnects. For quantized inference, VRAM capacity is often the primary bottleneck, making the 48GB per card of the RTX 6000 Ada a strong asset.

Pricing

The stated budget is $150,000. As of May 2026, individual NVIDIA H100 GPUs can cost upwards of $30,000-$40,000 each, making a 4-H100 system easily exceed the budget. NVIDIA RTX 6000 Ada Generation GPUs typically retail for around $8,000-$10,000 per card. A SuperMicro server chassis, CPU, RAM, and other components for a 4-GPU system would add another $10,000-$20,000. This places a 4x RTX 6000 Ada configuration comfortably within the $150K budget, leaving room for ancillary costs. DGX Station pricing varies widely based on configuration and availability, but a comparable system would likely start above $100,000, potentially leaving little room for a 4-GPU H100 equivalent within the budget.

Verdict

For Porespellar's requirement of a $150K failover inference server running 122b AWQ models with 256k context for 300 users, a custom SuperMicro rack server equipped with 4x NVIDIA RTX 6000 Ada Generation GPUs is the recommended path. The RTX 6000 Ada's 48GB VRAM per card is crucial for handling large quantized models and contexts, offering a superior VRAM-to-cost ratio compared to H100s for this specific workload. The DGX Station, while an integrated solution, presents significant acquisition challenges and budget overruns for a comparable setup. This recommendation prioritizes VRAM capacity and cost-effectiveness for quantized inference over raw FP16 throughput, which is less critical for AWQ models.

What We'd Test Next

A v2 review would benchmark the actual inference throughput (tokens/second) and latency of a 4x RTX 6000 Ada system running Porespellar's specified 122b AWQ model with 256k context under vLLM, comparing it directly against a 4x H100 system. We would also evaluate the system's performance with varying numbers of concurrent users (e.g., 50, 150, 300) to understand scaling characteristics and identify potential bottlenecks beyond GPU compute. Specific attention would be paid to memory bandwidth utilization and the impact of the interconnect (e.g., PCIe Gen5 vs. NVLink) on multi-GPU inference for this workload.

The investor read

The demand for local inference hardware, particularly for large language models, continues to outstrip supply, driving up prices for high-end GPUs like NVIDIA's H100s. This signal highlights a growing segment of the market where companies are building out on-premise or private cloud inference capabilities to manage costs, data privacy, or latency. The shift towards quantized models (like AWQ) and specialized inference engines (vLLM) indicates a maturing ecosystem focused on efficiency. Companies like SuperMicro, which provide flexible server platforms for various GPU configurations, are well-positioned. Investment opportunities exist in hardware optimization for specific inference workloads, software layers that abstract hardware complexity, and solutions that bridge the gap between cloud and on-premise LLM deployment. The challenge for startups remains acquiring sufficient GPU inventory to scale.

Pull quote: “For Porespellar's requirement of a $150K failover inference server running 122b AWQ models with 256k context for 300 users, a custom SuperMicro rack server equipped with 4x NVIDIA RTX 6000 Ada Generation GPUs is the recommended path.”