Gemma 4 E2B Serving on TPU v6e: Performance and QAT Limitations

A detailed report outlines serving Gemma 4 E2B on a single Google Cloud TPU v6e chip, achieving 213 tok/s for $1.35/hour, while highlighting critical failures for QAT variants.

The dev.to blog post details an experiment serving the 2-billion-parameter Gemma 4 E2B model on a single Google Cloud TPU v6e chip. The setup, using a GCE flex-start VM and vLLM 0.23.1rc1.dev1076, achieved a sustained 213 tokens per second for a single user with a 16 ms first token latency. This performance comes at a reported cost of $1.35 per chip-hour.

TPU v6e Performance Metrics

The report, measured on a ct6e-standard-1t TPU v6e chip with 32 GB HBM in europe-west4-a, indicates the plain google/gemma-4-E2B-it model performs effectively. Beyond the single-user benchmark, the system scaled to approximately 2,200 output tokens per second across concurrent streams. The model also handled OpenAI-style function calling, including parallel calls and refusal to hallucinate calls, and answered simple vision questions accurately in roughly 200 ms.

Deployment Mechanics and Costs

Deployment utilized a GCE flex-start VM, which is billed until deletion and hard-stopped at a four-hour maximum run. The boot disk required 200 GB, as the default 10 GB was insufficient for the vLLM image. The boot timeline involved the VM running at t+0, Docker installed by t+1:00, the image pulled by t+6:00, and weights downloaded, XLA compiled, and health green by t+8:30. The serving flags included --max-model-len 65536, --gpu-memory-utilization 0.9, --max_num_batched_tokens 4096, --enable-auto-tool-choice, --tool-call-parser gemma4, and --reasoning-parser gemma4, using bf16 weights and tensor-parallel 1. The vLLM setup automatically selected an fp8_e5m2 KV cache on the v6e, which is noted as the largest memory consumer before weight quantization.

QAT Variants Fail to Load

The report identifies significant issues with quantized versions of Gemma 4 E2B. Three specific QAT (Quantization Aware Training) checkpoints failed to load. The -qat-w4a16-ct JAX export encountered an unimplemented quantization path for E2B's per_layer_model_projection. Both qat-q4_0-unquantized JAX and torchax exports reported "missing" k_norm.weight for layers 15–34. The author attributes these failures to a loader bug, not the checkpoint itself, noting that the QAT export correctly omits k_norm for KV-shared layers (15–34) while the loader expects it. This issue was filed upstream as tpu-inference #3225.

Schema Enforcement Behavior

A specific interaction was observed regarding schema enforcement for function calling. The enforcement mechanism only engaged when "thinking is enabled" in the request configuration. Requests made with thinking disabled proceeded unconstrained, returning a 200 status. This behavior is described as a configuration interaction rather than a TPU limitation.

What We'd Change

The reported setup, while demonstrating raw performance, is not production-ready for continuous LLM serving. The GCE flex-start VM's four-hour hard stop makes it unsuitable for sustained inference workloads, requiring a transition to a standard GCE instance with a persistent TPU allocation. This change would likely impact the per-hour cost, requiring re-evaluation of the $1.35/chip-hour benchmark for long-term deployments.

The consistent failure of QAT variants to load is a critical limitation. Founders aiming for lower inference costs and higher throughput via quantization cannot currently rely on this specific vLLM/TPU v6e combination for Gemma 4 E2B. While the issue is attributed to a loader bug and a GitHub issue has been filed, this means that for now, the more memory-intensive and potentially slower plain bf16 model is the only viable option. Future iterations of vLLM or Gemma 4 E2B on TPU would need to resolve these quantization path and loader bugs before founders can realize the full cost-performance benefits of quantized models. The finding that schema enforcement is conditional on "thinking enabled" also indicates a need for clearer documentation or more robust default behavior, as this could lead to unexpected model responses if not explicitly managed.

Landing

The dev.to report offers a granular view into the current state of serving Gemma 4 E2B on Google Cloud's TPU v6e. While raw performance for the base model is compelling, the inability to utilize quantized variants and the operational constraints of flex-start VMs present clear challenges for production deployments. Founders evaluating this stack must weigh the reported performance against these significant limitations, particularly if cost-optimized inference or continuous uptime are primary concerns.

The investor read

This report signals continued investment in specialized hardware for LLM inference, with Google Cloud's TPU v6e positioning itself as a cost-effective option for smaller models like Gemma 4 E2B. The $1.35/chip-hour cost is competitive for a 2B parameter model, especially with reported throughput of 213 tok/s (single user) and 2,200 tok/s (concurrent). However, the critical failure of QAT variants to load indicates immaturity in the software stack (vLLM on TPU) for optimized inference. Investors should note that while raw performance is shown, the lack of robust quantization support limits immediate cost-efficiency gains and broad adoption for production workloads where every dollar per token counts. This suggests an early-stage market for TPU-based LLM serving, with significant software refinement still required to unlock its full potential.