Max Bob analyzed 1.19 million network probes across 46 inference providers in US Central, Tokyo, and São Paulo, detailing why geographic location dictates model selection and API edge architecture.

30-day inference benchmark reveals severe regional latency penalties across 46 providers

Max Bob analyzed 1.19 million network probes across 46 inference providers in US Central, Tokyo, and São Paulo, detailing why geographic location dictates model selection and API edge architecture.

The answer up front

For real-time AI applications, do not select an inference provider based purely on advertised tokens per second. If your compute runs outside US Central or Tokyo, edge routing penalties wipe out hardware speed gains. Developers deploying in South America should route through OpenRouter or Groq, while Tokyo workloads benefit heavily from SambaNova or Fireworks AI. Skip Google Gemini Flash-Lite for latency-critical streaming due to high tail latency (p95 exceeding 5.6 seconds) and 89% to 91% success rates.

Methodology

This review evaluates the 30-day inference latency dataset published by engineer Max Bob on Dev.to, covering September 27 through October 26, 2026. The dataset tracks 1,190,063 HTTP time-to-first-byte (TTFB) probes across 46 inference provider endpoints sent every 5 minutes from probe nodes in US Central, Tokyo, and São Paulo. Additionally, 12,992 real streaming completions (max_tokens: 1) were executed to measure time to first token (TTFT) for Groq (openai/gpt-oss-120b) and Google (gemini-flash-lite-latest).

Single-source caveat: this review draws on Max Bob's published benchmark logs at https://dev.to/max_bob/fastest-ai-inference-api-in-2026-30-days-of-latency-data-for-groq-cerebras-sambanova-and-43-386k; independent benchmarks on our own test rig are pending. Re-testing will occur as provider network topology updates. Cerebras and SambaNova TTFT numbers were excluded by the author due to missing paid keys during the test window. German probe nodes were removed from the final dataset after identifying container CPU throttling that inflated TTFB timings by 100 ms.

What it does

Front-door network latency varies dramatically

Across 46 providers, HTTP TTFB measures edge routing efficiency without API authentication overhead. In US Central, Fireworks AI achieved a median network TTFB of 28 ms, while Google posted 47 ms and Groq registered 121 ms. In Tokyo, Fireworks led at 19 ms, followed by SambaNova at 25 ms and Google at 64 ms.

Geographic variance penalizes regional clients

Edge distribution collapses outside primary hubs. In São Paulo, OpenRouter delivered the lowest network TTFB at 61 ms, followed by Google at 155 ms and Groq at 234 ms. SambaNova exploded to 371 ms (a 15x slowdown compared to Tokyo), while Fireworks reached 252 ms. Placing API front doors without regional points of presence adds 200–300 ms before model execution begins.

Time to first token performance

For full TTFT streaming, Groq running openai/gpt-oss-120b achieved a median TTFT of 794 ms in São Paulo (234 ms network TTFB, 560 ms compute), 1,379 ms in US Central, and 1,497 ms in Tokyo, maintaining a 99.8% to 100.0% success rate. Google gemini-flash-lite-latest delivered faster network TTFB but suffered compute instability: median TTFT reached 1,204 ms in São Paulo, 1,579 ms in US Central, and 1,682 ms in Tokyo. Worse, Google posted p95 tail latencies between 5,602 ms and 6,103 ms, alongside success rates of only 89.7% to 91.6%.

What's interesting vs what's not

The data proves that network transport dominates user-perceived responsiveness more than vendor-reported tokens-per-second marketing. SambaNova's 15x latency difference between Tokyo (25 ms) and São Paulo (371 ms) highlights how un-cached API front doors bottleneck regional clients. Groq's specialized LPU compute path achieves lower compute TTFT from São Paulo (560 ms) than US Central (1,258 ms), indicating regional load-balancing dynamics that override raw geographical distance.

On the downside, the dataset lacks TTFT benchmarks for custom silicon providers Cerebras and SambaNova due to authentication limits, omitting key comparisons against Groq. Furthermore, testing max_tokens: 1 isolates initial time-to-first-token but ignores sustained generation throughput (tokens per second under extended prompt load).

Pricing

The dataset is publicly accessible without charge on Dev.to as of October 2026. Provider pricing for underlying models varies by vendor: Groq and Google charge per million input/output tokens based on their standard usage tiers.

Verdict

Select inference providers based on regional edge footprint rather than synthetic hardware benchmarks. For low-latency streaming in US Central and Tokyo, Fireworks AI and SambaNova offer superior network responsiveness. For global consistency and resilience against tail latency, Groq's infrastructure beats Google Gemini Flash-Lite, which suffers from severe p95 tail latency spikes above 5.6 seconds.

What we'd test next

In future benchmarking, we will deploy paid test rigs across Cerebras, SambaNova, Fireworks, and Groq to measure TTFT across matching open-weights models like Llama 3.3 70B. We will also benchmark sustained throughput under concurrent load to map queueing behavior under traffic spikes.

The investor read

This benchmark signals an impending consolidation in specialized AI inference hosting, where physical edge presence matters as much as hardware acceleration. Silicon vendors like Groq, Cerebras, and SambaNova have built formidable custom chip hardware, but without distributed global CDN front doors (like Cloudflare Workers or Fastly edges), international developers suffer massive TTFB penalties. The 15x latency difference for SambaNova between Tokyo and São Paulo illustrates that compute efficiency cannot overcome speed-of-light networking constraints. Moving forward, inference providers that partner with global edge networks or build multi-region API proxies will win enterprise spend over standalone chip cloud providers.

Pull quote: “Select inference providers based on regional edge footprint rather than synthetic hardware benchmarks.”