A layered observability pipeline for LLM inference from silicon to tokens

This review analyzes a comprehensive strategy for end-to-end observability in vLLM and TGI inference, detailing specific tools and metrics across six critical layers, from GPU hardware to business-level insights.

TL;DR

Best for: SREs, ML platform engineers, and observability engineers operating or preparing to operate vLLM or TGI on GPUs in production environments. Skip if: Your organization is not running large language model inference, or if your LLM serving infrastructure does not involve vLLM or TGI. Bottom line: Standard observability practices fail for LLM inference. A dedicated, layered pipeline correlating GPU silicon metrics with application and business signals is essential for diagnosing and optimizing performance.

Methodology

This v0 review draws on the founder's published claims in the article "End-to-End Observability for vLLM and TGI: from DCGM to Tokens" by devto, accessed on 2026-05-21. The article, authored by Samuel Desseaux, outlines a comprehensive tooling strategy for LLM inference observability. We cover the challenges of LLM serving, the proposed six-layer observability map, and the specific metrics and tools recommended for the GPU silicon layer. This review is based solely on the architectural and technical details presented in the source. What is not covered includes independent performance benchmarks, long-term workflow integration, or edge cases beyond those explicitly mentioned. Update cadence: This strategy will be re-evaluated when claims diverge from observed behavior in real-world deployments or when more complete implementation details become available.

What It Does

Why LLM serving breaks standard observability

The article begins by identifying four core properties of large language model inference that invalidate traditional observability playbooks. First, latency is not scalar; metrics like Time to First Token (TTFT), Inter-Token Latency (ITL), and end-to-end latency tell distinct stories. Optimizing one often degrades another, and a single p99 number is meaningless without context on input distribution. Second, batching is dynamic and preemptive, with continuous batching scheduling in-flight requests. This leads to non-linear, bursty relationships between queue depth and tail latency. Third, the KV cache is the real bottleneck, residing in VRAM and dominating memory pressure. Its filling leads to request preemption or rejection, which is invisible from CPU or network metrics. Finally, hardware reaches into the application; issues like degraded NVLink or thermal throttling directly impact the request queue, requiring silicon-level observability.

A six-layer map for correlation

The core of the proposed strategy is a six-layer observability pipeline designed to correlate a token rendered to a user with underlying silicon activity. The layers, from top to bottom, are: Business and cost (€/token, €/tenant, €/h GPU), API and distributed tracing (using OTel GenAI), Inference engine (vLLM, TGI with Prometheus), Container and OS (cAdvisor, kubelet, eBPF), CUDA runtime and collectives (NCCL, cuPTI), and GPU silicon (DCGM exporter, NVLink, PCIe). The value of this end-to-end stack comes from the ability to cross-reference signals across these distinct layers.

GPU silicon signals

For the foundational GPU silicon layer, DCGM exporter is presented as the correct entry point. The article highlights specific DCGM metrics crucial for early wiring: DCGM_FI_DEV_GPU_UTIL (a coarse indicator, insufficient alone), DCGM_FI_PROF_SM_ACTIVE (fraction of cycles with active warps on an SM), and DCGM_FI_PROF_SM_OCCUPANCY (average warps active per SM normalized to maximum). The article also mentions DCGM_FI_PROF_PIPE_TENSOR_ACTIVE as a fraction of cycles, but the source text cuts off before detailing its significance.

What's Interesting / What's Not

This article provides a highly valuable, opinionated framework for a complex problem. What's interesting is the explicit, detailed breakdown of why LLM inference observability is different. The four properties identified – scalar latency, dynamic batching, KV cache as bottleneck, and hardware-application coupling – are critical insights that many teams discover only after painful production incidents. The layered pipeline is a pragmatic, well-structured approach that acknowledges the distinct nature of signals at different abstraction levels, from raw silicon to business metrics. The emphasis on the KV cache as the single most informative signal on the engine layer is particularly insightful, highlighting a common blind spot in general-purpose observability. Naming specific tools like DCGM exporter, Prometheus, and OTel GenAI provides concrete starting points.

What's less developed, due to the article's truncated nature, is the how of correlating these signals across layers, especially beyond the GPU silicon. While the article states the value comes from cross-referencing, it does not elaborate on the mechanisms or best practices for doing so effectively. The review is also limited by the source text cutting off mid-sentence, preventing a full understanding of all proposed layers and their specific metrics. This leaves a gap in understanding the full scope of the recommended signals for the CUDA runtime, container, inference engine, API, and business layers.

Pricing

The article describes an architectural strategy for LLM inference observability using a combination of open-source tools (e.g., DCGM exporter, Prometheus, cAdvisor, eBPF) and industry standards (e.g., OpenTelemetry GenAI). As such, there is no direct pricing for the