A Kubernetes service reported 100% health while serving 18% errors

An engineer details the command-line process for isolating a faulty pod when Kubernetes readiness probes lie, turning a green dashboard into a red herring for a critical checkout API.

An 18% error rate crept into a checkout API over twenty minutes on a Tuesday morning. The author of a technical post-mortem reports that while one in five user requests were returning a 500 error, every monitoring dashboard and Kubernetes control plane command showed the service in perfect health. All five pod replicas were listed as Running and Ready.

This scenario details the diagnostic process for an outage where the system's own health reporting is the primary obstacle. Green is not a fact. Green is a claim. The playbook here is about verifying that claim at the lowest possible level, bypassing the abstractions that are providing false assurance.

Verify the error, then the replicas

The first step was confirming the error was real and not a monitoring glitch. The author ran a curl command in a loop against a production endpoint, which confirmed that approximately one in five requests failed with a 500 status code. This pattern suggested the issue was not systemic but isolated to specific replicas. A load balancer was distributing requests evenly across a set of pods, some of which were healthy and some of which were not.

The initial check on the pods themselves, using kubectl get pods -n checkout -o wide, returned a deceptive all-clear. All five replicas showed Running status, 1/1 containers ready, and zero restarts. This is the central conflict of the incident: user experience was degraded, but the platform's own orchestration layer reported nominal performance.

Check the control plane's claims

If the pods report as healthy, the next logical component in the chain is the Service, which routes traffic to them. The author checked the service endpoints with kubectl get endpoints checkout-api -n checkout. This command confirmed that Kubernetes considered all five pod IPs to be valid, active targets for receiving traffic. The Service was correctly configured based on the (faulty) information it was receiving from the pods' readiness probes.

The control plane was operating as designed. It was receiving a “healthy” signal from all pods and therefore including all of them in the load balancing pool. The problem was not with the routing; it was with the signal itself.

Isolate pods by bypassing the service

The critical tactic was to bypass the Kubernetes Service abstraction and interrogate each pod directly. The post details using kubectl port-forward to connect directly to a specific pod's IP and port from a local machine. This allows an engineer to send test traffic to one replica at a time, isolating the faulty component from the group.

By port-forwarding to pod/checkout-api-6c8b9-x2k7p and then sending a curl request to localhost, the author could test that single pod's response. Repeating this process for each of the five pods would definitively identify which ones were returning 500 errors despite reporting as Ready.

What We'd Change

The playbook described is a competent, reactive debugging process. It effectively isolates a fault. The primary flaw, however, is that it addresses the symptom—a bad pod in rotation—without addressing the root cause: a readiness probe that provides false positives.

A readiness probe should answer the question, “Is this application replica actually ready to serve traffic?” A probe that simply checks if a web server process is running is insufficient. A robust readiness probe for an API like this one would perform a shallow, non-destructive check of its critical dependencies. It might execute a read-only query to its database, check its connection to a required cache, or call a health endpoint on a critical downstream service. This ensures the pod is not just running, but capable of performing its function.

Furthermore, this manual debugging process does not scale. For a larger team or a more critical service, the solution is systemic, not heroic. This includes implementing more sophisticated probes, building detailed runbooks for this specific failure mode, and using observability platforms that can correlate the increase in 500 errors with specific pod IPs, automating the initial steps of this investigation.

Landing

The core lesson is not a specific kubectl command, but an operational discipline of skepticism. A green dashboard is a starting hypothesis, not a conclusion. When user reports conflict with monitoring, trust the user reports. The most challenging outages are frequently not catastrophic failures, but subtle degradations masked by abstractions. The skill is in knowing how and when to dismantle those abstractions to see the underlying reality.

The investor read

This incident highlights the operational fragility that can exist even within modern, containerized infrastructure. For investors, it's a signal of the technical debt and operational maturity of a company. A team that frequently relies on manual, command-line heroics to debug production may lack the robust engineering practices needed to scale reliably. This specific failure mode—misleading health checks—is common in small to mid-sized teams adopting complex tools like Kubernetes without deep expertise. It also validates the market opportunity for observability and AIOps platforms that can automate this kind of root cause analysis, moving teams from reactive debugging to proactive system hardening. A product that can't stay online can't grow.

Pull quote: “Green is not a fact. Green is a claim.”