Debugging LLM Benchmarks: Nine Hidden Bugs

A developer's deep dive into an A2A agent benchmark reveals nine distinct technical flaws across four languages, highlighting the hidden complexities of LLM integration and performance measurement.

The founder behind a2a-benchmark discovered their agent performance comparison suite consistently halted at N=22 for Python, despite being designed to sweep N from 1 to 24. This was not a deliberate design choice, but a crash silently masked by a truncated output chart. The subsequent debugging process uncovered nine distinct technical flaws across four programming languages and their integrations with large language models.

Python's Hidden Integer Limit

The primary culprit for Python's N=22 failure was a default CPython 3.11 limit on int to str conversion, capped at 4,300 digits. The 24th Mersenne prime, 2^19937-1, contains 6,002 digits. The Python agent stringified every prime it found, triggering this limit. The founder reports the stringified list was never actually returned, making the str() call decorative. Removing this str() operation resolved the crash, allowing Python to report data for N=24 at 2,425.9 ms.

LLM Output Parsing Failures

The benchmark's harness relied on a regular expression, r"It took ([\d\.\-e]+) seconds", to extract elapsed time from the LLM's prose. However, the Gemini model never used the phrase "It took." Instead, it generated phrases like "Calculating the first 5 Mersenne primes took…" or "The calculation took…". This meant the primary parsing path for Python's data was effectively broken, relying solely on a fallback that read structured tool artifacts. The founder highlights this as a critical flaw: "My measurement pipeline's primary path was a bet on a language model's phrasing habits."

Agent Reporting and Formatting Errors

The Node and Rust agents exhibited their own issues. When asked for 100 Mersenne primes, they reported "Found first 100 Mersenne primes" but only computed 26, the size of their internal exponent table. Furthermore, these agents formatted elapsed time as %.2f ms. For fast runs, this resulted in 0.00ms, which parsed to zero. This zero value rendered those data points invisible or incorrect on a log-scale chart, effectively hiding valid, fast results. The benchmark's chart title itself was truncated, reading "A2A Round-Tri" instead of the full "A2A Round-Trip Latency."

What We'd Change

The debugging story of a2a-benchmark reveals common pitfalls in building and measuring LLM-integrated systems. Founders should prioritize structured data outputs from LLMs and agents. Relying on natural language parsing, as seen with the r"It took..." regex, introduces fragility. Explicit JSON or other machine-readable formats for critical metrics are more robust than attempting to parse prose.

The Python int to str limit and the 0.00ms formatting highlight the danger of silent failures. Decorative code or seemingly innocuous formatting choices can lead to crashes or data misrepresentation. Implementing comprehensive end-to-end integration tests that validate not just functionality but also data integrity and edge cases (like very large numbers or very small times) is crucial. These tests should cover the entire pipeline, from agent execution to data storage and visualization. Furthermore, agents should report their actual computation scope. Node and Rust claiming "Found first 100" while only computing 26 is a data integrity issue that can mislead users and invalidate benchmarks. Clear, accurate reporting of what was actually computed is non-negotiable for reliable systems.

Landing

The a2a-benchmark experience underscores that LLM-powered applications are, at their core, distributed systems. They combine language models, custom agents, and diverse programming environments, each with its own quirks and limitations. Thorough debugging in such an environment demands a multi-layered approach, scrutinizing not only the LLM's output but also language runtimes, data serialization, and reporting logic. The goal is to ensure the system measures what it claims to measure, without hidden assumptions or silent failures.

The investor read

This detailed debugging exercise in a2a-benchmark highlights the significant technical overhead and fragility inherent in building and benchmarking agent-to-agent (A2A) LLM systems. The prevalence of silent failures, parsing errors, and language-specific runtime quirks suggests that the tooling and best practices for reliable LLM integration are still nascent. For investors, this signals a high technical bar for early-stage LLM agent startups, requiring deep engineering expertise beyond just prompt engineering. Robust infrastructure, comprehensive testing, and structured output adherence will be critical differentiators. Products that simplify reliable benchmarking and integration for LLM agents could capture significant value, as the current landscape demands extensive custom debugging, making many A2A plays more suitable for bootstrapped, technically-intensive teams rather than venture-scale operations without significant infrastructure investment.

Pull quote: “My measurement pipeline's primary path was a bet on a language model's phrasing habits.”