How a $0.03 silent failure prompted a 131-test AI evaluation harness

An AI agent passed all unit tests but failed in production due to semantic drift. The fix was an 'eval-first' approach, building a comprehensive test harness before shipping new features.

An AI agent designed to pull sales reports was failing silently. It cost a claimed $0.03 per run in Groq and Oracle Cloud fees while passing all traditional unit tests. When a user requested "last month's sales figures," the system returned "no data found," even though the data existed.

The failure was not a code bug. It was a semantic mismatch between the LLM's interpretation of "last month" and the database schema's date logic. Developer Elena Revicheva documented the incident, arguing it reveals a fundamental weakness in standard testing for AI systems. Her solution, reported in a post on her blog AIdeazz, was to build a 131-test evaluation harness before writing new features.

The $0.03 semantic mismatch

The agent's task was to translate natural language into a database query. For the request "last month's sales," the LLM correctly identified the intent and date range. The query generator then translated "last month" into a SQL BETWEEN clause for a full month's date range.

The problem was the database schema. It stored sales data against a report_date field representing the end of the reporting period. The correct query needed a specific date, not a range. Each component passed its unit test in isolation. The LLM's extraction was correct. The query generator's logic was correct. The database was correct. The system-level failure emerged only when these correct components interacted, a blind spot for conventional testing.

Why unit tests fail for AI agents

Revicheva argues that unit tests are fundamentally unsuited for validating AI agents for three primary reasons. First, they cannot test for emergent behavior. A unit test cannot predict how Agent A's output, slightly perturbed by a new LLM version, will affect Agent B's subsequent action. The interaction between agents or components creates outcomes that isolated tests cannot predict.

Second, they cannot detect semantic drift. An LLM's interpretation of a phrase can shift with model updates, or a downstream system's interpretation might differ, as the "last month" example shows. Third, unit tests cannot cover the long tail of user intent. They are written for expected inputs, while users will inevitably phrase requests in unanticipated ways.

Adopting an 'eval-first' harness

The proposed solution is to shift from unit testing components to evaluating the entire system's behavior. Revicheva reports building a 131-test evaluation harness before adding new features. This "eval-first" approach treats the agent as a whole. It tests whether the final output meets the user's intent across a wide range of inputs, rather than verifying the internal logic of each piece. This harness becomes the gate for shipping updates, ensuring that changes to the prompt, model, or code do not cause regressions in system-level performance.

What we'd change

The post effectively diagnoses the problem but is light on the prescription. It advocates for a 131-test harness without providing a detailed breakdown of the tests themselves. To be actionable, a founder would need examples: what proportion of tests cover date logic, entity extraction, or handling ambiguous queries? The playbook is a "why-to," not a "how-to." The source does not provide the code or a list of the evaluations, which limits the tactic's immediate replicability.

The cost justification of $0.03 per run is also a weak anchor. While it provides a concrete number, the more significant cost of silent failure is the erosion of user trust and the potential for cascading errors in a multi-agent system. Framing the harness as a tool for maintaining product reliability and user confidence is a stronger argument than focusing on marginal inference costs.

Landing

The core insight is a necessary shift in testing philosophy for AI products. Validating an LLM-based system is less about code correctness and more about behavioral consistency. An evaluation harness codifies the expected behavior of the system against a set of known inputs and intents. This is critical because the system's logic is not entirely contained in its code; it resides partially within the weights of a third-party model. Building an eval harness first is a defensive measure against this external dependency, ensuring that model updates or prompt tweaks do not silently break the product.

The investor read

This tactic highlights a key operational risk in AI agent development: silent, compounding failures that erode user trust and burn capital. An 'eval-first' approach signals a startup's maturity beyond simply wrapping a model API. Companies that can demonstrate robust, automated evaluation frameworks are more investable because they have a mechanism to de-risk product reliability, especially when dependent on third-party model updates. An investor should view a comprehensive eval harness as a competitive moat. It indicates a team can scale an AI product without it constantly breaking in subtle, user-facing ways, separating serious builders from hobbyists.

Pull quote: “A unit test cannot predict how Agent A's output, slightly perturbed by a new LLM version, will affect Agent B's subsequent action.”