Benchmark shows AI agent context costs vary 24x depending on the task

A detailed, 936-run benchmark across four context-providing methods (grep, graph, LSP) reveals that the architectural choice has massive cost and accuracy implications for complex AI coding agents.

THE ANSWER UP FRONT

For founders building AI coding agents, this benchmark provides a clear directive. If your agent only performs simple lookups like "go to definition," a basic grep approach is functional, though it will cost you roughly 3x more in tokens than a structural tool. For any complex task involving code understanding, like estimating the blast radius of a change or finding all implementations of an interface, grep is a catastrophic choice. It fails on accuracy and costs between 6x and 24x more than graph-based or LSP-based context providers. The bottom line: for high-value agentic work, investing in a structural code intelligence backend is not optional.

METHODOLOGY

This v0 review analyzes the methodology and results published by the user neko1313_4 on dev.to on June 25, 2026. The source is a single, detailed blog post titled "How much does context cost an AI coding agent? grep vs graph vs LSP, measured across 936 runs." The results are based on the author's claims and published data; we have not independently reproduced the 936 test runs. Update cadence: this review will be updated if the benchmark harness is made public and we can verify the results.

The review covers the author's experimental design, which compares four distinct methods for providing code context to a single AI agent (Claude Code). The testbed was the apache/superset repository, a large, polyglot codebase. The four methods, or "arms," were: filesystem (grep + read), graphlens (a custom structural code graph), serena (an LSP-based provider), and codegraph (an off-the-shelf code graph product). The experiment measured accuracy and token cost across 26 distinct tasks, categorized as simple or complex. This review does not cover the implementation details of graphlens or serena, nor does it include latency measurements, which were not part of the original post.

WHAT IT DOES

The architectural choice with a price tag

The author frames a common problem for developers building AI agents: how do you give an agent awareness of a large codebase without feeding it the entire repository? The context provider is the component that answers the agent's questions about the code. The post measures the performance and cost implications of four different architectural answers to this question.

Four approaches to code context

The experiment isolates the context provider as the single variable. The four arms represent the dominant strategies available today:

1. filesystem: The simplest approach. The agent uses grep to search for strings and read to open files. It requires zero setup but is text-based, not structural.
2. graphlens: A custom-built tool that parses the repository into a structural code graph with typed nodes and edges. This allows for precise queries like "who calls this function?"
3. serena: A tool built over the Language Server Protocol (LSP), the same technology that powers IDE features like "go to definition" and "find references."
4. codegraph: An off-the-shelf, commercial code graph product, treated as a black box in the test.

A controlled, task-based experiment

The benchmark's strength lies in its controlled design. Using the apache/superset repository (~400k LOC, Python/TypeScript), the author ran the same 26 tasks through the same agent (Claude Code, with haiku, sonnet, and opus models). The only thing that changed between runs was which of the four tools provided the context. The tasks were split into simple lookups ("where is X defined") and complex analysis ("find every override of this method"). This design allows for a direct comparison of how each context strategy affects cost and accuracy.

WHAT'S INTERESTING / WHAT'S NOT

The most significant finding is how performance flips depending on task complexity. On simple definition lookups, all four tools achieved similar accuracy. The only differentiator was cost, where grep was about 3x more expensive in tokens. If the analysis had stopped there, the conclusion would have been that grep is inefficient but viable.

The data on complex tasks tells a completely different story. Here, the grep-based approach collapses. The author reports its accuracy drops to 0.71, with 17% of runs failing to complete at all. For the runs that did finish, the token cost was 6x to 24x higher than the structural tools. This is not an incremental difference; it's a phase change. It demonstrates that for any task requiring genuine code understanding, string matching is the wrong primitive.

This provides an unambiguous, data-backed heuristic for founders. Building an agent that can perform complex refactoring or deep analysis on a grep foundation is a technical dead end. The initial simplicity of avoiding a code graph or LSP setup is a false economy that is paid for with every complex query in token costs and incorrect answers. The author's work effectively quantifies the value of semantic code intelligence infrastructure.

PRICING

The post does not detail the pricing of the tools themselves (like codegraph). Instead, it measures the cost to operate each approach, denominated in LLM tokens consumed to complete a task. The key finding is the relative cost:

• Simple Tasks: grep is ~3x more expensive than structural tools.
• Complex Tasks: grep is 6x to 24x more expensive than structural tools.

This analysis was performed in June 2026, and relative costs will change as LLM token prices evolve, but the multiplicative factor is the critical insight.

VERDICT

This benchmark is a valuable contribution for anyone building on top of code-generating LLMs. It replaces architectural intuition with hard numbers. Our recommendation is clear and aligns with the data. For hobbyist projects or simple agents that only fetch definitions, a grep-based context provider is a valid, if inefficient, choice. For any commercial or serious open-source agent designed to perform complex analysis, automated refactoring, or impact assessment, a structural approach using a code graph or LSP is mandatory. The benchmark proves that failing to make this investment upfront leads to a product that is both cripplingly expensive to run and unreliable.

WHAT WE'D TEST NEXT

For a v2 analysis, we would first seek to reproduce these results on the author's harness. We would then expand the scope to test if the findings generalize. This includes running the same tasks on a different large, polyglot repository to control for any quirks in the apache/superset codebase. We would also test other popular agents, such as OpenAI's GPT series or Google's Gemini, to see if a more capable model can compensate for a poor context provider. Finally, we would add latency (wall-clock time) as a key metric, as token cost and accuracy are only two parts of the user experience equation.

The investor read

This benchmark provides a clear map of the AI devtool market's technical moats. The high-value segment is not simple code completion; it's complex, whole-repo automated engineering tasks. This analysis demonstrates that competing in that segment requires significant investment in proprietary code intelligence infrastructure (code graphs, custom LSPs). A startup pitching a sophisticated refactoring agent built on a simple grep architecture is a major red flag, as their COGS will be non-competitive and their accuracy will be low. Conversely, a team with a demonstrably faster or more insightful way to build and query code graphs is a 'picks and shovels' play on the entire agent market. The value is in the infrastructure that makes agents smart, not just the agent itself.

Pull quote: “For any complex task involving code understanding, like estimating the blast radius of a change or finding all implementations of an interface, grep is a catastrophic choice.”