A 2,430 dollar AI agent protocol refactored 189 files without human review

A newly published case study details how a specification-first protocol allowed an AI agent to dismantle a core architectural invariant across a 717k-line codebase in three days.

A newly published case study details how an AI coding agent dismantled a core lifetime invariant across a 717,725-line production TypeScript codebase in three days. The refactor touched 189 files, required 34,770 insertions and 16,422 deletions, and cost $2,430 in API fees. Crucially, the operation succeeded with zero human code review and no pre-existing test oracle to validate the new behavior.

The author of the paper, published on Hugging Face, had previously assessed this architectural shift as effectively infeasible through incremental human refactoring. Instead of a manual rewrite, the author used a strict, multi-phase "specification-first" protocol. This framework relies on 14 specification refinement cycles and 17 post-implementation verification cycles to force mathematical convergence.

Freezing the behavioral specification

The protocol began not with code generation, but with formal specification. The agent analyzed the existing 3,648-file codebase to draft a comprehensive behavioral document. This document defined how a streaming generation should survive the closing of its UI panel and reattach upon reopening without losing or duplicating data.

To ensure accuracy, the agent ran 14 refinement cycles. Each cycle audited the draft specification directly against the production source code, identifying discrepancies between the agent's assumptions and actual system behavior. Only when the specification accurately mapped to the codebase was the document frozen. This frozen specification served as the single source of truth, replacing the need for a traditional test oracle.

Executing the atomic implementation

With the specification locked, the agent executed the code changes. The implementation phase operated in a tight compile and test feedback loop, focusing on atomic modifications. The refactor was massive, ultimately touching 189 files directly and generating 31 new ones. When including the initial extraction phase, the work spanned two commits totaling 288 files.

The agent wrote 34,770 insertions and deleted 16,422 lines of code. Throughout this phase, no human reviewed the code lines or ran manual tests. The agent relied entirely on automated compiler feedback and basic syntax checks to ensure the codebase remained structurally sound during the massive rewrite.

Auditing to empirical convergence

The core of the protocol lies in the 17 verification cycles that followed the implementation. In these cycles, the agent audited the newly written code against the frozen specification. This was not a test suite execution, but a static, semantic audit of the code's logic against the defined behavioral rules.

Across the combined 31 audit passes, which include 14 refinement and 17 verification cycles, the agent identified and corrected 201 defects before any human executed the program. The author established a strict empirical convergence criterion: the protocol only ended when two consecutive verification passes returned exactly zero findings. Once achieved, the software behaved exactly as specified across dozens of subsequent sessions, with no observed bugs.

What we would change

While the $2,430 cost and three-day turnaround are highly attractive, founders must recognize the structural limitations of this case study. First, the entire 1,500-page log of raw sessions was generated in French. Language models often exhibit different reasoning densities and token efficiencies depending on the language used. Replicating this exact protocol in English or other languages may yield different token costs and convergence rates.

Second, the protocol lacks a traditional safety net. Relying on an AI agent to audit its own code against a specification it drafted introduces a high risk of confirmation bias. If the agent misinterprets a subtle dependency during the initial 14 refinement cycles, that error will be baked into the frozen specification and ignored during the 17 verification cycles.

For a production system, we would modify this playbook to introduce an independent "red team" agent. This second agent, running a separate model or system prompt, would be tasked solely with finding discrepancies between the generated code and the frozen specification. Relying on a single agent instance to act as both creator and auditor is an unnecessary risk when API calls are cheap relative to production downtime.

Landing

The paper's 1,500-page appendix provides a rare, fully instrumented look at agentic engineering at scale. It proves that massive, systemic codebase changes do not always require manual rewrites or dense test suites if a strict mathematical convergence protocol is enforced. For technical founders managing legacy technical debt, the lesson is clear: stop asking agents to write code incrementally. Instead, invest the time to build a multi-cycle verification loop that treats the specification, not the test suite, as the ultimate source of truth.

The investor read

This case study signals a fundamental shift in how software equity should be valued. Historically, a 717k-line codebase represented a significant moat, but also a massive liability due to technical debt and refactoring costs. If agentic protocols can reliably execute deep architectural changes for less than $2,500, the maintenance cost of legacy software drops by orders of magnitude. For investors, this shifts the focus from how much code has been written to how well-specified the system architecture is. Startups using these protocols can maintain high development velocity without hiring massive engineering teams to manage technical debt. The investability of a software product will increasingly depend on its architectural modularity and the team's ability to orchestrate automated verification pipelines, rather than raw developer headcount.

Pull quote: “The refactor touched 189 files, required 34,770 insertions and 16,422 deletions, and cost $2,430 in API fees.”