Two-Tier AI Code Review Cuts CI Costs

A founder implemented a two-tier AI code review system to reduce CI costs. It uses a small local model for triage and a larger model with prompt caching for high-risk changes, significantly lowering expenses.

An engineer, identified as "devto," significantly reduced CI costs by implementing a two-tier AI code review system. Initial attempts to send all code diffs to a large frontier model resulted in unexpectedly high bills. The revised approach uses a cheaper model for initial triage, reserving the expensive model for critical changes, which reportedly dropped costs "hard."

Tier One: Cheap Triage Pass

The first tier acts as a gatekeeper, evaluating code diffs to determine if they warrant a more expensive, in-depth review. This classification task does not require a frontier model. The founder uses a small local model, specifically Ollama with qwen2.5-coder locally, suggesting a self-hosted small model or the cheapest cloud tier for CI environments. The model's role is triage, not comprehensive judgment.

The triage prompt is designed to be narrow, focusing solely on routing rather than reviewing. It instructs the model to output JSON with a risk level ("low" or "high") and a reason phrase. A diff is marked "high" if it touches specific sensitive areas: authentication, access control, money/token math, cryptography, SQL or shell string building, deserialization, file paths, or network calls. Changes related to formatting, comments, documentation, tests, renames, and configuration bumps are classified as "low" risk. This initial pass reportedly shortcuts the majority of pull requests, as most changes fall into the low-risk category, avoiding expensive API calls.

Tier Two: Big Model with Caching

When Tier One flags a diff as "high" risk, the relevant files are escalated to a more powerful, "strong" model for a detailed review. This is where prompt caching becomes critical for cost efficiency. The core insight is that much of the input sent to the large model remains constant across reviews. The system prompt, which defines the review rubric, severity definitions, and house rules, is identical for every request. Similarly, the repository context—surrounding files, conventions, and interface dependencies—is stable for many small pull requests touching the same code areas.

By structuring the request to prioritize stable elements, the founder leverages Anthropic-style caching, which keys on a stable prefix. The request is ordered as follows: cached system prompt, then cached repository context, followed by the fresh diff. This ensures that the full price for the stable prefix is paid only once, with subsequent reviews reusing the cached context at a fraction of the cost.

What We'd Change

The described system offers a pragmatic approach to cost control in AI-powered CI. However, its reliance on keyword matching for triage, while efficient, introduces potential blind spots. A subtle logic error in a low-risk file, or a novel attack vector not covered by the explicit keywords, could bypass the expensive review. Expanding the triage model's understanding beyond direct keyword hits to include semantic risk assessment, even in a lightweight manner, would enhance security without necessarily escalating costs to frontier model levels.

Implementing and maintaining local or self-hosted small models, as suggested for Tier One, requires specific infrastructure and MLOps expertise that not all teams possess. For smaller teams or those without dedicated DevOps resources, relying on the "cheapest cloud tier" might still incur unexpected costs or complexity if not carefully managed. Benchmarking different cloud-hosted small models for both cost and accuracy on a team's specific codebase would be crucial. Furthermore, the effectiveness of prompt caching is dependent on the LLM provider's implementation; not all APIs offer identical caching mechanisms or cost structures for prefix reuse. Teams should verify this capability with their chosen provider.

This two-tier strategy demonstrates a practical method for integrating AI code review without incurring prohibitive costs. It highlights the importance of architectural choices in managing AI expenses, particularly by segmenting tasks based on complexity and leveraging caching for recurring context. The approach offers a blueprint for teams looking to adopt AI-driven development tools while maintaining budget discipline. The efficacy of the triage prompt, and the magnitude of cost savings, depend on the specific codebase and development workflow. It provides a foundational strategy for optimizing AI resource allocation in CI. This system provides a clear pathway for teams to integrate AI-driven development tools while maintaining budget discipline.

The investor read

This signal points to a maturing developer tooling market where AI integration is shifting from experimental to cost-optimized. The tiered model approach, leveraging smaller, specialized models for triage and caching for larger models, indicates a growing focus on efficiency in AI inference. This creates opportunities for companies offering optimized AI APIs, prompt management solutions, or specialized, fine-tuned models for specific developer tasks like code classification. For indie and micro-SaaS founders, this playbook offers a pathway to integrate powerful AI capabilities without prohibitive cloud spend, making AI-driven features more accessible and sustainable for bootstrapped ventures. It suggests that capital will increasingly flow towards solutions that enable cost-effective, targeted AI application rather than just raw compute power.