How a 50-Line Python Router Cut an LLM Bill from $340 to $67

A developer reduced their AI API costs by 80% using a three-part playbook: a complexity router, a simple cache, and per-call cost logging. Here is the implementation and its limitations.

An LLM API bill dropped from $340 to $67 in one month. The developer reports this 80% cost reduction came not from complex infrastructure, but from a 50-line Python function. The function acts as a router, triaging user queries to the most cost-effective model for the job. It stops sending simple requests to expensive, high-capability models when a cheaper alternative is sufficient.

A rule-based complexity router

The core of the system is a function that classifies queries as "simple" or "complex." The author hardcoded this logic based on a few heuristics. A query is deemed simple if it meets at least two of three conditions: its length is under 15 words, it contains only one question mark, and it lacks keywords like "compare," "summarize," or "analyze."

Simple queries are sent to OpenAI's gpt-4o-mini, which the author prices at $0.15 per million input tokens. Complex queries are routed to Anthropic's claude-sonnet-4-6, priced at $3.00 per million input tokens. This is a 20x cost difference on input. The author claims this router alone cut the monthly bill by approximately 50%. You're paying Sonnet prices for return policy questions. That's the bug.

Adding a cache layer

The next optimization was caching. The author reports that roughly 30% of user queries were near-identical, such as "What are your hours?" To avoid paying for repeated LLM calls, a simple cache was implemented. For prototyping, this was a Python dictionary.

The system generates a SHA256 hash from the query and its context to create a unique cache key. Before routing a request to an LLM, the system checks if a response for that key already exists in the cache. If a match is found, the cached response is returned for free. The author notes that for a production environment, a shared cache like Redis would be necessary. This caching layer allegedly pushed the total savings from 50% to 80%.

Logging costs per call

The final component of the playbook is measurement. The author implemented a function to calculate the cost of each individual API call based on the model used and the number of input and output tokens. This provides real-time visibility into spending.

A dictionary stores the known costs for each model. After a call is completed and the token counts are returned by the API, the logging function calculates the exact cost. This allows for precise tracking of which types of queries are generating the most expense, enabling further optimization.

What we'd change

The heuristic-based router is clever for a proof-of-concept but brittle in production. Its rules are rigid. A simple user query that happens to contain the word "explain" would be misclassified as complex and routed to the more expensive model, negating the savings. The logic is also English-specific and would fail on other languages. A more robust solution would use a cheap, fast LLM (like Claude 3 Haiku or GPT-4o mini itself) to perform the classification, providing more nuanced routing.

The caching strategy is also a prototype. The author's in-memory dictionary cache only works for a single, non-restarting server process. Any real application with multiple server instances or deployments would find this cache useless, as it is not shared. The recommendation for Redis is correct but understated. This is a mandatory change for any production system.

Finally, the reported 80% savings is highly dependent on the application's specific query distribution. The author's success implies a high volume of simple, repetitive questions. A product that primarily handles complex, unique requests would see far smaller savings from this playbook.

Landing

This playbook's value is not in the specific Python snippets, but in the strategic shift it represents. It moves from treating LLM access as a monolithic capability to managing it as a tiered and expensive resource. The combination of routing, caching, and logging provides a basic framework for controlling the unit economics of an AI product. As API costs become a primary constraint on profitability, this level of financial discipline, even implemented simply, is what separates sustainable applications from expensive experiments.

The investor read

This tactic signals a market shift from prioritizing AI capability at all costs to optimizing unit economics. Early-stage AI companies demonstrating this cost discipline are more attractive and suggest greater operational maturity. The playbook illustrates the emergence of an "AI gateway" or "model routing" layer in the stack, a category currently being built out by venture-backed companies. For investors, a key diligence question for any AI-native product is its cost-of-goods-sold per user and the strategy to manage it. A founder implementing even a crude version of this playbook shows they are focused on building a profitable business, not just a technically interesting prototype. The claimed 80% cost reduction, while specific to one product's query distribution, makes a strong case for routing as a default architectural pattern for any application with variable query complexity.

Pull quote: “You're paying Sonnet prices for return policy questions. That's the bug.”