DIY LLM distillation vs DeepSeek API: The $87,000 unit economics reality check

We analyze Thomson Nguyen's financial extraction benchmark comparing Claude Sonnet 5 distillation against DeepSeek V4.1 Flash API routing, revealing why DIY model training often loses to commodity APIs.

The extraction task at hand

The core engineering challenge was a structured extraction task: reading financial text to record who said what, what they claimed, and how certain the source was. Accuracy was critical. A wrong speaker or an invented certainty would propagate downstream as corrupted data. Claude Sonnet 5 served as the quality anchor. Running Sonnet 5 directly over the entire corpus of 6.5 million semantic atoms was estimated to cost $87,000.

To bypass this cost, the team planned a distillation pipeline. The plan was to use Sonnet 5 to label training examples, train a Qwen2.5-7B-Instruct student model, and run that student over the corpus. Kepler, an AI data scientist agent, built the training harness. The modeled cost for training and running this student model was roughly $1,000 in GPU time.

The API routing alternative

Before committing to the distillation run, Tycho, a FinOps and science agent, pushed the team to test off-the-shelf API models. The initial bake-off used a hand-built set of 23 difficult examples:

• Claude Sonnet 5: 21/23 (the teacher and quality reference)
• DeepSeek V3.2: 18/23 (the lower-cost API candidate)
• Qwen3-Next-80B-A3B: 8/23 (the open-weight candidate)

Initially, this supported the case for distillation. DeepSeek missed more hard cases than Sonnet 5. However, the team realized their extraction rubric was treating uncertainty as something to overcome. By adding "Unknown" as a legitimate output, they refined the instructions.

In a subsequent evaluation, DeepSeek V4.1 Flash with the corrected rubric passed 46/46 cases across three separate held-out sets. The optimized DeepSeek route modeled at roughly $1,000 for the extraction stage, matching the projected cost of the distilled student model.

What is interesting

The most compelling takeaway is that rubric optimization yielded a massive quality improvement without a single training run. Adding "Unknown" as a valid state allowed DeepSeek V4.1 Flash to achieve a perfect 46/46 on the held-out test sets. This highlights a common anti-pattern in AI engineering: rushing to fine-tune or distill a model before fully optimizing the prompt schema and system instructions.

Furthermore, the unit economics challenge the default assumption that hosting a small, distilled model is always cheaper than calling a commodity API. At $1,000 for both options, the DeepSeek API route eliminates the engineering overhead of maintaining a custom training harness, managing GPU instances, and handling cold starts.

What is not

The comparison is incomplete. The Qwen2.5-7B-Instruct student model was never actually benchmarked on the hard-case set, nor was it served over the full corpus. There is no honest student score to put beside DeepSeek's later result.

Additionally, both the $1,000 student GPU cost and the $1,000 DeepSeek API cost are modeled estimates, not actual invoices. The API estimate relies heavily on prompt caching. In production, if cache hit rates fluctuate, the API costs could easily exceed the fixed cost of self-hosting a distilled model.

Pricing snapshot

Cost projections as of September 2026 for processing 6.5 million semantic atoms:

• Claude Sonnet 5 API: $87,000 (estimated direct run)
• DIY Distillation (Qwen2.5-7B-Instruct): ~$1,000 (modeled GPU training and serving time)
• DeepSeek V4.1 Flash API: ~$1,000 (modeled API cost, assuming optimal prompt caching)

The final verdict

Skip DIY distillation if a commodity API like DeepSeek V4.1 Flash can meet your accuracy requirements after prompt and rubric optimization. For this 6.5-million-atom extraction task, the economics of distillation and commodity APIs are a wash at $1,000, but the API route spares you the engineering debt of maintaining custom training pipelines.

Only choose distillation if you have strict data-privacy mandates that prohibit third-party APIs, or if you require offline execution.

What we would test next

We need to run the distilled Qwen2.5-7B-Instruct student through the same 46-case held-out set to establish its actual performance baseline. We also need to stress-test DeepSeek V4.1 Flash's prompt caching under varying traffic patterns to see how quickly the $1,000 API estimate degrades when cache hit rates drop below 90%.

The investor read

This case study signals a critical shift in the LLM tooling market: the rapid commoditization of frontier-class reasoning and extraction capabilities. As models like DeepSeek V4.1 Flash drop API costs to near-parity with self-hosted open-weight models (both modeled at $1,000 for 6.5M atoms), the economic moat of proprietary distillation pipelines evaporates. Investors should be highly skeptical of startups pitching proprietary fine-tuned models as a core defense mechanism. Unless there is a strict data-sovereignty or offline requirement, the market is shifting toward dynamic API routing and prompt/rubric optimization. The value is consolidating in the orchestration and evaluation layers, not in the fine-tuned weights themselves.

Pull quote: “There is no honest student score to put beside DeepSeek's later result.”