Benchmarking eight LLMs on sports trivia shows hallucination risks in structured data

A benchmark of eight frontier and small language models across 200 professional footballers demonstrates that zero-shot LLMs remain unsuitable for generating unverified relational databases due to high hallucination rates.

The answer up front

If you build software requiring precise relational data, do not use standalone zero-shot LLM prompts to populate your database. Even top performer DeepSeek V4.1 Flash hallucinated on 24% of generated club affiliations, while fast low-cost alternatives like Llama 4 Maverick and GPT-5.4 mini missed crucial player career steps. Use LLMs for initial extraction or drafting only, followed by deterministic validation against authoritative datasets.

Methodology

This evaluation synthesizes a single-source public benchmark conducted by the developer of ExtraTime, a football trivia platform. The test queried eight LLM models across 200 curated player profiles (50 each across EASY, NORMAL, HARD, and EXTREME difficulty tiers) representing 1,292 real-world club tenures. The developer queried all models via DigitalOcean serverless inference endpoints using identical JSON prompts containing full player names and dates of birth.

Scoring mapped model outputs against ExtraTime's reference database of 5,451 clubs and 2,337 alias strings, followed by manual review of 150 edge cases where 43 decisions were adjusted by hand. The v0 review draws on the founder's published claims at https://dev.to/remdore/i-asked-eight-models-which-clubs-footballers-played-for-11kh; independent benchmarks pending. Update cadence: re-tested when claims diverge from observed behavior.

What it does

Evaluation setup and dataset composition

The ExtraTime benchmark evaluated models against a balanced set of 200 professional footballers. To prevent entity ambiguity, each prompt supplied the player's full name and exact birthdate, such as Fabio Grosso born 1977-11-28. The benchmark provided an explicit exit clause instructing models to output {"known": false} rather than guessing uncertain facts. Outputs were evaluated across precision (percentage of predicted clubs that were correct), recall (percentage of actual historical clubs identified), and complete accuracy (producing every real club without single false additions).

Top precision performers

DeepSeek V4.1 Flash achieved the highest overall accuracy, recording 76% precision and 69% recall, successfully generating complete and perfect club histories for 50 out of 200 players. However, it required a median latency of 20.3 s and cost $0.526 across the 200 tests. Llama 4 Maverick delivered comparable precision at 75% with a dramatically faster 1.9 s median response time and $0.012 cost, though its recall dropped to 50%, completely missing minor career loans and early transfers.

Tiered efficiency models

GPT-5.4 mini generated balanced results with 62% precision and 58% recall at 1.3 s median latency and $0.060 total cost, while answering 188 of 191 nationality queries correctly. Claude Haiku 4.5 lagged behind on exact recall at 30% with 44% precision. Smaller open weights like Ministral 3 14B failed catastrophically, scoring 14% precision and generating false club transfers in 86% of its output.

What's interesting / what's not

Precision tradeoffs in production pipelines

The benchmark proves that explicit "do not guess" system instructions do not stop hallucination in long-tail fact retrieval. Even when provided with explicit failure permission, models chose to fabricate plausible club transfers rather than declining to answer. For instance, Gemma 4 31B declined 48 players but still suffered a dismal 49% precision on the players it attempted.

What is notable is Llama 4 Maverick's position on the performance curve. While DeepSeek V4.1 Flash offers the highest raw recall, its 20.3 s latency makes it unusable for interactive workflows. Llama 4 Maverick delivers 75% precision in 1.9 s at $0.012 for 200 requests, making it a far superior candidate for preliminary extraction when combined with automated downstream filtering.

What is missing from LLM capability claims is autonomous reliability for ground-truth database creation. Not a single model achieved higher than 25% complete historical accuracy across the full 200-player dataset.

Pricing

DigitalOcean serverless inference total cost for all 1,600 API queries (8 models across 200 players) was $1.08 as of October 2026. Model pricing per 200 players ranged from $0.007 for Ministral 3 14B and $0.012 for Llama 4 Maverick up to $0.060 for GPT-5.4 mini, $0.081 for Claude Haiku 4.5, and $0.526 for DeepSeek V4.1 Flash.

Verdict

Zero-shot LLM generation is unviable as an unmonitored source of truth for factual databases. For founders building domain-specific applications, models should be used strictly as parsing engines or candidate generators sitting behind human review or deterministic database validators. If speed and cost matter, Llama 4 Maverick is the clear developer pick for candidate extraction, providing 75% precision at under 2 seconds per query.

What we'd test next

In future benchmarks, we would test whether retrieval-augmented generation (RAG) using structured Wikidata dumps resolves the 24% error rate seen in DeepSeek V4.1 Flash. We would also evaluate fine-tuning Llama 4 Maverick on synthetic career path schema to determine if fine-tuning reduces hallucinated transfers while maintaining sub-2-second response times.

The investor read

This benchmark highlights a persistent moat for curated domain-specific structured datasets over raw LLM retrieval. Startup founders attempting to replace human-verified reference databases with zero-shot model queries will face unacceptable error rates in long-tail entity lookup. Expect enterprise tooling spend to favor hybrid extraction pipelines, specifically small and ultra-cheap models like Llama 4 Maverick paired with deterministic validation graph databases, rather than expensive frontier models.

Pull quote: “Zero-shot LLM generation is unviable as an unmonitored source of truth for factual databases.”