Building a reliable AI search monitor requires filtering non-deterministic LLM retrieval

A founder built a 300-line script to track product mentions in search-augmented AI answers. Six technical pitfalls revealed why conventional keyword rank tracking fails in conversational search.

The founder behind DOS AI, a customer messaging tool, built a 300-line script to audit whether AI search engines recommended his software. The script ran a battery of 35 buyer queries through a web-search-enabled large language model, costing roughly $1 per execution across four test runs.

Early outputs generated metrics that appeared clean on paper but failed under technical scrutiny. Brand queries logged false positives by counting name echoes, silent CLI parameter errors presented as zero-visibility reports, and publishing campaigns showed zero movement due to unindexed third-party hosts. Measuring generative engine optimization requires programmatic validation rather than simple string matching.

Structuring the prompt battery

The audit uses 35 prompts across seven defined categories: service selection, task resolution, pricing queries, agency recommendations, direct brand queries, Kazakh-language questions, and English-language questions. Thirty-two of the 35 prompts contain no explicit reference to the product name.

Each query fires against a model with web search toggled on and the approximate location pinned to the targeted city. For each response, the script logs the raw text and parses cited URLs to tally domain appearances against competitor citations.

Filtering false positives and silent failures

The initial script classified any response containing the string "DOS AI" as a successful mention. When prompted with "What is DOS AI?", the model defined the term as legacy MS-DOS software or a denial-of-service attack. The string matched, but the product was not recommended. The founder reports that two out of three logged hits in the baseline run were false positives.

To fix this, the evaluation logic was updated: for brand-specific prompts, an answer only counts as a hit if it includes a direct citation to the company's domain.

A parallel issue surfaced in command-line argument handling. Passing an empty --limit flag resolved to NaN, causing array slices to return empty sets. The script finished cleanly with "named in 0 of 0," which mimicked a total lack of market visibility. The updated logic throws exit code 2 on invalid parameters and exit code 1 when zero queries evaluate.

Accounting for search non-determinism

Executing the identical 35-prompt test twice within four minutes produced divergent outputs. In the first run, the model answered from internal parameters. In the second run, it executed a live search, identified an unrelated firm sharing the brand name, and cited the competitor's pricing as the product's own.

Between 31% and 48% of queries in the test battery triggered no live web search at all, relying strictly on static training data. On these queries, newly published content has zero impact on output. To eliminate variance, the founder instituted a strict rule: single-run score shifts are treated as background noise. A visibility change is recognized only when unbranded counts shift or when three consecutive runs demonstrate identical movement.

Expanding geo-targeting and multi-model sampling

Relying on a single static geographic location ignores how search-augmented LLMs route queries through distributed edge networks. Regional intent flags and localized retrieval caches yield vastly different vendor recommendations for buyers in adjacent municipalities. The audit script should parameterize IP proxies across at least three target markets.

Testing a single LLM vendor fails to capture the fragmented landscape of AI search. OpenAI, Perplexity, and Google Gemini run distinct retrieval pipelines, indexing schedules, and citation rules. A strategy that achieves high visibility in one model often yields zero mentions in another due to variations in crawler access and domain authority scoring.

Validating crawler access upstream

Checking HTTP response headers for noindex and nofollow directives solves third-party indexing traps, but it does not account for active AI crawler blocks. A growing number of platforms permit traditional search engine indexing while serving explicit block headers to AI scraper user-agents. The script should issue HEAD requests using specific AI crawler user-agent strings, such as GPTBot and PerplexityBot, to verify actual indexability.

Tracking product visibility in conversational search engines requires treating LLM outputs as non-deterministic system responses rather than static rank lists. Simple keyword matching and single-run snapshots create phantom visibility metrics that distort marketing allocation. Founders auditing generative search channels must isolate live retrieval from parametric memory, enforce strict programmatic validation, and measure directional trends across multiple test executions.

The investor read

Monitoring Generative Engine Optimization (GEO) remains a highly fragmented, unstandardized tooling layer. The founder's experiment highlights why early enterprise GEO monitoring tools struggle with retention: non-deterministic LLM sampling and variable web-search triggers make point-in-time rank tracking functionally meaningless. For venture investors, products building in the AI search optimization space must move beyond passive keyword monitoring and develop defensible attribution methodologies that map LLM citations directly to downstream pipeline conversion. Until standardized APIs exist for conversational search impression data, bootstrapped tools built for internal operations reflect the nascent state of the market.

Pull quote: “Between 31% and 48% of queries in the test battery triggered no live web search at all, relying strictly on static training data.”