How a micro-SaaS founder audited zero AI crawler visibility and rebuilt indexation

An audit of openprofiles.io revealed zero citations across twelve target AI queries. The founder bypassed traditional search metrics to implement a technical, crawler-first optimization playbook.

The founder of openprofiles.io ran twelve search-enabled LLM queries to test his product's visibility and received zero citations. For his exact brand name, the model cited a dead, third-party .app domain returning a 404 error. For seven problem-space queries, platform help centers like OpenAI and Google captured the citations.

The diagnostic lay in his server logs. Over a 30-day period, GPTBot, OAI-SearchBot, and ChatGPT-User recorded zero hits, while ClaudeBot registered only four hits, all on unserved subdomains. Despite Google Search Console reporting the homepage as healthy and indexed, the micro-SaaS remained entirely invisible to LLM crawlers.

To address the indexing failure, the founder shifted his primary tracking metrics from traditional search impressions to crawler hits and cited-domain counts. This transition required a four-step technical intervention designed to make the site legible to LLM parsers.

Auditing the server logs

The diagnostic phase began with a direct inspection of the site's 30-day access logs. The founder searched for specific user-agent strings representing major AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, and ClaudeBot. The results showed that these crawlers were not failing to index the site; they were not attempting to visit it.

The only crawler activity came from ClaudeBot, which made four requests to unserved subdomains, resulting in 404 errors. The primary crawler for OpenAI, GPTBot, recorded zero hits. This log analysis demonstrated that standard Google indexing does not guarantee LLM crawler discovery. The models relied heavily on historical domain age and existing inbound link profiles, which allowed a defunct .app domain to claim the brand's primary citation.

Deploying machine-readable assets

The first remediation step was deploying an llms.txt file at the root directory. Previously, requests to openprofiles.io/llms.txt returned a 404 error. This file serves as a standardized, markdown-formatted directory designed to give LLMs a clean overview of a website's structure and content without requiring them to parse complex HTML.

In tandem with the llms.txt file, the founder added structured data schemas at the root directory. He implemented Organization and WebSite JSON-LD schemas. These standardized formats provide explicit clues about the meaning of a page, helping LLMs map the entity relationship between the brand name and the domain.

Eliminating client-side rendering barriers

The product's documentation and guides were originally built using client-side rendering. While standard search engines can execute JavaScript to index content, LLM crawlers often skip or fail to execute client-side scripts to save compute resources.

To resolve this, the founder pre-rendered four long-form guides on the server. He injected specific Article and FAQPage JSON-LD schemas directly into the pre-rendered HTML. This ensured that when an LLM crawler fetched the page, it received fully formed text and structured metadata in the initial payload. Finally, the founder manually requested indexing for each individual URL to force crawler attention.

What we would change

While the founder's playbook addresses the immediate technical barriers to LLM crawling, it relies on several assumptions that may not hold over time.

First, relying on manual URL submission is an unscalable tactic. For a micro-SaaS with a handful of landing pages, manual submission works. For any platform generating dynamic user profiles or programmatic SEO pages, this approach breaks down. A programmatic submission pipeline using indexing APIs is necessary to maintain indexation at scale.

Second, the introduction of an llms.txt file is necessary but insufficient. LLM providers have not standardized how they parse or prioritize these files. Simply hosting the file does not guarantee a crawler will read it or prioritize it over a high-authority domain like a platform help center.

To make this playbook viable, founders must focus on building external entity authority. The reason the dead .app domain won the brand citation is domain authority and historical backlink profiles. Technical on-page optimization cannot fully compensate for a lack of external citations. Founders should pair on-page JSON-LD with structured digital PR and external citations on high-authority platforms like GitHub, LinkedIn, and established directory sites. This creates the co-occurrence signals that LLMs use to verify that a domain is the authoritative source for a brand name.

The openprofiles.io audit highlights a shift in search engine optimization. Technical hygiene is no longer just about serving clean HTML to Googlebot; it requires structuring data specifically for resource-constrained LLM crawlers that default to the easiest path. For bootstrapped products, the immediate priority is verifying crawler access via server logs before assuming that standard search indexing translates to AI visibility.

The investor read

This case highlights a structural shift in search traffic distribution. As LLMs increasingly synthesize web data directly, standard SEO metrics like Google indexation are decoupling from actual user referral traffic. For investors, this signals that early-stage SaaS companies face a new discovery bottleneck: LLM crawler optimization. Startups that rely on organic search for distribution must treat LLM-readability as a core technical requirement rather than a post-launch optimization. Products like openprofiles.io that attempt to audit and solve this visibility gap represent an emerging category of LLM-SEO tooling, though their long-term defensibility remains unproven as search platforms continuously update their retrieval mechanisms.

Pull quote: “Despite Google Search Console reporting the homepage as healthy and indexed, the micro-SaaS remained entirely invisible to LLM crawlers.”