Build a Deterministic AI Content Quality Gate for Under 50ms

A Reddit founder developed a hybrid AI content quality gate, combining regex and heuristics for speed and a final LLM pass for nuance. This approach processes text in under 50ms, offering a repeatable tactic for content tool builders.

A founder operating under the Reddit handle Competitive-Fun-7148 developed a deterministic AI content quality gate that processes text in under 50ms, integrating it directly into the generation loop. This system was designed to address common AI content tells, such as uniform sentence length and specific vocabulary, before the user sees a draft. The goal was a low false-positive rate and high utility for an internal content tool.

Problem of AI Content Tells

The core problem identified was the consistent presence of specific patterns in AI-generated text. These include high em-dash density, a lack of sentence length variance, and a reliance on particular vocabulary (e.g., "leverage," "delve," "navigate," "tapestry"). Other tells involved structural elements like mechanical parallelism and predictable opening or closing phrases. The founder sought a solution that could detect these issues deterministically and quickly, avoiding the latency and cost associated with LLM-only checks.

Hybrid Architecture for Speed and Nuance

The architecture implements a two-stage review process. First, a streamText function generates content in steps. During each step, an onStep hook triggers a runDeterministicGate function on the partial text. This gate assigns a 0-100 quality score. If the score falls below a set threshold (70 in this case), a tool_call('revise', { feedback }) is initiated, providing specific instructions like "4 instances of 'leverage', 3 consecutive sentences over 25 words, em dash density 2.3 per 100 words." After the full generation, an onFinish hook runs a second, more nuanced runLLMReview pass on the final output. This hybrid approach leverages the speed of deterministic checks for initial filtering and the contextual understanding of LLMs for final refinement.

Eight Deterministic Detection Categories

The deterministic gate employs plain regex and heuristics across eight categories to identify AI tells. These include: 1) AI vocabulary, which flags specific wordlist matches; 2) Sycophantic openers, detecting phrases like "Great question!" at sentence position zero; 3) Generic conclusions, identifying common closing patterns; 4) Mechanical parallelism, flagging three-element lists with abstract nouns; 5) Metronome rhythm, which checks for sentence length variance below a human-like burstiness threshold; 6) Hedge stacking, identifying multiple qualifiers in a single phrase; 7) Em dash density, flagging more than a specified number of em dashes per 100 words; and 8) Format compliance, checking for character limits or incorrect hashtag counts. Each category contributes to the overall quality score.

Performance and Dogfooding Results

The primary advantage of the deterministic gate is performance: it runs in less than 50ms, compared to approximately 3 seconds for an LLM pass. This speed allows it to operate within the streaming generation loop with zero additional cost. Predictability is another benefit; the same input consistently yields the same flags, and adding new patterns requires only a regex update. After three weeks of dogfooding, specific false-positive rates emerged. The em dash check fired on legitimate use about 15% of the time. The rule-of-three check, due to its tight parallel-structure heuristic, exhibited the lowest false-positive rate at less than 5%. The AI vocabulary check proved the most controversial, sometimes flagging genuinely appropriate word usage. Consequently, the founder made the AI vocabulary check advisory, highlighting issues without dropping the score below the revision threshold.

WHAT WE'D CHANGE

The deterministic approach, while fast and cost-effective, presents several areas for refinement to enhance its robustness and applicability. The source itself notes that static wordlists for

Pull quote: “”