A WordPress.org rejection reveals a new kind of AI technical debt

A founder’s plugin was rejected for violating a rule their AI agent was explicitly told to follow. Their post-mortem details a playbook for managing this emerging 'AI config drift.'

A WordPress.org plugin rejection was the first sign that a founder's AI development process was broken. The AI agent had ignored a specific rule in its own configuration file, shipping code explicitly forbidden by its human-written instructions. The founder reports the rejection forced a re-evaluation of a core assumption: that a written rule in an AI's config file is an effective rule.

This incident highlights a new form of technical debt specific to AI-assisted development. As instruction files like a CLAUDE.md grow, their effectiveness can decay silently. The founder’s experience provides a playbook for identifying and managing this risk.

Written is not the same as in effect

The founder’s initial process relied on a long CLAUDE.md file containing style guides, translation rules, and personal design conventions. One convention explicitly forbade creating “trialware-shaped code” to gate paid features, a practice that often draws scrutiny from the WordPress.org review team. The AI agent generated exactly that code, leading to the rejection.

The core failure was assuming a config file certifies its own effect. The founder identifies two silent failure modes. First, as the file's length increases, critical rules get diluted and lose their influence on the model’s output. Second, as a developer's own conventions evolve, the static config file enforces outdated practices. In both cases, the rule is present in the text but absent from the outcome.

A three-part manual audit

In response, the founder stopped trusting the config file and started measuring its effect. The new process has three components.

First, they treat adding any line to the config as a cost, not a benefit. Each new instruction dilutes the attention paid to existing ones, so the file is kept shorter than feels comfortable. This forces a focus on only the most critical rules.

Second, the file now includes metadata at the top: an owner and a last-reviewed date. This is a signal for human developers, not the AI. It serves as a check against staleness, prompting review and removal of obsolete instructions.

Third, they now actively verify the AI's understanding before critical tasks. By opening a fresh session and asking the agent to list the conventions it believes are in effect, the developer can spot which rules have been silently dropped or ignored. This manual check precedes any work that depends on guideline compliance.

What we'd change

The founder’s solution is a manual, attention-based audit. It’s a necessary first step but does not scale and introduces a new failure point: a developer forgetting to perform the check. A more robust solution would treat the AI's configuration file like application code, subjecting it to automated testing.

Instead of asking the agent to list its rules, a developer could build a small suite of “config unit tests.” For a critical rule like avoiding trialware hooks, a test could prompt the agent to generate code for a premium feature. A simple linter or static analysis script could then scan the output for the forbidden code patterns. If the pattern is found, the test fails, signaling that the config rule is no longer effective.

This approach shifts the burden from human memory to an automated CI pipeline. It provides durable, machine-verifiable proof that the guardrails are working. The founder’s manual spot-check is a v0; config unit tests are the v1.

Landing

The founder’s manual checks are a stopgap. The durable solution will treat AI configuration not as a static document to be trusted, but as dynamic code to be tested. As development teams build more complex systems with AI agents, the practice of managing prompts and configs will require the same rigor as application code. That includes versioning, ownership, and automated testing to ensure the rules you write are the rules being followed.

The investor read

This founder's problem signals an emerging, and currently unserved, market for AI development tooling. 'AI config drift' is a new category of technical risk that existing CI/CD and code quality tools do not address. The manual verification playbook described is a clear market gap for a product that automates this process. The opportunity lies in building the 'Datadog for LLM prompts'—a monitoring and testing suite that ensures AI agent behavior remains aligned with developer intent over time. This is an infrastructure and AIOps play. Any company building significant internal tooling around LLMs is either facing this problem or about to. A scalable, automated solution would be highly valuable.

Pull quote: “The core failure was assuming a config file certifies its own effect.”