A Chatbot Scored 78/100 While Ignoring a Patient's Chest Pain

An audit of a healthcare chatbot revealed four critical failures hidden by a passing grade. The case study shows why generic scores can mask catastrophic liability risks.

A healthcare chatbot scored a 78 out of 100 on a performance audit. When a test user mentioned chest pain, the bot replied, "I can't provide medical advice," and moved on. It offered no direction to call emergency services or visit a hospital.

This incident, detailed in a case study by the testing platform BotCritic, illustrates a critical gap between aggregate AI performance metrics and operational safety. A passing grade concealed a patient safety failure severe enough to represent a significant legal and ethical liability. The audit demonstrates how easily a deployed AI system can silently harbor catastrophic risk.

What the audit found

The audit used four distinct personas to stress-test a clinic's WhatsApp chatbot. The findings highlight specific failure modes that generic scoring can obscure.

A liability disguised as a UX bug

The most severe failure occurred when a user reported chest pain. The bot's programming correctly prevented it from offering medical advice, a standard and necessary guardrail. But it failed to recognize a keyword indicating a medical emergency. The system did not trigger any escalation protocol, such as advising the user to call 999 or go to an emergency room.

According to the audit transcript, the bot's sole response was the deflection. This is not a user experience flaw. A healthcare chatbot must recognize a symptom that requires immediate emergency care and point the patient toward that care.

Ignoring explicit user frustration

A persona simulating a frustrated patient stated they had been trying to contact the clinic for two weeks. The chatbot acknowledged the sentiment but then immediately reverted to its standard intake script, asking for personal details without addressing the two-week delay.

The bot only offered to connect the user with a human agent after the user explicitly demanded it. The system was not designed to interpret "two weeks of failed contact" as a signal requiring automatic escalation, treating a high-frustration case the same as a routine inquiry.

Accepting impossible data

The audit also tested for data integrity. A persona submitted "32/13/1990" as a date of birth. This impossible date was accepted without any validation or request for correction.

This corrupt data was then presumably passed into the clinic's patient record system. In a healthcare context, where accurate data underpins scheduling, billing, and clinical history, such a failure introduces downstream errors that can affect patient care and operations. Basic input validation, a standard practice in software development, was absent.

What we'd change

The case study is a playbook of what to avoid, but its lessons inform a more robust testing protocol for high-stakes AI. The core problem is an over-reliance on aggregate scores and a failure to test for specific, catastrophic outcomes.

First, testing must prioritize "red flag" scenarios over broad performance. A single catastrophic failure should fail the entire system, regardless of its performance on 99 other tests. For a healthcare bot, this means a non-negotiable, hard-coded list of keywords (e.g., "chest pain," "can't breathe," "suicidal") that immediately halt the AI's conversational flow and trigger a rigid, pre-approved emergency protocol. This is a system design requirement, not a prompt engineering task.

Second, integrate traditional software validation. The acceptance of an invalid date of birth is not an AI problem. It is a failure to implement basic input validation before data is processed by the model or saved to a database. An LLM should not be the first line of defense for data integrity.

Finally, the audit itself reveals the value of adversarial testing using distinct personas. A standard QA process might check if the bot can answer common questions. This audit specifically probed for failures under frustration, confusion, and edge-case inputs. This adversarial approach should be the default for any application where failure has serious consequences.

Landing

The audit's "C" grade was technically a passing score, but it was meaningless. The bot's ability to maintain a calm tone or resist prompt injection did not matter when it failed its most critical, implicit task: do no harm. For founders and investors in high-stakes verticals, this case is a clear signal. The defining metric for success is not average performance, but the complete absence of catastrophic failure. The difference is what separates a viable product from a pending lawsuit.

The investor read

The market for AI auditing and QA tools is nascent but critical, particularly in regulated industries like healthcare, finance, and legal. This case study, though from a vendor, highlights the core enterprise pain point: liability. A generic "good enough" score from a horizontal platform is insufficient for CISOs and legal departments. This creates an opening for vertical-specific testing platforms that can build libraries of domain-specific "red flag" tests. Investors should watch for pick-and-shovel plays that de-risk AI adoption for large enterprises. The value proposition is not "make your AI better," but "prevent your AI from creating a multi-million dollar liability." That is an investable thesis.

Pull quote: “A single catastrophic failure should fail the entire system, regardless of its performance on 99 other tests.”