Removing option labels from LLM classifier prompts increases accuracy by 15%
Tactic · Hugging Face · stat: +15.1% Researchers analyzing Jev-style typed decision models find that LLMs ignore written definitions when explicit option labels are present in prompts. Formatting…
Tactic · Hugging Face · stat: +15.1%
Researchers analyzing Jev-style typed decision models find that LLMs ignore written definitions when explicit option labels are present in prompts. Formatting options as generic letters or omitting labels entirely resolves this option-label bias, raising classification accuracy by up to 15.1% on Qwen2.5 backbones.
Prompt formatting, not model weights, causes LLM classification failures Founders building LLM routers should strip descriptive labels from prompts to prevent classifiers from ignoring complex routing rules.
Hugging Face paper 2610.02586
Based on laya-td model performance on PolicyBench from the Hugging Face paper.
Deleting definitions barely degrades performance, confirming that the model relies almost entirely on option labels rather than written instructions.
Every claim ties to a primary source. See our methodology.