Home›Read›Tools desk›FINAL-Bench tests 20 LLMs on detecting obfuscated shell execution in AI agents
Tools·Nov 19, 2026

FINAL-Bench tests 20 LLMs on detecting obfuscated shell execution in AI agents

Vidraft benchmarked 20 language models against 221 obfuscated shell commands to evaluate AI agent security gates, finding high risk detection but severe false-positive variance on benign commands.…

Vidraft benchmarked 20 language models against 221 obfuscated shell commands to evaluate AI agent security gates, finding high risk detection but severe false-positive variance on benign commands.

For teams building production AI coding agents that execute shell commands, model choice for safety inspection cannot rely solely on standard safety alignment scores. OpenAI GPT-6.1 Terra leads overall balanced accuracy at 99.5%, while Kimi K3.1 Pro is the top open-weights option at 97.9% balanced accuracy and 99.3% adversarial robustness. If you need low-latency self-hosted safety gate filtering, DeepSeek V4.1 Flash hits 95.2%. Teams should skip overly aggressive safety models that block benign encoded scripts; while almost all 20 tested models block dangerous shell calls at over 90% recall, benign pass rates collapse down to 13.6% on overly paranoid models.

Methodology

This v0 review draws on the founder's published claims at https://dev.to/vidraft_lab/base64ro-gamcun-wiheom-myeongryeong-ai-anjeon-gwanmuneun-alabolgga-nandoghwa-syel-anjeondo-rideobodeu-20gae-model-silceug-3jgk; independent benchmarks pending. Update cadence: re-tested when claims diverge from observed behavior.

The evaluated artifact is the Obfuscated Shell Safety Leaderboard maintained by Vidraft (FINAL-Bench), observed in November 2026. The evaluation protocol uses a private benchmark dataset of 221 shell commands made up of malicious and benign execution attempts. The public dataset details are hosted on Hugging Face at FINAL-Bench/Obfuscated-Shell-Gate.

Scoring relies on a 3-run majority voting procedure per test case without using an LLM-as-a-judge pattern, relying on deterministic output parsing for ALLOW versus DENY actions. The benchmark measures balanced accuracy, risky recall, benign pass rate, and adversarial robustness when wrapped in persuasion or execution wrappers.

This review covers Vidraft's published methodology, reported model accuracy scores across 20 evaluated models, and published dataset structure. It does not cover long-term agent execution latency, custom runtime sandbox escape prevention, or independent reproduction of the 221 private test items.

Obfuscation techniques and threat vectors

The benchmark evaluates how model-based guardrails parse shell commands disguised using structural and encoding tricks. Obfuscation vectors tested in the dataset include base64, hex, octal, rot13, string reversal, variable splitting with custom shell variables, inline quotes or backslashes, process substitution, interpreter decoders, and multi-layered combinations.

Vidraft frames this benchmark around real-world agent compromise patterns, citing a Microsoft threat report from November 10, 2026, where attackers used curl | sh execution chains inside agent workflows. The goal of an execution gate is to decode payload intent before deciding whether to permit execution inside the agent sandbox.

Measuring risk detection against benign pass rates

The benchmark separates model evaluation into two primary metrics: risky recall (the percentage of malicious shell calls blocked) and benign pass rate (the percentage of harmless shell calls allowed).

Vidraft's findings show that risky recall is solved across almost all 20 tested LLMs, with nearly every model blocking over 90% of obfuscated malicious commands. However, benign pass rates diverge dramatically across the field, spanning from 100% down to 13.6%. Models with poor benign pass rates reject routine administrative scripts that encode configuration payloads in base64, rendering coding agents unresponsive.

What's interesting and what's missing

The most useful contribution of Vidraft's benchmark is isolating false-positive friction in agent safety filters. Most AI safety benchmarks treat security as a binary classification of detecting bad inputs. By tracking benign pass rates alongside risky recall, FINAL-Bench highlights the actual engineering constraint: guardrails that flag valid base64 strings break developer experience.

The top-ranking closed model, GPT-6.1 Terra (99.5% balanced accuracy), and top open model, Kimi K3.1 Pro (97.9% balanced accuracy), demonstrate that advanced models can unwrap layered obfuscation without hallucinating threats into normal commands. DeepSeek V4.1 Flash placing third at 95.2% shows that small fast models can perform guardrail duty effectively.

What is missing from the current implementation is runtime context awareness. The benchmark evaluates single-line command strings in isolation. In real agent architectures, shell execution safety depends heavily on workspace state, environment variables, directory permissions, and past conversation history. A cleanup command in a temporary scratch directory is standard operation, whereas the same command directed at system root is destructive. Static payload classification without session state remains an incomplete test rig.

Pricing

The Obfuscated Shell Safety Leaderboard and public evaluation dataset are available at no charge on Hugging Face (FINAL-Bench/Obfuscated-Shell-Gate) as of November 2026. Running evaluations requires API access or local inference compute for the 20 tested LLMs.

Verdict

Vidraft's FINAL-Bench provides a pragmatic evaluation framework for safety gates in AI coding agents. For production systems requiring cloud API performance, GPT-6.1 Terra is the top recommendation based on its 99.5% balanced accuracy score. For self-hosted or open-weights security pipelines, Kimi K3.1 Pro (97.9% balanced accuracy, 99.3% robustness) and DeepSeek V4.1 Flash (95.2%) are the strongest choices.

Avoid choosing models based purely on threat block rates. If a security model achieves 98% risky recall but drops benign pass rates to 20%, it will break user workflows. Select models that maintain at least 95% benign pass rate to avoid blocking legitimate administrative shell operations.

What we'd test next

In a v2 benchmark suite, we would test multi-turn agent context, evaluating whether models alter their DENY/ALLOW decisions when presented with preceding shell outputs and directory manifests. We would also benchmark inference latency and cost overhead of running dedicated guardrail checks like DeepSeek V4.1 Flash on every tool call versus inline system prompt guardrails.

The investor read

Vidraft's benchmark highlights a structural market shift in AI agent infrastructure: security is moving from generic system-prompt guardrails to dedicated deterministic-plus-LLM execution gates. As autonomous agents take destructive shell actions, enterprise buyers will mandate auditable gatekeeping middleware. Top open-weights performance from Kimi K3.1 Pro (97.9%) and DeepSeek V4.1 Flash (95.2%) indicates that lightweight specialized guardrail models will compete directly with expensive frontier API calls for tool-use filtering. The benchmark project itself is an open dataset contribution; investable commercial opportunities lie in agent runtime security proxy platforms (such as e2b, Modal, or specialized security gateways) that can package these high-pass, high-recall evaluation pipelines into zero-latency execution sidecars.

Pull quote: “while almost all 20 tested models block dangerous shell calls at over 90% recall, benign pass rates collapse down to 13.6% on overly paranoid models.”

Sources · how we verified
  1. base64로 감춘 위험 명령, AI 안전 관문은 알아볼까: 난독화 셸 안전도 리더보드 20개 모델 실측 ↗

Every claim ties to a primary source. See our methodology.

Reported by the Riley desk on Founderr Pulse’s Tools beat. Every factual claim is tied to a primary source and linked; anything that can’t be stood up doesn’t run. Founderr (RIKHATH LLC) is the accountable publisher and corrects in place. How we work · About · File a correction.
R
Riley

The Riley desk covers tools — what founders are building with, switching to, and abandoning. Every claim is sourced and linked. Operated by Founderr (RIKHATH LLC) See the desk →

Founderr Pulse — free & independent. The desk for people who build & back.