AgentGuard benchmarks 100% detection on AI agent attacks that Semgrep and CodeQL miss

A new static analysis tool, AgentGuard, claims perfect detection on a reproducible benchmark of 39 AI agent vulnerabilities, highlighting a coverage gap in established scanners for this emerging attack surface.

The Answer Up Front

This tool is for development teams building with agentic AI frameworks like Microsoft AutoGen or LlamaIndex. If you are shipping products that use chained AI actions, you should evaluate AgentGuard immediately. Teams working on traditional applications without an AI agent component can skip it; existing SAST tools remain sufficient for non-AI codebases. The bottom line is that AgentGuard is a highly specialized, early-stage tool that addresses a real and emerging vulnerability class with verifiable, reproducible results, making it a compelling addition for any team working on the agentic frontier.

Methodology

This v0 review covers AgentGuard v0.6.4, as observed on July 5, 2026. The analysis is based entirely on the vendor's launch announcement and accompanying public benchmark repository. The source signal is a blog post titled "AgentGuard vs Semgrep vs CodeQL: 100 Percent vs 0 Percent on AI Agent Security" published on dev.to by the tool's creator, Dockfix Labs, available at https://dev.to/dockfixlabs/agentguard-vs-semgrep-vs-codeql-100-percent-vs-0-percent-on-ai-agent-security-4iil.

This review covers the vendor's claims, the structure of the provided benchmark repository (https://github.com/dockfixlabs/agentguard-benchmark), and the cited vulnerability reports filed against open-source projects. What is not covered is any independent performance testing on a wider corpus of code, the tool's false positive rate on large, real-world codebases, or its long-term integration into a production CI/CD pipeline. This review is based on the vendor's published claims; independent benchmarks are pending.

What It Does

A specialized static analysis scanner

AgentGuard is a command-line static analysis security testing (SAST) tool distributed as a Python package. It is designed specifically to find security vulnerabilities in applications that use AI agents. Unlike general-purpose scanners, its ruleset is tailored to the unique ways that chained Large Language Model (LLM) calls, tool usage, and multi-agent interactions can be exploited.

Covers OWASP ASI and novel attacks

The tool's creator claims its 17 detection rules cover all 10 categories from the Open Web Application Security Project's (OWASP) AI Security Initiative (ASI). It also introduces detection for four novel attack vectors specific to agents: Memory Poisoning, Tool Output Trust, Action Chain Amplification, and Multi-Agent Collusion. These represent a new class of security risks not typically found in traditional web or mobile applications.

Reproducible benchmark against incumbents

The core of AgentGuard's launch is a public benchmark containing 39 code samples, each with a specific AI agent vulnerability. According to the vendor's tests, AgentGuard detects 100% (39/39) of these issues with zero false positives. The same benchmark run against popular SAST tools Semgrep and CodeQL resulted in 0% detection. The entire benchmark is available on GitHub and can be run with a simple python benchmark.py command.

Real-world findings

To demonstrate that these are not just theoretical vulnerabilities, the vendor reports finding 332 critical issues across two major open-source agentic frameworks, Microsoft AutoGen and LlamaIndex. The post includes links to specific issues filed on GitHub (autogen#7917, autogen#7918, llama_index#22245), lending significant credibility to the tool's practical utility.

What's Interesting / What's Not

The most compelling aspect of AgentGuard's launch is its verifiability. By shipping a simple, runnable benchmark, Dockfix Labs has bypassed the typical marketing claims and provided a direct, reproducible test case. This is the correct way to build trust for a new security tool. It allows any potential user to verify the core claim (100% vs 0% on this specific test set) in minutes.

Also notable is the tool's sharp focus. Instead of trying to be another general-purpose SAST tool, AgentGuard is hyper-focused on a new, well-defined problem space. This is a classic and effective go-to-market strategy for a new entrant. The cited vulnerabilities in major projects like AutoGen prove the problem is real and that the incumbent tools are, for now, blind to it.

The primary caveat is that the benchmark is, by definition, authored by the vendor to showcase their tool's strengths. It is designed to test for vulnerabilities AgentGuard can find. This is not deceptive, but it means the 100% detection rate and 0% false positive rate apply only to this curated set of examples. The tool's true performance and false positive rate on large, messy, production codebases are unknown.

Pricing

As of July 2026, AgentGuard is an open-source tool available on GitHub and installable via PyPI (pip install dfx-agentguard). The source materials do not mention any paid tiers, enterprise plans, or commercial offerings.

Verdict

For teams building applications with agentic AI frameworks, AgentGuard appears to be a necessary, specialized security scanner. Its vendor-authored benchmark is reproducible and demonstrates a clear, and currently wide, capability gap in established tools like Semgrep and CodeQL for this new attack surface. While its real-world false positive rate on diverse codebases remains an open question, its documented success in finding vulnerabilities in major open-source projects makes it a credible and valuable addition to a modern security toolchain. Teams not using agentic architectures have no need for it today.

What We'd Test Next

A v2 review would require independent testing. First, we would run AgentGuard against a large, private corpus of production agentic code to assess its real-world false positive rate and performance at scale. Second, we would write our own novel AI agent exploits, not included in the vendor's benchmark, to determine if AgentGuard's rule-based system can detect them or if it is overfit to its initial test suite. Finally, we would evaluate its CI/CD integration story, including output formats, exit codes, and configuration options for a production environment.

The investor read

AgentGuard signals the emergence of a new, verticalized SAST category: AI Agent Security. This follows the classic pattern of specialized security domains (e.g., container security, API security) spinning out of general Application Security as new technology stacks create novel attack surfaces. The playbook is familiar: identify a high-stakes vulnerability class that incumbents are blind to (the 0% detection from Semgrep/CodeQL is the hook) and provide a targeted solution. To become investable, Dockfix Labs must demonstrate this is a durable category, not just a feature Semgrep will add next quarter. A moat could be built through proprietary research on novel agent attacks, a commercial offering with advanced rules and management features, and evidence of early enterprise adoption. The open-source tool is the lead generation engine. The key risk is platform risk: will major agentic frameworks like AutoGen build these security protections in natively, obviating the need for a third-party tool?

Pull quote: “The most compelling aspect of AgentGuard's launch is its verifiability.”