Continuous Runtime Monitoring for AI Agents: Edison Flores' L3 Pipeline

Edison Flores built L3, a continuous runtime monitoring layer, to address the vulnerability of AI agent skills. This system re-runs skills in a sandbox, comparing behavior against a baseline to detect critical drifts.

Edison Flores, building the Sentinel pipeline for AI agent skill security, identified a core vulnerability: "Certification is point-in-time. Attacks are runtime." This observation, which Flores reports was echoed by four independent reviewers, frames the challenge of securing AI agents whose behaviors can change post-deployment.

His existing 8-layer Sentinel pipeline included static analysis (L1.5-L1.8) and a gVisor sandbox (L2) to capture an initial behavioral baseline. However, these methods could not detect changes occurring after certification. Such changes include configuration drifts expanding allowed_paths, supply chain updates injecting new payloads, compromised credentials leading to data exfiltration, or new tools appearing in a skill's catalog. Static analysis is blind to these runtime changes, and the L2 baseline is merely a snapshot.

L3: Continuous Runtime Monitoring

Flores developed L3 to address this gap. L3 continuously re-runs skills within the sandbox environment on a scheduled basis, specifically weekly via GitHub Actions. It then compares the observed runtime behavior against the established L2 baseline. Any deviation from this baseline triggers a flag, indicating a potential security drift.

Eight Drift Detection Types

L3 incorporates eight distinct drift detection types, each targeting specific vulnerabilities. These include TOOL_CATALOG_NEW_TOOLS and TOOL_CATALOG_CHANGED_SCHEMA, both critical for identifying unauthorized modifications to an agent's capabilities. Supply chain integrity is monitored through SUPPLY_CHAIN_GIT_SHA_CHANGED (critical) and SUPPLY_CHAIN_NPM_VERSION_CHANGED (high), flagging any updates to the underlying code or packages.

Network-level anomalies are caught by NETWORK_NEW_DOMAINS (high), which detects attempts to contact domains not present in the baseline. CONFIG_PERMISSIONS_EXPANDED (critical) flags unauthorized increases in allowed_paths or scopes. Credential security is addressed by CREDENTIAL_NEW_ENV_ACCESS (high), identifying access to environment variables not in the baseline. Finally, PROCESS_NEW_SPAWNS (high) detects new processes initiated by the skill that were not part of its original behavior.

Addressing Attack Vectors

L3 directly counters several attack vectors. For instance, a configuration drift changing allowed_paths from /data to / is detected by comparing current permissions against the baseline, triggering a critical alert and re-quarantining the skill. Supply chain updates, such as injected payloads, are caught by checking Git commit SHAs and npm versions; any change since certification results in a critical alert and requires re-auditing. The system also mitigates compromised credential exfiltration attempts by re-running the sandbox with network disabled, flagging any unexpected credential accesses.

Public Auditability and Status

Results from L3 are stored as publicly auditable JSON files in _data/l3_results/{skill_id}.json. If a skill exhibits drift, it is flagged at the /api/security?view=quarantine endpoint with reason: 'l3_drift_detected'. A live security overview, now showing all nine layers of the pipeline, is available at https://marketnow.site/api/security.

What We'd Change

While L3 represents a concrete step in AI agent security, several areas warrant further consideration. Edison Flores acknowledges that L3 does not fully solve client-specific behavior, where a skill might present clean descriptions to a scanner but malicious ones to a real agent. His mitigation of running with multiple User-Agent strings is a partial solution, but a more robust adversarial testing framework would be necessary to truly address this. Similarly, L3 focuses on behavioral drift, not zero-day protocol exploits, which remain a persistent threat.

The reliance on weekly re-runs via GitHub Actions, while functional, introduces potential scalability and cost considerations for a large number of skills or for scenarios requiring higher-frequency monitoring. Automating the re-auditing or re-quarantining process beyond simple flagging would reduce operational overhead. Furthermore, the system's efficacy depends on the accuracy of the L2 baseline; mechanisms for baseline refinement or adaptive learning could enhance its robustness against legitimate, evolving skill behaviors versus malicious drifts.

The investor read

The development of L3 by Edison Flores highlights a critical and emerging need in the AI agent market: continuous runtime security. As AI agents become more autonomous and integrated, their attack surface expands beyond initial static checks. Solutions like L3, which provide behavioral baselining and drift detection, signal a maturing security landscape for AI-native applications. Investors should note the increasing demand for runtime verification tools, particularly those that can integrate into CI/CD pipelines and offer auditable logs. While L3 is presented as a bootstrapped initiative, its underlying problem space—securing dynamic AI systems—is a significant area for venture capital, potentially attracting investment for more comprehensive platforms that offer enterprise-grade scalability, advanced threat intelligence, and automated remediation for AI agent security.