AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan
Summary
Gray Swan’s core bet is that enterprise AI security will become a standalone control layer, because models are untrusted components rather than conventional software. A few widely used models and agents, including Codex and Claude Code, create correlated failure risk across customers. The labs will keep investing internally, but Zico Kolter expects the familiar platform pattern: “a security system separate from it” becomes its own service category.
Automated red teaming may already be crossing the human-performance threshold for bounded evaluations. Gray Swan’s SHADE system found more model breaks than human red teamers within a fixed task set and time window, although Zico Kolter cautions, “I don’t think we’re quite at superhuman levels of red teaming yet.” The attack-data loop connects SHADE with an Arena community whose Discord has roughly 15,000 members.
Agentic tool use converts model weirdness into an enterprise loss surface: leaked credentials, unauthorized actions, and even erased production databases. The “lethal trifecta” combines untrusted external data, access to private information, and a channel for exfiltration; tool access can additionally enable harmful actions.
Raw model capability does not reliably buy adversarial robustness, leaving room for a specialized runtime-defense market. The IPI benchmark showed only a small, somewhat confounded correlation between GPQA Diamond capability and indirect-prompt-injection attack success; robustness improves through explicit training, not scale alone. Cygnal monitors untrusted inputs and proposed tool calls against configurable enterprise policies.
Computer use and OpenClaw crystallize the adoption paradox: the permissions that make agents valuable are the permissions that make them dangerous. Gray Swan threw SHADE at real OpenClaw usage trajectories and “found breaks for each and every one of them,” while conceding that protecting every possible tool remains future work. Cygnal therefore complements, rather than replaces, isolation, authentication, access controls, and narrow permissions.
A potentially powerful distribution loop connects automated risk assessment, runtime mitigation, and AI insurance. SHADE and private Arenas can measure deployment risk; Cygnal can reduce it; an insurer can make those controls a procurement or underwriting requirement. The framework is not yet an AI equivalent of SOC 2, but insurance and compliance pressure could accelerate enterprise spending.
The longer-duration upside is that agents may automate the labor-intensive sciences needed to secure agents themselves. Kolter has become more optimistic that coding agents can systematize mechanistic interpretability, while the discussion also points to agents making secure-code and formal-verification work more practical despite historically requiring “10 or 20 times as long” as ordinary coding. Their thesis is that security and interpretability could “explode, not because we’re going to get better at it, but because agents can do it for us now.”
Deep dive
Not yet available upstream; scheduled sync will retry.