Agentic security SoK: tools, failure modes and design laws for LLM-driven pentesting

What happened on this date

At 07:18 UTC on 2026-08-16, arXiv received v1 of "Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing" (primary category cs.CL). The authors study systems in which LLM agents plan, dispatch and interpret security tools, and systematize recurring operational failures through a hands-on evaluation of ten widely used static, dynamic, cloud, orchestration and AI red-teaming tools in unattended pipelines. They introduce a four-dimensional Integration Friction Index separating one-time engineering cost from recurring organizational, legal and maintenance cost, and derive quantitative regularities: long-lived sessions lose resident evidence as phases accumulate, while short-lived sub-agents extend the usable horizon; a two-stage verdict cascade adds little when scorer errors are correlated; and counting unevaluable outcomes as attack failures biases downstream measurements. Finally, they argue that scope and budget enforcement cannot be delegated to system prompts, because prompts do not constrain what actually executes.

Why it matters for people protecting app logic

More developers now use AI agents to security-test their own apps and backends, and this paper is essentially a list of where such agents are unreliable, which matters for how much you trust the results. If long sessions lose evidence, an extended automated test that reports nothing is not proof that nothing is there; and treating "could not evaluate" as "defended" will make you overestimate your protection. When thinking about attackers, the same failure modes exist in automated analysis pipelines, but they are not a reason to underestimate an adversary: the paper itself shows that engineering design can partly mitigate them. The work is built around the authors' own platform, Inspectra, with some mechanisms labeled planned or partial, so how far the conclusions generalize still needs more evidence.

Concrete checks

First, when you use agents to test your own app, constrain scope with deterministic controls outside the prompt, such as network allowlists, dedicated test accounts and a separate test environment. Second, report "unevaluable" outcomes as their own category in results instead of folding them into "passed". Third, split long tasks into several short ones and keep the raw evidence from each step so a human can review it. Fourth, have a person review both the agent's high-risk findings and its "nothing found" conclusions before they go into a protection assessment report.

Your checklist

• Bound agent testing with network allowlists and dedicated test accounts
• Report unevaluable outcomes separately, never as passed
• Split long tasks and keep raw evidence for each step
• Have humans review both findings and nothing-found conclusions