The autonomy illusion: Why today’s AI cybersecurity tools are far from self-sufficient

A new study warns that a fundamental mischaracterization in cybersecurity technology may be endangering digital systems. While headlines trumpet the rise of “autonomous” AI-driven penetration testers, the reality is far less independent, and far more reliant on human oversight, than the term implies.

The study, titled "Cybersecurity AI: The Dangerous Gap Between Automation and Autonomy,"  and published on arXiv, challenges prevailing narratives surrounding AI in cybersecurity. It presents a comprehensive taxonomy of autonomy levels in AI-driven security tools and exposes the risks of mislabeling automated systems as fully autonomous.

Where does automation end and autonomy begin?

Automation, as described by the author, involves systems that execute preprogrammed routines, efficient but limited to predictable environments. Autonomy, by contrast, demands the ability to perceive, reason, and adapt to unstructured or unforeseen conditions.

This difference between automation and autonomy is not just semantic. In cybersecurity, labeling a tool “autonomous” implies it can operate independently, assess complex risks, and make decisions without human intervention. But most AI-based penetration testing tools in the market operate at Level 3 or Level 4 on a proposed six-tier scale of autonomy, far from the “Level 5” threshold of complete independence.

The study adapts the SAE J3016 levels of driving autonomy to cybersecurity contexts, introducing a novel scale from Level 0 (no tools) to Level 5 (full autonomy). At Level 2, tools like PentestGPT assist humans in planning but rely on them for execution. Level 3 tools such as AutoPT and Vulnbot can perform full attack sequences, but still need manual intervention for strategic decisions and validations. Level 4 systems, like the open-source CAI framework, attempt full-spectrum automation, planning, scanning, exploiting, and mitigating, but require a supervisory human to manage complex or novel situations.

Only at Level 5 would a system be considered truly autonomous, capable of operating entirely on its own across all conditions. But no such cybersecurity AI system exists today. Even XBOW, a penetration testing AI that made headlines for topping a HackerOne leaderboard, is shown to require significant human validation before publishing any findings.

Why human oversight remains essential

The myth of self-sufficient AI in cybersecurity crumbles when examining the role of the Human-In-The-Loop (HITL). Mayoral-Vilches argues that even the most advanced systems today rely heavily on human intervention. Whether through prompt engineering, validating findings, or strategic oversight, humans continue to play a vital role in the operation of these AI agents.

A notable example from the paper is XBOW, whose developers claim full autonomy. Yet before submitting over a thousand vulnerability findings, a human security team had to manually review each submission. These reviews are necessary to filter out false positives, long recognized as a challenge in cybersecurity automation, and ensure compliance with bug bounty rules. The study likens this to robotic perception errors, where sensors misread benign conditions as threats.

Automated validators, such as headless browsers that test XSS exploits or LLMs that double-check AI-generated results, offer improvements but fall short of replacing strategic reasoning. These are sophisticated scripts, not decision-makers.

This reality, the study notes, should temper the overhyped marketing language surrounding “autonomous hacking.” AI systems are powerful tools but lack the contextual understanding and ethical judgment of human operators. The ideal, then, is not human replacement but partnership: letting AI shoulder the repetitive or time-consuming tasks, while experts focus on oversight, ethical considerations, and high-level decision-making.

The risks of overstating AI capabilities

The dangers of exaggerating AI’s autonomy are both practical and ethical. When vendors sell systems as “fully autonomous,” buyers may reduce their scrutiny and remove necessary layers of human control. The study warns this could create new vulnerabilities, precisely the outcome cybersecurity tools are meant to prevent.

Investors are also fueling this narrative. XBOW recently secured $75 million in Series B funding; Horizon3.ai raised $100 million; Mindgard collected $8 million. Yet despite bold claims from company founders about surpassing human hackers, the technical reality is far more nuanced. In XBOW’s case, the AI depends on proprietary scaffolding, pre-existing security tools, and third-party LLMs, all orchestrated by human developers.

This disconnect between marketing and engineering mirrors past overpromises in the self-driving car industry. The study argues for a more responsible advancement of cybersecurity AI, emphasizing transparency and evidence-based evaluations. The open-source CAI project stands out in this regard, openly publishing benchmarks, limitations, and real-world results to set realistic expectations.

The study highlights that claims of Level 5 autonomy are often undermined by even a single human veto point. Strategic decisions, such as triaging the severity of a vulnerability or choosing a method of disclosure, still require human involvement. Additionally, cybersecurity operates in a constantly changing digital landscape, comparable to dynamic environments in robotics, where real autonomy is even more difficult to achieve.

Going ahead, the cybersecurity industry must adopt precise language and accurate classifications. Tools operating at Level 3 or 4 should not be marketed as “fully autonomous.” The goal should not be to eliminate humans from the loop, but to build systems that amplify human intelligence and ensure responsible, secure outcomes, the study recommends.