The annual pentest has always bothered me
Every team I’ve worked on has treated penetration testing as a periodic event. Once or twice a year, you hire an external firm, wait a few weeks for the report, fix the critical findings, and move on. Meanwhile the codebase keeps changing daily. The security assessment you got in March is already stale by May. I’ve always found this rhythm uncomfortable - you know the gap is there, but the cost and lead time of a proper engagement made it hard to do anything about it.
That’s why two open-source tools that showed up recently caught my attention: Shannon and PentAGI. They’re both fully autonomous AI agents that run pentests without human hand-holding, and they’re both free to self-host. What interests me isn’t so much the tools themselves, but what they signal about where security testing is heading.
What actually changed
AI agents are getting good at the structured, pattern-matching part of pentesting: known vulnerability classes, common exploitation techniques, source review, browser actions, and report generation. I would still want a senior human tester for business logic, chained attacks, and the weird context around how a product is actually used.
Shannon comes at this from the web application side. It reads your source code to understand how the app works, then attacks it through a real browser. It’s forming hypotheses, executing exploits, and pivoting based on what it observes. On the XBOW benchmark it scored 96% against senior human pentesters’ 85%. A full run costs ~$50-100 in API credits and takes an hour-two depending on the project complexity. You can also use OAuth token with quotas included in your plan. I have used it with my personal Max plan and it used 80% of the session quota, which resets every few hours and no extra cost. That’s the part that made me stop and reconsider how I think about testing cadence.
PentAGI goes wider instead of deeper. It orchestrates 20+ existing tools like Nmap, Metasploit, sqlmap, through a multi-agent system that plans and executes across network, infrastructure, and application layers. It supports multiple LLMs including local models through Ollama, which matters if you can’t send target information to a third-party API. Everything runs in Docker containers so there’s no risk of the agent doing something unexpected to your host.
Shannon and PentAGI approach the work from different sides. Together they make security testing feel less like a calendar event and more like something you can run after a meaningful release.
How I’d set up security testing now
If I were building a security testing workflow from scratch today, it would look different from what I’ve set up in the past. I’d still want a human-led pentest at least once a year for the creative, context-heavy work, like business logic flaws, multi-step attack chains, and the kind of finding that requires understanding what the application is for. Between those engagements, I would add an AI agent to the release process. Shannon could run against the web app on major releases to catch OWASP Top 10 regressions, while PentAGI could periodically check the broader infrastructure for newly exposed surfaces.
The cost changes the cadence. A five-figure engagement still belongs on the calendar, and a run that costs roughly the same as a few hours of engineering time can happen around releases, migrations, and bigger dependency changes. I would treat the output as another security signal, not final truth. I would still rather have that extra pass than wait six months for the next report.
They’re not there yet, and that’s fine
Both tools are early. Shannon only covers web apps and sometimes it runs in circles. PentAGI’s breadth means it won’t go as deep on any single target as a specialist would. Neither will find the subtle business logic flaw that requires understanding your domain. And I haven’t stress-tested either on a production codebase yet. Benchmark scores are promising, but real-world results always have surprises.
A year ago, “autonomous pentesting” mostly sounded like a vulnerability scanner with better marketing. These tools are more interesting because they read source code, use a browser, execute real exploits, and produce reports with reproducible proof-of-concepts. A serious assessment still has a place. The part I would change is what runs in the months between them.