Table of contents
19
By Salman Khan, OSCP+, Founder of XHack, SRT (Synack Red Team member)
Read this in 30 seconds: AI vs human penetration testing sounds like a contest with a winner. The evidence points somewhere gentler and more useful, which is that each one covers the other’s blind spot.
- Give an AI agent a human plan and its success rate roughly triples. In the AutoPenBench study it solved 21% of tasks alone and 64% when a person broke the goal into steps first. On real CVEs the gap was wider still, 9% against 73%.
- On a live 8,000-host network, the best AI framework came second. It beat nine of the ten human professionals it was tested against, and the tenth still beat it.
- Practitioners have quietly changed their minds. Cobalt’s 2026 survey found trust in fully automated testing fell from 29% to 9% in a year, while preference for a human-and-AI blend rose 22 points to 47%.
- Some rules simply ask for a person. FedRAMP states that automated scanning alone does not satisfy the requirement, and DORA asks for proven human red-team expertise by name.
- The efficiency gap is larger than most people realise. Our own agent found a live OAuth client secret on a public bug bounty programme in roughly 25 minutes, on one line of human input, for about $5 of compute. A human engagement covering the same ground starts at $2,500.
I should say upfront that I run XHack, and XHack sells both an AI agent and human-led testing. So I have a foot in each camp, which is either a conflict of interest or a decent vantage point depending on how generous you are feeling. Judge the numbers rather than me.
Here is what makes the question hard. Most of what you read compares the two at their best moments.
The AI looks unbeatable when you watch it solve a capture-the-flag challenge in ninety seconds. The human looks irreplaceable when you hear about the tester who noticed something odd in a checkout flow and turned it into a six-figure finding.
Both stories are true. Neither tells you what to buy.
What follows is the evidence, laid out as plainly as I can manage, including the parts that are inconvenient for a company like mine. By the end you should know which kind of testing your situation actually calls for, and roughly what it will cost you.
When someone asks whether AI can replace a human pentester, they are usually asking one of three quite different things.
Sometimes they mean, can I stop paying for pentests. Sometimes they mean, will my job still exist in five years. And sometimes, most often in my experience, they mean, how do I get more testing than my budget currently allows.
Those deserve different answers, and in the AI vs human penetration testing debate only the third one has a genuinely happy ending right now.
It helps to be clear about what each side actually is.
An AI pentesting agent is software that plans and carries out an attack on a target you point it at, chaining steps together the way an intruder would rather than working down a list. The broader category is covered in the complete guide to AI penetration testing, and what AI pentesting actually is covers the mechanics without the marketing.
A human pentest is a person, usually certified, spending days inside your systems with a scope agreed in advance and a report at the end. The full process from scoping to reporting has not changed much in twenty years, and that stability is part of why regulators like it.
They are not the same product, and comparing them as though they were is how people end up disappointed.
Let me start with the side I am least likely to be accused of underselling, and be honest about how good it has become.
It is fast in a way that is difficult to overstate. Anthropic reported that its Mythos Preview model wrote working exploits in hours that experienced testers had estimated would take them weeks. On one Firefox exploitation task, the previous generation of the model succeeded twice in several hundred attempts. The newer one succeeded 181 times. That is not an incremental improvement.
It does not get bored. A human tester works through a target with finite attention and a deadline. An agent will re-run the same checks after every deployment, at three in the morning, for the four-hundredth time, with exactly the same care as the first. For catching the regression somebody reintroduced last Tuesday, this matters enormously, and it is the clearest practical argument for agentic pentesting.
It finds real things that careful tooling missed. Google’s Big Sleep agent found a buffer underflow in SQLite that both OSS-Fuzz and SQLite’s own test suite had walked past. It later caught a SQLite zero-day, CVE-2025-6965, in time for Google to patch ahead of exploitation. These are not toy results.
It covers ground cheaply. The ARTEMIS study measured its agent running at roughly $18 an hour of compute against roughly $60 an hour for a professional human tester. Different work, but the direction is clear, and what AI penetration testing costs breaks the pricing down properly.
And it scales past what any team could staff. One vendor reports over 310,000 production-safe autonomous tests across its customer base. Whatever you make of a single company’s numbers, no human organisation is running three hundred thousand engagements.
And I can point at our own results rather than only other people’s studies. Four of the engagements we publish were found by the agent working on its own.
On a YesWeHack programme it picked its own target, found a live OAuth client secret hardcoded in a public Flutter web bundle, confirmed the secret actually worked against the token endpoint, and wrote a Python proof of concept. That took about 25 minutes, one line of human input, and roughly $5 of compute. The programme accepted it and paid.
It found a live email-provider key baked into a survey tool’s JavaScript in under 24 hours, confirmed with a single read-only call, rotated the same day. It escalated a standard member to workspace owner through a mass-assignment flaw in under 48 hours. And it uncovered a critical privilege escalation for an automotive client.
Set the $5 and 25 minutes against what the same finding costs through a scoped human engagement, which starts at $2,500 and takes four days. That is not a marginal efficiency gain.
If your problem is that large parts of your estate are never tested at all, an agent solves a real problem, and autonomous penetration testing explains how continuous coverage tends to work in practice.
Now the other half of AI vs human penetration testing, and it holds up better than the hype suggests.
Business logic is the stubborn one. Michiel Prins, who co-founded HackerOne, put it better than I could. A logic flaw, he said, is “a loophole in the workflow so that an adversary can accomplish something that is unintended,” and that is very hard for an AI to find “because the AI needs to really understand the intent of the application, the business context it operates in, and the whole environment around it.”
Think about what that means concretely. An agent can tell you a discount code endpoint accepts unexpected input. The harder thing is spotting that stacking three legitimate coupons in a particular order lets someone buy a laptop for four pounds, because nothing in that sequence is malformed. Every request is valid. The business rule is what breaks.
I should complicate that immediately, because our own results do. One of the flaws our agent surfaced on its own was a mass-assignment bug that turned a standard member into a workspace owner by submitting a field that was not on the form. That is a workflow loophole, not a malformed request, and it is exactly the category this section is supposed to belong to humans.
So the boundary is moving, and anyone telling you business logic is permanently safe from automation is guessing. What is still true today is that a human is more reliable at deciding whether a logic flaw matters, which is a narrower claim than the one usually made.
Judgment about what a finding is worth. Amélie Koran, reviewing one prominent AI system’s public record, noted its badges were “some of the more basic things you can find with automation: data leaks, XML exposure, cross-site scripting, command injection and access control.” All genuine issues. None of them tells you which one would actually hurt your company on a Monday morning.
Knowing when to stop, and when to keep going. A researcher looking at a partial result decides whether it is a dead end or the first half of something serious. Víctor Mayoral-Vilches put the technical version of this neatly in a 2025 paper: validators are “scripted unit tests, not strategic reasoning. They reduce false positives but cannot arbitrate novel edge cases.”
Everything that is not a network. Social engineering, a conversation with a receptionist, a badge held open, a phone call to a help desk. These stay firmly human, and they remain how a great many real breaches begin. If you are weighing testing models more broadly, bug bounty against penetration testing is a related decision worth understanding.
And someone has to answer for the report. Which turns out to matter more than any of the above once a regulator is involved.
This is where most of the confusion in AI vs human penetration testing comes from, and it is worth slowing down for.
Most impressive AI security numbers come from capture-the-flag challenges. CTFs are puzzles built to be solved. There is a flag, it is definitely findable, and the environment was designed so that a clever solution exists. Real systems offer none of those guarantees.
Watch what happens to the same category of agent as the tests get more realistic.
| Benchmark | What it tests | Best result |
|---|---|---|
| InterCode-CTF | High-school level CTF puzzles | around 95% |
| Cybench | Harder CTF tasks | 17.5% unguided, 51.1% guided |
| AutoPenBench | Mixed, including 11 real CVEs | 21% fully autonomous |
| CVE-Bench | Real 2024 CVEs in real applications | roughly 2.5% to 25% |
That is the same technology falling from ninety-five percent to single digits, purely because the target stopped being a puzzle and started being software.
One detail from CVE-Bench deserves its own sentence. An agent that had been tuned for CTF performance dropped to 2.5% on real one-day vulnerabilities, because the things that made it good at puzzles did not transfer at all.
Anthropic, to its credit, flagged the same problem about its own evaluations, noting that benchmarks built on previously known vulnerabilities “can make it difficult to distinguish novel capabilities from cases where the model simply remembered the solution.”
So when a vendor shows you a benchmark score, the useful question is gentle but pointed: was the target a puzzle, or was it software somebody actually shipped?
There is one study that did the obvious experiment, and it is far more interesting than the benchmark scores.
Researchers built a test on a live network of 8,000 hosts across 12 subnets, in a real institutional environment rather than a lab. Then they put ten human security professionals, six existing agent frameworks, and a new multi-agent framework of their own onto the same network and compared results.
The AI framework placed second overall. It beat nine of the ten humans, finding nine valid vulnerabilities with 82% of its submissions holding up.
Sit with both halves of that. An agent outperforming nine out of ten professionals is a genuinely remarkable result, and anyone dismissing AI testing as a toy should read it carefully. And the best human still won, which is not what you would expect from the marketing.
The paper also noted the agents produced more false positives than the strongest humans and struggled with anything driven through a graphical interface. Its own conclusion leaned toward augmenting human testers rather than replacing them.
Roughly one in five AI submissions did not hold up. That is not a disaster, but somebody has to be the one who checks.
If you only remember one piece of evidence from this article, make it this one.
The AutoPenBench researchers ran the same agent against the same 33 tasks in two modes. In the first it worked entirely alone. In the second, a person broke the goal into ordered sub-tasks and handed them over one at a time, adjusting as the agent reported back. No manual exploitation, no human doing the actual work. Just a plan.
| Mode | Overall success | On real CVE tasks |
|---|---|---|
| Fully autonomous | 21% | 9% |
| Human wrote the plan first | 64% | 73% |
Success roughly tripled, and the effect was strongest exactly where autonomous agents are weakest. On real CVEs, one lonely task became three quarters of them.
The gains show up across categories too. Access control went from 20% to 80%. Web security from 29% to 57%. Cryptography from nothing at all to 25%.
What that suggests is quietly encouraging. The bottleneck for these agents is not their ability to execute. It is knowing what to attempt and in what order, which happens to be the thing an experienced tester is best at and finds least tedious.
This is why the framing of AI vs human penetration testing as a competition misses the point. The most capable configuration tested was not either one alone.
Survey data is softer evidence than a controlled study, but when a lot of people change their minds about AI vs human penetration testing at once, it is worth noticing.
Cobalt surveyed 450 practitioners alongside data from more than 16,500 pentests for its 2026 State of Pentesting report. Three findings stand out.
Organisations relying entirely on AI automation fell from 29% to 9% in a single year. Preference for a hybrid model rose 22 points to 47%. And 78% said fully automated scanning had missed critical vulnerabilities for them.
There is a fourth finding that gets less attention and probably deserves more. AI and LLM-related issues had the worst remediation rate of any category, 38% fixed against 62% still open, and the average time to fix them grew from 19 days to 36. Finding problems faster only helps if somebody closes them.
None of this means automation disappointed. It means the first wave of expectations was set too high, and the market has settled somewhere more realistic.
This section surprises people, so it is worth being precise rather than dramatic. Different rules ask for quite different things, and several do not mention penetration testing at all.
| Framework | What the text actually says | Needs a qualified person? |
|---|---|---|
| FedRAMP | Requires testing by an accredited 3PAO, and states plainly that automated scanning alone does not satisfy the control. Reports must show human-driven exploitation with evidence. | Yes, explicitly |
| DORA (EU) | Article 27 requires testers with “proven threat-intelligence and red-team expertise,” with accreditation or a formal code of conduct | Yes, by name |
| PCI DSS 4.0 | Requirement 11.4 asks for a “qualified internal resource or qualified external third party” with organisational independence. Notably, no specific certification is demanded, not even QSA or ASV | Yes, though “qualified” is left open |
| ISO/IEC 27001:2022 | The phrase “penetration test” appears nowhere in the 93 Annex A controls. Control 8.29 asks for appropriate security testing | No, method is left to you |
| SOC 2 | The Trust Services Criteria never use the words “penetration test” at all | No, it is auditor practice |
| NIS2 | The directive text does not say “pentest.” Supporting ENISA guidance recommends it | No, guidance only |
| HIPAA | The rule in force today requires no penetration testing. A proposed rule would add annual testing, still pending | Not currently |
Two things follow from this, and they pull in opposite directions.
If you are chasing FedRAMP or fall under DORA, a purely automated test will not carry you across the line. The wording is unusually direct about wanting a person.
But if you are working toward SOC 2 or ISO 27001, nobody is going to hand you a rule that says a human must do it, because no such sentence exists in either standard. What auditors want is evidence that you test properly and act on what you find.
PCI DSS sits in the interesting middle. It asks for a “qualified” resource with independence, and it does not say whether a well-supervised AI-assisted test satisfies that. Anyone who tells you the answer with total confidence is guessing, and if you are relying on it, ask your assessor rather than a blog. Ours included.
Prices vary so much by scope that any single number deserves suspicion, but here is the honest shape of it.
For human testing, the widely quoted industry average is around $18,300 per engagement. That number comes from a compilation of about ten industry sources rather than a rigorous study, so treat it as a rough middle.
Cobalt’s own delivery data is more concrete. A medium-complexity web application typically takes four to eight days of tester time and runs $10,000 to $25,000.
For AI testing, on-demand products from established vendors sit around $4,000 to $8,000 per test. Subscription agents start far lower. XHack’s own individual plans begin at $20 a month and company plans at $560, though those buy a tool you operate rather than a finished report.
Underneath the pricing sits a supply problem no budget fixes. ISC2’s 2025 workforce study put global demand at 10.2 million security professionals against a workforce of 5.5 million, and 95% of respondents reported at least one skills gap on their team.
That shortage is the honest argument for automation. Not that agents are cheaper than people, though they are, but that there were never going to be enough people.
Rather than settling AI vs human penetration testing in the abstract, work out which situation you are in.
Choose human-led testing when you need a report for FedRAMP, DORA or PCI DSS, when your application’s real risk lives in business logic rather than infrastructure, when you are testing something genuinely unusual, or when you need someone accountable who can sit in a room and explain the findings.
Choose an AI agent when you ship frequently and want every release checked, when large parts of your estate currently get no testing at all, when you want to catch the obvious classes early so your human budget goes on the hard problems, or when you simply cannot afford five figures per engagement.
Choose both, in sequence, when you can. Run the agent continuously for coverage, then bring a human in periodically for depth and for the report your auditor will accept. The evidence in this article points fairly consistently at that arrangement.
If you are still weighing providers, nine questions that expose a weak pentest provider is a more useful filter than any vendor comparison, and when to choose a vulnerability assessment instead is worth reading if you are not certain you need a full pentest yet.
So, the part where I talk about what we built. I have tried to earn it by being fair above.
XHack did not pick a side in AI vs human penetration testing, because the research above is fairly clear that picking a side is the weaker option. Instead the human step is built into the product rather than sold separately.
Our agent runs a seven-stage pipeline, and the sixth stage is a person. Prompt, then planner, then orchestrator, then executor, then findings, then human triage and verification, then the report. Every stage writes to an audit log, and every finding traces back to the exact tool invocation that produced it and the reviewer who signed it off.
Read that carefully, because the human is at stage six, not stage one. The agent does the finding. It picks targets, plans, executes and produces evidence without being led, which is how it found an OAuth secret in 25 minutes on a single line of input.
The reviewer exists for a different reason. Somebody has to be accountable for what reaches a client, roughly one AI submission in five does not survive scrutiny industry-wide, and no regulator accepts a report nobody signed. That is an accountability decision rather than a limit on what the agent can do.
If you want the finding, the agent gets there faster and cheaper than we can staff. If you want a report you can hand an auditor, a person has to put their name on it. We built for both rather than pretending the second problem does not exist.
If you would rather not run anything yourself, our researchers do the testing for you.
Managed engagements start at $2,500 for a single web application or up to 50 host IPs, delivered in four days with a retest included. $5,000 covers up to three applications across web, host, API and mobile. $12,000 covers up to 20 applications internally and externally.
Prices are published, scope is agreed before anyone starts, and the reports are built for compliance review. That is human and agentic testing together rather than one pretending to be the other.
The people doing that work hold OSCP and OSCP+ certifications, with Synack Red Team and AI security credentials on the team as well.
I will also say plainly that we are a small shop. Six clients secured and 32 assessments done, which is what our site says and what is true. Our published engagements run to fourteen write-ups, four of them found by the agent alone. If you need a vendor with a decade of enterprise references, we are not that yet.
If you want to try the agent first, there is a seven-day trial with no card required, though we do verify your identity before handing over offensive tooling, which seems reasonable given what it does.
And if we are not the right fit, an honest comparison of the AI pentesting tools worth considering includes plenty of alternatives, several of them better funded than us.
For finding vulnerabilities, agents are already faster and cheaper than staffing the work, and our own results show it: an OAuth secret confirmed and proven in about 25 minutes on one line of input for roughly $5. What they cannot yet do is replace the person who decides what a finding is worth and signs the report, which is what FedRAMP and DORA actually require. The published research still shows autonomous agents improving sharply when a human supplies a plan first, 21% to 64% in the AutoPenBench study, so the strongest configuration remains a person and an agent rather than either alone.
It is accurate enough to be useful and not accurate enough to go unchecked. On a live network, 82% of one agent’s submissions held up, meaning roughly one in five needed correcting. Even Anthropic, reporting on its own model, found human reviewers agreed with its severity assessment 89% of the time, so about one finding in nine required a human to step in. Both numbers are good. Neither is good enough to send a report to a client unread.
It depends entirely on the framework, and the differences are larger than most people expect. FedRAMP states that automated scanning alone is not sufficient and wants human-driven exploitation evidence. DORA asks for proven human red-team expertise by name. PCI DSS 4.0 requires a “qualified” and independent resource without defining whether AI-assisted testing counts. ISO 27001 and SOC 2 never mention penetration testing in their actual text at all. Check the wording that applies to you, and when it is ambiguous, ask your assessor before your vendor.
Per test, considerably. On-demand AI tests from established vendors run roughly $4,000 to $8,000, subscription agents start in the tens of dollars a month, and one study measured agent compute at around $18 an hour against $60 for a professional. A human engagement more commonly lands between $10,000 and $25,000 for a medium-complexity application. The fair comparison is not price alone, though, because the two produce different things. One gives you continuous coverage, the other gives you a defensible report.
Anything that depends on understanding what your application is for. Business logic flaws, where every individual request is valid but the sequence produces something you never intended, are the most cited example. Alongside those sit multi-step abuse of legitimate features, novel vulnerability classes with no prior pattern to learn from, anything reachable only through a graphical interface, and the entire social engineering surface. Agents also tend to under-weight impact, correctly identifying an issue while missing that it happens to sit on your payments path.
Start with one scoped human test if you have never had one, because a first engagement usually surfaces structural problems that shape everything after it, and because you will likely need the report for a customer or an auditor eventually. Then use an agent for continuous coverage between engagements. That order matters. An agent pointed at a system nobody has ever properly assessed will find real issues while quietly leaving the important ones untouched.
AI vs human penetration testing turns out to be the wrong question, and pleasantly so, because the right answer costs less than picking the expensive side and catches more than picking the cheap one.
The evidence is fairly consistent. Agents are quick, tireless, inexpensive and genuinely capable, they beat most human testers on most routine work, and on raw speed to a confirmed finding nothing staffed by people comes close.
They also miss the flaws that depend on understanding your business, get roughly one finding in five wrong, and are not accepted on their own by the strictest regulators. Humans are slower, scarcer and dearer, and they stay better at judgment, novelty and accountability.
Put a person in front of the agent to plan, and behind it to verify, and you get most of both. That is not a compromise position, it is simply what the strongest study on the subject measured.
If you are choosing right now, start from your constraint rather than the technology. Regulated and needing a report, hire a human. Shipping weekly with gaps you never test, get an agent running. Somewhere in between, which is most people, do both and let each cover for the other.
And whichever way you go, the finding that never gets fixed is worth nothing at all. Thirty-eight percent remediation on AI findings is the most quietly damning number in this entire article, and no testing method fixes that one for you.
Categories
Related articles
