XHack
Author
Table of Contents
23
By Salman Khan, OSCP+, Founder of XHack, SRT (Synack Red Team member)
Read this in 30 seconds: You have been told an AI agent can run a security engagement end to end. It cannot yet. It is fast at recon, then breaks on credentials and shell state, the part you wanted to hand off.
- The tools are free but the audits still cost $15,000. Garak and PyRIT are open source and cover most of a paid audit. You are paying a vendor to prove the bug causes real damage and to have a human confirm it.
- Fully autonomous agents look great in a demo but lose to a human plan. One study hit 85% success against Meta’s Llama Scout with no human-written code. Another found the same agents did worse without an expert-written plan.
- Your model refuses you, and it did not refuse them. A state-backed group ran 30 targets with AI doing 80-90% of the work by telling the model it was an authorized pentest. The same sentence you type before it says no.
- Poisoning takes 250 documents, not a percentage of the training data. Anthropic found that number held at 600 million parameters and at 13 billion.
- Two jobs share this one name and the prices differ by 10x. Testing a model costs $8,000 to $150,000. Using AI to test everything else starts at $2,500.
Every vendor in this category sells the same promise: point the agent at the target, come back to a finished report.
I have run both halves of this work, adversarial testing against LLM-backed applications and conventional engagements where agents handled the recon, and that promise breaks in the same place every time. The agent is genuinely quick through the wide, repetitive part. Then it hits something that needs it to remember what it did five steps ago, and it starts guessing.
That is worth knowing before you buy anything or build a program around it. It is also not the only problem here.
The bigger one is the name. AI red teaming describes two different jobs, and the prices differ by more than 10x. So you get vendors pitching a prompt-injection scanner to a bank that wanted its Active Directory tested, and network pentesters pitched to a team that just shipped a customer-facing agent with database access. Nobody is doing the homework on the distinction, so I did.
The confusion is almost entirely a grammar problem. The phrase AI red teaming describes both attacking AI and attacking things with AI, and those are separate trades.
Red teaming of AI treats the model as the target. You make a system behave in ways its builders did not intend: leak its system prompt, ignore its guardrails, call a tool it should not, act on instructions buried in a retrieved document, or absorb a backdoor during training. The adversary is anyone who can send text, images or files in.
Red teaming with AI treats the model as the operator. You run a conventional adversary simulation against servers, identities, cloud accounts and applications, except AI agents handle recon, hypothesis generation, exploit adaptation and reporting. The target is your normal infrastructure. The AI is just labor.
They overlap in one place: when an organization runs AI agents in production with real permissions, attacking those agents is attacking the infrastructure. That is why the 2026 OWASP list moved Excessive Agency up the rankings. But treating the two halves of AI red teaming as interchangeable is how budgets get burned.
| Dimension | Red teaming of AI | Red teaming with AI |
|---|---|---|
| Target | Models, agents, RAG pipelines, guardrails | Networks, apps, cloud, identities, people |
| Core skill | Adversarial ML, prompt engineering | Exploit dev, post-exploitation |
| Frameworks | OWASP LLM Top 10, MITRE ATLAS, NIST AI 100-2 | MITRE ATT&CK, PTES, TIBER-EU |
| Typical finding | Indirect injection to data exfiltration | Chained privilege escalation to domain admin |
| Deliverable | Attack success rate per vulnerability class | Attack path narrative, detection gaps |
| Regulatory driver | EU AI Act Art. 55, NIST AI RMF | PCI DSS, DORA, SOC 2, ISO 27001 |
| 2026 price band | $8,000 to $150,000 | $2,500 to $100,000+ |

One thing to take from this playbook: write down which half of AI red teaming you are buying before you talk to a single vendor. Scope, price, who you hire, what a “finding” even looks like, all of it follows from that one decision.
So which tools actually do which half? Here is the whole AI red teaming field. The pricing column reads “custom quote” wherever a vendor refuses to publish, because pretending otherwise helps nobody.
| Tool / Platform | What it actually does | Pricing (2026) | Best for | The catch |
|---|---|---|---|---|
| NVIDIA Garak | Model scanner, 37+ probe modules: jailbreaks, encoding, injection | Free (Apache 2.0) | Broad first-pass model coverage | Single-turn focused, results need interpretation |
| Microsoft PyRIT | Multi-turn and multi-modal orchestration, crescendo and TAP | Free (MIT) | Depth a scanner structurally misses | A framework, not a product. You write the harness |
| Promptfoo (OpenAI) | App-layer testing, YAML configs, CI/CD gates, 50+ vuln types | Open source + enterprise | Regression testing AI apps in a pipeline | Acquired by OpenAI March 2026, roadmap serves Frontier |
| DeepTeam | 40+ vulnerability types mapped to OWASP LLM Top 10 | Free + cloud tier | Teams wanting OWASP and NIST mapping built in | Younger ecosystem, smaller probe library than Garak |
| Mindgard | Shadow-AI discovery, adversarial testing, runtime defense | Custom quote | Enterprises needing AI asset discovery first | Opaque pricing, enterprise sales motion |
| HiddenLayer | Model-agnostic automated red teaming (AutoRT) | Custom quote | Government and large-enterprise, third-party models | Enterprise-only, no self-serve path |
| Dreadnode | Agentic platform, 45+ attacks, 450+ transforms, 130+ scorers | Custom quote | Research-grade offensive AI work | Built for operators, steep learning curve |
| XHack AI | Both disciplines: AI Probe (350+ payloads) plus an autonomous offensive agent | $20 to $150/mo individual, $560 to $3,000/mo company, $2,500+ VAPT | Offensive ops and AI testing without two vendors | Smaller company, no federal track record, we say so |

Three things stand out. The open-source AI red teaming tools are genuinely good and cost nothing, so there is no budget excuse for skipping a first pass. The commercial platforms mostly refuse to publish a price, which tells you how their sales process works. And almost every entry covers one side of the split, which is why teams end up paying two vendors for one problem.
The reference map for this half of AI red teaming changed in August 2026. The OWASP Top 10 for LLM Applications 2026 landed on August 3, and the reshuffle shows where real damage is happening.
Excessive Agency jumped to LLM03. In the previous edition it sat well down the list as a theoretical concern. Then everyone shipped agents with tool access, database credentials and the ability to send email, and the theoretical concern started emptying accounts.
Hidden Context Exposure entered at LLM08, a new category covering the gap between what a user sees and what the model receives. Retrieved documents, tool outputs, injected memory, system scaffolding: invisible to the person typing, fully live as instructions.
The 2026 list in order: Prompt Injection, Sensitive Information Disclosure, Excessive Agency, Supply Chain, Data and Model Poisoning, Unbounded Consumption, Misinformation, Hidden Context Exposure, Vector and Embedding Weaknesses, Improper Output Handling.
Direct injection (“ignore your instructions”) is the version people demo. It is also the version that mostly gets patched.
Indirect injection is the one that actually pays, and where AI red teaming earns its fee. You do not talk to the model. You put instructions where it will read them on its own: a webpage it browses, a PDF it summarizes, a ticket it triages, a code comment it reviews, a calendar invite it parses. The user never sees the payload. The model obeys because to a language model there is no structural difference between “content” and “command.”
I have written before on how prompt injection works against production chatbots, and the pattern has not changed. The blast radius has: in 2024 a successful injection produced a rude chatbot, in 2026 it produces an agent holding your Postgres credentials and a mail relay.
Prompt injection is the attack every AI red teaming engagement tests for. Poisoning is the one almost nobody budgets for, and it turns out to be cheap.
Anthropic, the UK AI Security Institute and the Alan Turing Institute ran the largest data-poisoning study to date and found that as few as 250 malicious documents can install a backdoor in a large language model, with that number staying roughly fixed regardless of model size. A 13-billion-parameter model trained on more than twenty times the data of a 600-million-parameter model was backdoored by the same 250 documents. At 100 the backdoor did not hold reliably. At 250 it did.
The assumption the whole industry ran on, that an attacker needs a percentage of training data, is wrong. They need a small, fixed, achievable number of documents on the open internet.
Translation: if your model or fine-tune touches scraped public data, poisoning belongs in your AI red teaming scope whether you budgeted for it or not.
AI red teaming needs a shared vocabulary, otherwise two reports describing the same bug read like different findings. Three reference sets handle that, and they complement each other rather than compete.
MITRE ATLAS is the adversary TTP map for AI systems, the ATT&CK equivalent. As of v5.4.0 in February 2026 it covers 16 tactics and 84 techniques, up from 15 and 66 in October 2025. That growth is the story: the technique count rose 27% in four months. Two tactics are unique to ATLAS, ML Model Access and ML Attack Staging, and they are exactly the ones a team borrowing an ATT&CK playbook would miss. If you already run MITRE ATT&CK-driven penetration testing, ATLAS slots into the same threat-modeling workflow.
NIST AI 100-2e2025 (March 2025) is the taxonomy layer, classifying attacks across five dimensions: system type, lifecycle stage, attacker goal, attacker capability, attacker knowledge. It is dry. It is also what makes two AI red teaming reports from different vendors comparable.
OWASP’s Gen AI Red Teaming Guide is the methodology layer, structuring engagements around lifecycle phase and risk scope (model, infrastructure, ecosystem) rather than a flat checklist.
Use all three. ATLAS for what attackers do, NIST for what to call it, OWASP for how to run the engagement.
That covers attacking the model. The other half of AI red teaming points the model at everything else, and it is where I spend most of my own time.
The pitch is straightforward: recon, enumeration, hypothesis generation and initial exploitation are labor-intensive, repetitive and highly parallelizable. That is the exact shape of work agents are good at. Hand it over and a human operator starts the engagement at hour four instead of day four.
The measured results are better than skeptics expected and worse than vendors imply. Both halves matter.
The honest answer sits between the sales deck and the backlash, and two 2026 papers bracket it neatly. The strongest published result comes from Dreadnode. In Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours (Dheekonda, Pearce and Landers, May 2026), the team reports an 85% attack success rate with severity up to 1.0 against Meta’s Llama Scout, using zero human-developed code, driving 45+ attacks, 450+ transforms and 130+ scorers from natural-language goals. Weeks compressed to hours, as the title says.
Now the counterweight, which nobody quotes because it is inconvenient. In Autonomous Adversary: Red-Teaming in the Age of LLM (Mamun, Gaber, Buffett and Saad, ACISP 2026), researchers compared fully autonomous execution, self-scaffolded planning and expert-defined action plans across lateral-movement scenarios. Expert-defined action plans won. Fully autonomous execution did not. In the authors’ own words, failure remained frequent across all three modes, driven by brittle command invocation, environmental instability and recurring errors in credential management and state handling.
Put both papers side by side and the 2026 picture is clear: agents are excellent at breadth and adaptation inside a plan a human wrote, and unreliable the moment they are asked to write the plan. So when a vendor sells “fully autonomous, no humans needed,” they are selling you the configuration that measured worse.
On November 14, 2025, Anthropic disclosed the first reported AI-orchestrated cyber espionage campaign. A Chinese state-linked group tracked as GTG-1002 used agentic tooling against roughly 30 targets across technology, finance, chemical manufacturing and government, with the AI executing an estimated 80% to 90% of the tactical work and humans stepping in only at strategic decision points.
Here is the detail that matters. The operators got past the model’s safety filters by convincing it that it was performing authorized penetration testing.
No clever jailbreak string was involved. The filter was beaten by someone claiming to do AI red teaming for a living. The exact professional use case that gets legitimate practitioners refused every day worked fine as a cover story for a state-linked espionage crew, because a refusal filter cannot verify authorization. It can only pattern-match on phrasing, and phrasing is free.
That is the whole argument for verified access over content filtering, and why I wrote a piece on why AI models refuse legitimate security work.
Knowing where agents help and where they quit only matters if it changes how you run an AI red teaming engagement. This is the AI red teaming process I actually run, adapted from the OWASP Gen AI methodology and ordinary engagement structure. It works for both disciplines with the obvious substitutions.

Phases 5 and 6 are the ones vendors quietly skip. Chaining a finding into proven impact takes skill, and reproducing every result from a clean state takes time. Neither shows up on a feature comparison sheet, so ask about both by name before you sign anything.
The AI red teaming services market went from $1.75 billion in 2025 to $2.26 billion in 2026, a 28.8% compound growth rate. Fast growth plus no standard pricing unit equals a market where quotes for identical scope differ by 10x.
| Engagement type | 2026 price band | What you get |
|---|---|---|
| Open-source tooling, in-house | $0 plus staff time | Garak, PyRIT, DeepTeam. Real coverage, zero hand-holding |
| One-time AI system audit | $8,000 to $25,000 | Single chatbot or RAG app, point-in-time |
| Multi-agent system engagement | $50,000 to $150,000 | Tool chains, orchestration, cross-agent attacks |
| Continuous AI testing | From $5,000/month | Recurring runs, periodic human review |
| Enterprise annual contract | $75,000 to $400,000/year | Two or more AI applications, ongoing |
| Conventional red team (human) | $10,000 to $100,000+ | Adversary simulation, no AI-specific scope |
| XHack AI, individual | $20 to $150/month | Autonomous agent plus AI Probe, single user |
| XHack AI, company | $560 to $3,000/month | Multi-user, RBAC, GitGuard, AI Probe |
| XHack VAPT engagement | $2,500 to $12,000 | Human-led, AI-assisted where authorized |

Two things about AI red teaming pricing are worth saying out loud.
First, the open-source row is not a joke entry. A competent engineer with Garak, PyRIT and a week produces most of what a $15,000 point-in-time audit delivers. The higher tiers buy chaining, verification, and someone whose name is on the report.
Second, the enterprise band is priced on budget, not on cost. Running an agent against an API does not cost $400,000 a year. Enterprise sales motions do. Quoted in that band, make the vendor itemize human hours, the only line item that scales with real work.
For how these compare against traditional testing, I went deep on AI penetration testing costs separately.
Price is one reason to run AI red teaming. For a small group of companies, the law is now another.
On August 2, 2026, the European Commission’s enforcement powers over general-purpose AI model providers came into application. The AI Office can now request documentation, evaluate models directly, order corrective measures, restrict or withdraw a model from the EU market, and fine.
Article 55 applies to GPAI providers with systemic risk, requiring documented adversarial testing, structured vulnerability reporting, and evidence of mitigation. Penalties under Article 99 reach €15 million or 3% of global annual turnover, whichever is higher.
Three qualifications, because precision beats alarm:
The US has no equivalent mandate. NIST’s AI Risk Management Framework and Generative AI Profile stay voluntary, which in practice means they surface as contractual requirements in enterprise procurement rather than as law.
For most organizations reading this, though, the regulator is not the reason to do any of this. The reason is that you shipped an agent with database credentials and nobody has tried to break it yet.
Every AI red teaming vendor page reads like the problem is solved, but six things are still genuinely broken, and you should hear them from someone selling into this market.
Non-determinism breaks regression testing. The same prompt against the same model can succeed and fail on consecutive runs. Attack success rate is a probability, not a boolean, and a “fixed” vulnerability may just be a lower success rate. Any tool reporting binary pass/fail on a probabilistic system is lying to you by rounding.
Guardrail patches regress. A model update, a system-prompt tweak or a new retrieval source can silently reopen a class you closed three months ago. Point-in-time AI red teaming has a shorter useful life than point-in-time network testing, and that is already short.
Automated tools over-report. Scanners flag refusals as failures, harmless outputs as leaks, and duplicate one issue across a dozen probe categories. Skip human verification and you ship noise, and engineering teams learn to ignore you.
Agent autonomy is still unreliable. Brittle command invocation and state-handling errors are not edge cases, they are the dominant failure mode in fully autonomous operation.
Only about a quarter of organizations run proactive AI security testing at all. Tooling maturity ran well ahead of practice maturity, which is a polite way of saying most production AI systems have never been attacked by anyone friendly.
And the deepest one: coverage is unbounded. A network has a finite number of hosts and ports. A language model has an infinite input space. You are never “done” with AI red teaming, only ever “current.” Price accordingly.
Four failures I see repeatedly, in rough order of money wasted.
Buying the wrong discipline. This is still the most expensive mistake in the category, because a prompt-injection audit will not test your network and a network pentest will not test your agent.
Scoping the model instead of the system. The interesting vulnerabilities are almost never in model weights. They live in tool definitions, the retrieval corpus, permission boundaries and output handling. A scope that says “test the LLM” and stops has scoped out the actual attack surface. Same pattern as LLM and RAG security testing: the pipeline is the target, not the model.
Treating a scanner run as an engagement. Garak finishing without errors is step three of seven, not an assessment. Automated breadth without manual depth produces low-severity refusal bypasses and zero evidence of real impact.
Never re-testing. Given how fast guardrails regress, annual AI red teaming is closer to theater than security. Continuous, or quarterly at minimum, is the honest cadence for anything with tool access.
This is the part where I talk about my own company doing AI red teaming, so read it with that in mind. I will tell you where XHack does not fit as clearly as where it does.
The structural point: XHack does both halves of AI red teaming, and almost nobody else does. The AI-security vendors test models and cannot run a network engagement. The offensive firms run engagements and have no adversarial ML practice. Buying both means two vendors, two scoping calls, and two reports that never reference each other.
For red teaming of AI, our platform includes AI Probe, testing AI systems against 350+ attack payloads across the OWASP LLM Top 10 categories: prompt injection, data leakage, jailbreaks and safety failures, with interactive bypass testing for findings you want to push further by hand.
For red teaming with AI, our Red Team Operations service covers adversary emulation, custom tooling, social engineering assessment design and engagement planning, with the XHack AI agent running autonomous recon, discovery and exploit chaining in parallel with human testers. Sub-agents fan out at once, and a senior human reviews everything before it reaches a report, because phase 6 is not optional. Same agent behind our autonomous AI hacking agent work and our AI penetration testing practice.
On privacy, which matters more in AI red teaming than in most categories: XHack does not store user data and is privacy-focused by design. Pentest chats and session data stay on your own local computer and you can delete them any time. Run entirely offline against local Ollama models with nothing leaving your network, or bring your own key, encrypted and never displayed. When the whole point of the engagement is documenting how your AI system fails, keeping that evidence off a vendor’s servers is not a nice-to-have.
Pricing, plainly. Individual plans run $20 to $150 per month for solo researchers and bug hunters, including someone at a company buying for personal use. Company plans run $560 to $3,000 per month at app.xhack.io for multi-user teams with RBAC, GitGuard and AI Probe. Human-led VAPT starts at $2,500 (Essential), $5,000 (Assurance) and $12,000 (Comprehensive), scoped to the assessment, with AI-agent assistance used only if you authorize it. There is a 7-day free trial on individual and company plans, no credit card required, nothing to cancel when it ends.
Now the honest part. We are a smaller company with no federal or defense track record, so if procurement requires a decade of government deployments, HiddenLayer clears that bar and we do not. If you need AI asset discovery across a sprawling enterprise before you can even test, Mindgard is ahead of us. And if you want a cheap one-time scan with a nice cover page, we are the wrong fit entirely.
Want a quote? Tell us which half of AI red teaming you need and we will scope it. Not sure which? Book a free consultation and we will tell you straight, even if the honest answer is that open-source tooling and one competent engineer serves you better than paying us. Brutal honesty is kind of our thing.
AI red teaming is adversarial testing that deliberately tries to make an AI system fail in ways its builders did not intend. The term covers two distinct disciplines: attacking AI systems themselves (prompt injection, jailbreaks, data poisoning, excessive agency, the OWASP LLM Top 10), and using AI agents to run conventional offensive engagements against ordinary infrastructure faster than a human team alone. Both are legitimate uses of the term, they need different skills and tooling, and the most useful thing a buyer can do is decide which one they need before contacting vendors.
Traditional penetration testing works against deterministic, bounded systems: a port is open or closed, an input is sanitized or it is not, a fixed vulnerability stays fixed. AI red teaming works against probabilistic systems with an effectively infinite input space, where the same attack succeeds on one run and fails on the next, where “attack success rate” replaces pass/fail, and where a model update can silently reopen a vulnerability you closed last quarter. The methodology is recognizably offensive security, but the reporting, the cadence and the definition of “remediated” are all different.
A one-time audit of a single AI system such as a chatbot or RAG application typically runs $8,000 to $25,000, multi-agent engagements $50,000 to $150,000, continuous testing from around $5,000 per month, and enterprise annual contracts covering two or more AI applications commonly land between $75,000 and $400,000 per year. Open-source tooling like Garak, PyRIT and DeepTeam costs nothing beyond staff time and covers a genuine portion of what paid audits deliver. XHack’s individual plans run $20 to $150 per month, company plans $560 to $3,000, and human-led VAPT engagements start at $2,500.
Start with NVIDIA’s Garak for broad model-layer coverage, since it ships 37+ probe modules and gives a fast first-pass picture. Add Microsoft’s PyRIT for multi-turn and multi-modal depth, particularly crescendo attacks and TAP, which single-shot scanners structurally cannot find. Use DeepTeam if you want findings mapped to the OWASP LLM Top 10 and NIST AI RMF without building that mapping yourself, and Promptfoo to wire AI red teaming checks into CI/CD as regression gates. All four are open source, and Garak plus PyRIT covers breadth and depth better than any single tool alone.
Article 55 requires providers of general-purpose AI models with systemic risk to conduct documented adversarial testing, report identified vulnerabilities in a structured way, and evidence their mitigations. The Commission’s enforcement powers over GPAI providers came into application on August 2, 2026, with penalties reaching €15 million or 3% of global annual turnover. New providers have been in scope since August 2, 2025, while providers with models already on the market have until August 2, 2027. This binds model providers, not every organization consuming an AI API, so most companies using a third-party LLM inherit the obligation through their vendor rather than carrying it directly.
Not reliably, and the research says so clearly. Agentic platforms produce strong results inside human-defined plans, with one May 2026 study reporting an 85% attack success rate against Meta’s Llama Scout using zero human-written code. But a separate ACISP 2026 study comparing fully autonomous execution, self-scaffolded planning and expert-defined action plans found expert-defined plans delivered the highest task completion, with failures frequent across all modes due to brittle command invocation and credential and state-handling errors. The honest reading: automation owns breadth and adaptation, humans still own the plan, the chaining and the verification.
Two things about AI red teaming are true at once. Agents have genuinely compressed the wide, repetitive part of the work from weeks into hours, and they still cannot finish a job without someone competent holding the plan.
Decide which one you are buying. If it is red teaming of AI, your map is the OWASP LLM Top 10 2026 and MITRE ATLAS, your first tools are free, and your hardest problem is that coverage is unbounded and guardrails regress. If it is red teaming with AI, your map is still ATT&CK, the agents are labor rather than magic, and the research is unambiguous that they perform best inside a plan a human wrote.
GTG-1002 settled the strategic question. Attackers already run AI-orchestrated operations at 80% to 90% autonomy, and they got past the safety filters by claiming to be authorized red teamers. Legitimate practitioners get refused by the same systems a state-linked espionage crew talked its way through. Content filtering is not what fixes that asymmetry. Verified access is.
So pick your discipline, scope the system and not just the model, run the free breadth tools before you pay anyone, and hold whoever you hire to phases 5 and 6. The rest is detail.
Start with whatever you shipped that holds credentials. That is where it will happen.
Related articles

Read this in 30 seconds: Unrestricted AI coding assistant security means an AI that will actually write the code a [&hel...

Read this in 30 seconds: AI penetration testing cost breaks into three genuinely different markets in 2026, individual r...

Read this in 30 seconds: If you’re searching for a “WormGPT alternative,” it’s worth pausing on ...