
Table of contents
17
By Salman Khan, OSCP+, Founder of XHack, SRT (Synack Red Team member)
Read this in 30 seconds: AI red team operations means using AI inside an authorised, objective-based attack simulation, and it helps most in planning, logging and reporting, not in running the whole operation unattended. The evidence says the tooling around the model matters more than the model, and the rules say a human must own the scope. I read the primary sources this week.
- A red team operation is not a bigger pentest. NIST’s glossary describes a red team as people authorized and organized to emulate an adversary, and the goal is an objective such as reaching a critical asset, not a list of every bug.
- Raw AI agents struggle with multi-host operations. In a Carnegie Mellon and Anthropic study on 40 emulated networks, leading LLM-assisted systems reached a critical asset in 3 of 40, but the same models with a purpose-built abstraction layer reached one in 37 of 40.
- Regulators already set the rules in finance. DORA requires threat-led penetration testing at least every 3 years on live production systems, and Article 27 sets tester requirements that include accreditation or a code of conduct and professional indemnity insurance.
- The legal AI route is organisation-only. Anthropic’s Red Team Access tier covers authorised red teaming, but it is not open to individuals, and real-time blocks remain on things like deploying ransomware.
- A rules-of-engagement gate costs almost nothing in AI red team operations. Any action an AI agent takes should be checked against scope, window and permitted tactics, and I include a tested script that does it.
An AI agent can be fast at an operation and still be a liability if nobody decided what it was allowed to touch.
Search for AI red team operations and you will find two kinds of pages: tool lists that skip the rules, and vendor pages that skip the limits. This guide is the practical version. It covers what AI red team operations are, which phases AI genuinely speeds up, what the research shows about autonomy, how finance regulators treat threat-led testing, and how to keep an AI agent inside its authorisation during AI red team operations.
It contains no attack techniques, no payloads and no evasion advice. It is written for authorised engagements only. I run XHack, so I also show where we fit and where we do not.

A red team operation is an authorised attack simulation with a goal, and AI red team operations add AI to that process. NIST’s glossary, quoting CNSSI 4009, defines a red team as a group “authorized and organized to emulate a potential adversary’s attack or exploitation capabilities”. Its purpose is to show the impact of a successful attack and what works for defenders in a real environment.
That makes AI red team operations different from a penetration test. A pentest is scope-limited and tries to find as many vulnerabilities as it can in one target. A red team operation is objective-based: it tries to reach a defined goal, such as a specific database or an administrator account, by whatever authorised path exists, and it tests whether the defenders notice. Our guide on AI penetration testing covers the first job, and this one covers the second.
Two more definitions keep the rest of this article clean.
AI helps most where the work is reading, organising and writing, and humans must hold everything that carries risk. Here is each phase of AI red team operations, what AI can do in it and what a person has to own. It stays at concept level on purpose.
| Phase | What happens | Where AI helps | What a human must hold |
|---|---|---|---|
| Objectives and scenario | Agree the goal and the threat to emulate | Summarise public threat reports into candidate scenarios, map them to ATT&CK | Choosing objectives that match business risk |
| Rules of engagement | Document scope, exclusions, windows, stop procedure | Draft checklists, spot gaps and contradictions | Signing off, and the emergency stop |
| Reconnaissance | Learn the in-scope attack surface | Organise notes, deduplicate and correlate findings on approved assets | Deciding what counts as in scope |
| Execution | Pursue the objective inside the rules | Plan steps, run in-scope tooling under supervision | Every boundary decision, anything touching people or safety systems |
| Observation and logging | Record every action with a timestamp | Keep structured logs, flag actions outside the rules | Reviewing the log, owning the evidence |
| Reporting and purple debrief | Attack narrative, detection gaps, fixes | Draft the narrative, build the timeline against blue team events | Judging severity and what to recommend |

Notice the pattern in AI red team operations. AI is strongest at the paperwork around the operation, which is where operators lose hours, and at planning steps for a person to approve. The riskiest phase, execution, is also where it needs the most supervision.
The best study on this question says the surrounding system matters more than the model. In “Incalmo”, researchers at Carnegie Mellon University and Anthropic built MHBench, a benchmark of 40 emulated multi-host networks with 22 to 50 hosts each. Success meant acquiring at least one critical asset. The paper was first submitted in January 2025 and last revised in November 2025.
The results are stark.

Three honest caveats apply. These were emulated networks, not production environments with real defenders. The authors call the technology dual-use and say they open-sourced the code and disclosed the results to LLM vendors. And one study is not the whole field.
Translation: if you evaluate a product for AI red team operations, ask about the orchestration, scope control and logging, because that is where the difference showed up. A better chat model alone is not the answer.
If you work in EU finance, the rules for AI red team operations are already written, and AI use is a question about your testers. Two frameworks matter.
DORA (Regulation (EU) 2022/2554) requires in-scope financial entities to carry out advanced testing by means of threat-led penetration testing (TLPT) at least every 3 years, and the competent authority can raise or lower that frequency. Article 26 says each TLPT must cover several or all critical or important functions and be performed on live production systems. DORA applies from 17 January 2025.
Article 27 sets who may test. Testers must be of the highest suitability and reputability, have specific expertise in threat intelligence, penetration testing and red team testing, be certified by an accreditation body or adhere to formal codes of conduct, provide an independent assurance or audit report, and carry professional indemnity insurance that covers misconduct and negligence. Internal testers need authority approval, and an entity using them must contract external testers every three tests.
TIBER-EU, the ECB’s framework for threat intelligence-based ethical red-teaming, was published in May 2018 and updated in 2024 to align with the DORA technical standards. It defines five teams: the blue team, a threat intelligence provider, the red-team testers, a small control team that alone knows about the test, and the TIBER cyber team at the authority. It describes the outcome as not a pass or fail, and finishes with a test summary report and an attestation for mutual recognition.

I could not retrieve the 2025 technical standard on TLPT or the Bank of England’s CBEST page, so check those directly if they apply to you. This is not legal advice. What matters here: none of these rules bans AI red team operations, but they put responsibility on named, qualified testers, so a provider’s AI agent does not remove any of the Article 27 requirements.
Legal AI red team operations rest on three things: written authorisation, the provider’s rules and logging you can show a client. The first is the law, and the second is the contract with your AI vendor.
On the vendor side, Anthropic’s Cyber Verification Program has a Red Team Access tier for authorised penetration testing and red teaming. According to Anthropic’s page, it is for organisations such as in-house red teams, government red teams and security and penetration testing firms, individuals are not eligible, and organisations can only perform adversarial testing against systems they are authorised to test. Real-time blocks remain on actions that could cause physical harm or mass disruption, with deploying ransomware, damaging physical systems and pen testing high-risk safety systems as examples.
Other conditions are worth reading before you apply: data retention is required for enrolment, applicants sit in Defense Access while Red Team review takes a few weeks, and the page was updated on 18 November 2026 to change plan availability. OpenAI’s Daybreak Red is similarly limited to approved business and enterprise organisations. See Anthropic’s Cyber Verification Program page for the current terms.
On the client side, the habits for AI red team operations are the same as any engagement:
The cheapest safety control in AI red team operations is a script that checks every action an agent took against the rules. This one reads a rules-of-engagement file and an action log, and flags anything out of scope, excluded, outside the window or using a tactic the client did not permit. It uses only the standard library, and I ran it before publishing.
The rules file uses a reserved documentation network, so nothing in it points at a real system:
{
"in_scope": ["203.0.113.0/24"],
"excluded": ["203.0.113.200/29"],
"window": {"start": "2026-12-01T09:00", "end": "2026-12-05T18:00"},
"tactics": ["Reconnaissance", "Discovery", "Credential Access"]
}
The checker:
import ipaddress
import json
import sys
from datetime import datetime
def load_roe(path):
roe = json.load(open(path))
roe["nets"] = [ipaddress.ip_network(n) for n in roe["in_scope"]]
roe["excluded_nets"] = [ipaddress.ip_network(n) for n in roe["excluded"]]
roe["start"] = datetime.fromisoformat(roe["window"]["start"])
roe["end"] = datetime.fromisoformat(roe["window"]["end"])
return roe
def check(action, roe):
ip = ipaddress.ip_address(action["target"])
when = datetime.fromisoformat(action["time"])
if any(ip in n for n in roe["excluded_nets"]):
return "EXCLUDED"
if not any(ip in n for n in roe["nets"]):
return "OUT OF SCOPE"
if not roe["start"] <= when <= roe["end"]:
return "OUTSIDE WINDOW"
if action["tactic"] not in roe["tactics"]:
return "TACTIC NOT PERMITTED"
return "OK"
def main(roe_path, log_path):
roe = load_roe(roe_path)
bad = 0
for line in open(log_path):
if not line.strip():
continue
a = json.loads(line)
verdict = check(a, roe)
bad += verdict != "OK"
print(f"{verdict:<20} {a['time']} {a['target']:<15} {a['tactic']}")
print(f"{bad} action(s) outside the rules of engagement")
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1], sys.argv[2]))
The action log is one JSON object per line, with a time, a target and an ATT&CK tactic name. Run it like this:
python3 roe_check.py roe.json actions.jsonl
On a six-line sample log it printed this and exited with status 1:
OK 2026-12-01T10:15 203.0.113.10 Reconnaissance
OK 2026-12-02T14:40 203.0.113.25 Credential Access
EXCLUDED 2026-12-03T11:20 203.0.113.202 Discovery
OUT OF SCOPE 2026-12-04T16:30 198.51.100.7 Reconnaissance
TACTIC NOT PERMITTED 2026-12-04T17:10 203.0.113.30 Impact
OUTSIDE WINDOW 2026-12-06T03:05 203.0.113.10 Discovery
4 action(s) outside the rules of engagement

It checks a log after the fact, so a real deployment should also run the check before an agent acts. It also trusts the log, so the log itself must come from your tooling, not from the agent’s own account of what it did.
Most failures in AI red team operations are process failures, not model failures. These are the ones I would watch for.
Long, messy operations remain the hard part of AI red team operations. The Incalmo baselines show it: long contexts, irrelevant commands and tasks implemented wrongly. AI also cannot walk into a building, read a room or judge when a phishing test has become unfair to staff.
AI red team operations do not replace the control team either. In TIBER-EU, a small group alone knows about the test and keeps it safe, and that role needs a person with authority. Our guide on AI vs human penetration testing covers where each side wins.
I am biased, it is my company, and I think XHack is the strongest answer because it combines three things most providers sell separately: an autonomous AI agent, a continuous security platform and human testers.
Here is what I can state from our public pages about AI red team operations.
One honest limit: if you are in DORA TLPT scope, check Article 27 and your supervisor’s rules before choosing any provider, because I am not claiming that any provider, XHack included, satisfies them for you. See our AI VAPT services, the unrestricted AI agent guide and the pricing page. Brutal honesty is kind of our thing.
AI red team operations are authorised, objective-based attack simulations that use AI to speed up planning, reconnaissance notes, logging and reporting. People still own the objectives, the rules of engagement and the decisions that carry risk. The goal is to test whether defenders detect and respond, not just to list vulnerabilities.
Not reliably. For AI red team operations, in the Carnegie Mellon and Anthropic study, leading LLM-assisted systems reached a critical asset in only 3 of 40 emulated networks, and 37 of 40 once a purpose-built abstraction layer was added. Even then the networks were emulated, so a human still has to own scope and stop conditions.
AI red team operations are legal when the work is authorised in writing and stays inside the agreed scope, and your AI provider’s rules allow it. Anthropic’s Red Team Access, for example, is for organisations and limits testing to systems they are authorised to test. This is not legal advice, so check your contract and local law.
A penetration test is scope-limited and finds as many vulnerabilities as possible in a defined target. An AI red team operation is objective-based, pursues a goal such as a critical asset by any authorised path, and tests detection and response. The two answer different questions, so many organisations run both.
DORA requires in-scope financial entities to run threat-led penetration testing at least every 3 years, on live production systems supporting critical or important functions. The competent authority can change the frequency, and Article 27 sets requirements for the testers. Check which entities are in scope with your regulator.
In AI red team operations, log every action with a timestamp, target and technique, so you can trace it against the rules of engagement and against the blue team’s detections. Generate the log from your tooling rather than the agent’s own summary, and keep it as evidence for the client and the debrief.
AI red team operations work best as an accelerator for planning, logging and reporting inside rules that humans wrote and enforce. The evidence says autonomy depends on the system around the model, regulators in finance already put responsibility on qualified testers, and the legal AI routes are organisation-only. Write the rules of engagement first, gate every action against them, verify every claim, and run the purple debrief. That is how you get the speed without the liability.
Categories
Related articles