XHack Logo
XHack
Products
Services
Compliance
Pricing
Resources
Company
Sign upLogin
XHack Logo
XHackOffensive Security

Certified offensive security team delivering penetration testing evidence written for your auditor.

OSCP+OSCPC-AI/MLPenCASA
support@xhack.io

24/7 SOC Operations

XHack Status
Under attack? Get help now
Services
  • VAPT Services
  • Red Teaming
  • SOC Services
  • Threat Intelligence
  • Incident Response
  • Managed Testing
Pricing
  • Platform Plans
  • Services Pricing
Compliance
  • SOC 2
  • PCI DSS
  • ISO 27001
  • GDPR
  • HIPAA
  • ISO 42001
  • AI Maturity Assessment
  • TX-RAMP
  • NBFC / SECP
  • All Frameworks
Products
  • Vulnerability Assessment
  • GitGuard
  • AI Probe
  • SOC Dashboard
  • AI Agent
  • Cloud Investigation
Comparison
  • XBOW vs XHack
  • Horizon3 vs XHack
  • Strix vs XHack
  • Pentera vs XHack
Resources
  • Platform Tour
  • All Features
  • Install the Agent
  • XHack AI
  • Documentation
  • Blog
  • Case Studies
  • Documents
  • FAQ
Company
  • About Us
  • Our Team
  • Certifications
  • Security and Trust
  • VAPT Explained
  • Contact

© 2026 XHACK (SMC-PRIVATE) LIMITED. All rights reserved.

Registered with the SECP, CUIN 0361945. Verify

Security & TrustVulnerability DisclosurePrivacy PolicyTerms of ServiceRefund Policy
Blog/Security

AI Red Team Operations: Where AI Fits in a Real Engagement and Where It Does Not

XHack

XHack

Author

November 19, 2026

19 min read

AI Red Team Operations: Where AI Fits in a Real Engagement and Where It Does Not

Table of contents

17

What AI Red Team Operations Actually Are

The Phases of AI Red Team Operations and Where AI Helps

What the Evidence Says About Autonomy in AI Red Team Operations

Threat-Led Red Teaming in Finance: DORA and TIBER-EU

Using AI Legally in a Red Team Operation

A Rules-of-Engagement Gate for AI Red Team Operations

Common Mistakes in AI Red Team Operations

Where AI Red Team Operations Still Fall Apart

How XHack Handles AI Red Team Operations

FAQ: AI Red Team Operations Questions Answered

What are AI red team operations?

Can AI run red team operations by itself?

Are AI red team operations legal?

How are AI red team operations different from a penetration test?

How often does DORA require red team testing?

What should I log during an AI red team operation?

The Bottom Line

By Salman Khan, OSCP+, Founder of XHack, SRT (Synack Red Team member)

Read this in 30 seconds: AI red team operations means using AI inside an authorised, objective-based attack simulation, and it helps most in planning, logging and reporting, not in running the whole operation unattended. The evidence says the tooling around the model matters more than the model, and the rules say a human must own the scope. I read the primary sources this week.

  • A red team operation is not a bigger pentest. NIST’s glossary describes a red team as people authorized and organized to emulate an adversary, and the goal is an objective such as reaching a critical asset, not a list of every bug.
  • Raw AI agents struggle with multi-host operations. In a Carnegie Mellon and Anthropic study on 40 emulated networks, leading LLM-assisted systems reached a critical asset in 3 of 40, but the same models with a purpose-built abstraction layer reached one in 37 of 40.
  • Regulators already set the rules in finance. DORA requires threat-led penetration testing at least every 3 years on live production systems, and Article 27 sets tester requirements that include accreditation or a code of conduct and professional indemnity insurance.
  • The legal AI route is organisation-only. Anthropic’s Red Team Access tier covers authorised red teaming, but it is not open to individuals, and real-time blocks remain on things like deploying ransomware.
  • A rules-of-engagement gate costs almost nothing in AI red team operations. Any action an AI agent takes should be checked against scope, window and permitted tactics, and I include a tested script that does it.

An AI agent can be fast at an operation and still be a liability if nobody decided what it was allowed to touch.

Search for AI red team operations and you will find two kinds of pages: tool lists that skip the rules, and vendor pages that skip the limits. This guide is the practical version. It covers what AI red team operations are, which phases AI genuinely speeds up, what the research shows about autonomy, how finance regulators treat threat-led testing, and how to keep an AI agent inside its authorisation during AI red team operations.

It contains no attack techniques, no payloads and no evasion advice. It is written for authorised engagements only. I run XHack, so I also show where we fit and where we do not.

AI red team operations animation: AI assists with threat scenarios, reconnaissance notes, logging and report drafting, while humans hold the objectives, rules of engagement, emergency stop and sign-off
Who holds what in AI red team operations: AI assists in the lower lane, humans keep the decisions that carry legal and business risk

What AI Red Team Operations Actually Are

A red team operation is an authorised attack simulation with a goal, and AI red team operations add AI to that process. NIST’s glossary, quoting CNSSI 4009, defines a red team as a group “authorized and organized to emulate a potential adversary’s attack or exploitation capabilities”. Its purpose is to show the impact of a successful attack and what works for defenders in a real environment.

That makes AI red team operations different from a penetration test. A pentest is scope-limited and tries to find as many vulnerabilities as it can in one target. A red team operation is objective-based: it tries to reach a defined goal, such as a specific database or an administrator account, by whatever authorised path exists, and it tests whether the defenders notice. Our guide on AI penetration testing covers the first job, and this one covers the second.

Two more definitions keep the rest of this article clean.

  • AI red teaming often means testing an AI system itself, such as a chatbot. Our AI red teaming playbook covers both meanings. Here the AI is a tool inside a conventional red team operation.
  • MITRE ATT&CK is the shared vocabulary. Its Enterprise matrix currently lists 15 tactics, from Reconnaissance to Impact, and the current release is v20.1, dated 13 October 2026.

The Phases of AI Red Team Operations and Where AI Helps

AI helps most where the work is reading, organising and writing, and humans must hold everything that carries risk. Here is each phase of AI red team operations, what AI can do in it and what a person has to own. It stays at concept level on purpose.

PhaseWhat happensWhere AI helpsWhat a human must hold
Objectives and scenarioAgree the goal and the threat to emulateSummarise public threat reports into candidate scenarios, map them to ATT&CKChoosing objectives that match business risk
Rules of engagementDocument scope, exclusions, windows, stop procedureDraft checklists, spot gaps and contradictionsSigning off, and the emergency stop
ReconnaissanceLearn the in-scope attack surfaceOrganise notes, deduplicate and correlate findings on approved assetsDeciding what counts as in scope
ExecutionPursue the objective inside the rulesPlan steps, run in-scope tooling under supervisionEvery boundary decision, anything touching people or safety systems
Observation and loggingRecord every action with a timestampKeep structured logs, flag actions outside the rulesReviewing the log, owning the evidence
Reporting and purple debriefAttack narrative, detection gaps, fixesDraft the narrative, build the timeline against blue team eventsJudging severity and what to recommend
AI red team operations phase table: six engagement phases with what AI can assist with and what a human must hold
AI red team operations by phase: where AI assists and what a person must always hold

Notice the pattern in AI red team operations. AI is strongest at the paperwork around the operation, which is where operators lose hours, and at planning steps for a person to approve. The riskiest phase, execution, is also where it needs the most supervision.

What the Evidence Says About Autonomy in AI Red Team Operations

The best study on this question says the surrounding system matters more than the model. In “Incalmo”, researchers at Carnegie Mellon University and Anthropic built MHBench, a benchmark of 40 emulated multi-host networks with 22 to 50 hosts each. Success meant acquiring at least one critical asset. The paper was first submitted in January 2025 and last revised in November 2025.

The results are stark.

  • Leading LLM-assisted systems such as PentestGPT, CyberSecEval3 and CAI, run with top models, succeeded in 3 of 40 environments.
  • Incalmo with Sonnet 4 succeeded in 37 of 40, in runs of 12 to 54 minutes costing at most $15 in LLM credits.
  • Baseline failures were messy. Between 47% and 90% of the best baseline’s commands were irrelevant to the attack chain, and 6% to 41% of its relevant tasks were implemented incorrectly.
  • Model size was not the lever. The authors report that the abstractions matter more, and that a smaller model such as Haiku 3.5 still succeeded in most environments with them.
Incalmo study animation: 40 emulated networks light up, 3 of 40 for leading LLM-assisted systems and 37 of 40 for Incalmo with an abstraction layer
Carnegie Mellon and Anthropic’s MHBench results: 3 of 40 networks for baselines against 37 of 40 with an abstraction layer

Three honest caveats apply. These were emulated networks, not production environments with real defenders. The authors call the technology dual-use and say they open-sourced the code and disclosed the results to LLM vendors. And one study is not the whole field.

Translation: if you evaluate a product for AI red team operations, ask about the orchestration, scope control and logging, because that is where the difference showed up. A better chat model alone is not the answer.

Threat-Led Red Teaming in Finance: DORA and TIBER-EU

If you work in EU finance, the rules for AI red team operations are already written, and AI use is a question about your testers. Two frameworks matter.

DORA (Regulation (EU) 2022/2554) requires in-scope financial entities to carry out advanced testing by means of threat-led penetration testing (TLPT) at least every 3 years, and the competent authority can raise or lower that frequency. Article 26 says each TLPT must cover several or all critical or important functions and be performed on live production systems. DORA applies from 17 January 2025.

Article 27 sets who may test. Testers must be of the highest suitability and reputability, have specific expertise in threat intelligence, penetration testing and red team testing, be certified by an accreditation body or adhere to formal codes of conduct, provide an independent assurance or audit report, and carry professional indemnity insurance that covers misconduct and negligence. Internal testers need authority approval, and an entity using them must contract external testers every three tests.

TIBER-EU, the ECB’s framework for threat intelligence-based ethical red-teaming, was published in May 2018 and updated in 2024 to align with the DORA technical standards. It defines five teams: the blue team, a threat intelligence provider, the red-team testers, a small control team that alone knows about the test, and the TIBER cyber team at the authority. It describes the outcome as not a pass or fail, and finishes with a test summary report and an attestation for mutual recognition.

Threat-led red teaming in finance: DORA TLPT every three years on live production systems, Article 27 tester requirements, and the five TIBER-EU teams
DORA and TIBER-EU in one view: how often, on what systems, who may test and who is on which team

I could not retrieve the 2025 technical standard on TLPT or the Bank of England’s CBEST page, so check those directly if they apply to you. This is not legal advice. What matters here: none of these rules bans AI red team operations, but they put responsibility on named, qualified testers, so a provider’s AI agent does not remove any of the Article 27 requirements.

Test it on your own target

The XHack agent finds, verifies and reports real vulnerabilities in web apps, APIs, hosts and mobile.

Start free for 7 days

Using AI Legally in a Red Team Operation

Legal AI red team operations rest on three things: written authorisation, the provider’s rules and logging you can show a client. The first is the law, and the second is the contract with your AI vendor.

On the vendor side, Anthropic’s Cyber Verification Program has a Red Team Access tier for authorised penetration testing and red teaming. According to Anthropic’s page, it is for organisations such as in-house red teams, government red teams and security and penetration testing firms, individuals are not eligible, and organisations can only perform adversarial testing against systems they are authorised to test. Real-time blocks remain on actions that could cause physical harm or mass disruption, with deploying ransomware, damaging physical systems and pen testing high-risk safety systems as examples.

Other conditions are worth reading before you apply: data retention is required for enrolment, applicants sit in Defense Access while Red Team review takes a few weeks, and the page was updated on 18 November 2026 to change plan availability. OpenAI’s Daybreak Red is similarly limited to approved business and enterprise organisations. See Anthropic’s Cyber Verification Program page for the current terms.

On the client side, the habits for AI red team operations are the same as any engagement:

  1. Get written permission with a signed scope and named contacts.
  2. Write the rules of engagement down, including exclusions, windows and the emergency stop.
  3. Log everything with timestamps so any action can be traced.
  4. Keep humans in the loop for anything that crosses a boundary.

A Rules-of-Engagement Gate for AI Red Team Operations

The cheapest safety control in AI red team operations is a script that checks every action an agent took against the rules. This one reads a rules-of-engagement file and an action log, and flags anything out of scope, excluded, outside the window or using a tactic the client did not permit. It uses only the standard library, and I ran it before publishing.

The rules file uses a reserved documentation network, so nothing in it points at a real system:

{
  "in_scope": ["203.0.113.0/24"],
  "excluded": ["203.0.113.200/29"],
  "window": {"start": "2026-12-01T09:00", "end": "2026-12-05T18:00"},
  "tactics": ["Reconnaissance", "Discovery", "Credential Access"]
}

The checker:

import ipaddress
import json
import sys
from datetime import datetime


def load_roe(path):
    roe = json.load(open(path))
    roe["nets"] = [ipaddress.ip_network(n) for n in roe["in_scope"]]
    roe["excluded_nets"] = [ipaddress.ip_network(n) for n in roe["excluded"]]
    roe["start"] = datetime.fromisoformat(roe["window"]["start"])
    roe["end"] = datetime.fromisoformat(roe["window"]["end"])
    return roe


def check(action, roe):
    ip = ipaddress.ip_address(action["target"])
    when = datetime.fromisoformat(action["time"])
    if any(ip in n for n in roe["excluded_nets"]):
        return "EXCLUDED"
    if not any(ip in n for n in roe["nets"]):
        return "OUT OF SCOPE"
    if not roe["start"] <= when <= roe["end"]:
        return "OUTSIDE WINDOW"
    if action["tactic"] not in roe["tactics"]:
        return "TACTIC NOT PERMITTED"
    return "OK"


def main(roe_path, log_path):
    roe = load_roe(roe_path)
    bad = 0
    for line in open(log_path):
        if not line.strip():
            continue
        a = json.loads(line)
        verdict = check(a, roe)
        bad += verdict != "OK"
        print(f"{verdict:<20} {a['time']}  {a['target']:<15} {a['tactic']}")
    print(f"{bad} action(s) outside the rules of engagement")
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

The action log is one JSON object per line, with a time, a target and an ATT&CK tactic name. Run it like this:

python3 roe_check.py roe.json actions.jsonl

On a six-line sample log it printed this and exited with status 1:

OK                   2026-12-01T10:15  203.0.113.10    Reconnaissance
OK                   2026-12-02T14:40  203.0.113.25    Credential Access
EXCLUDED             2026-12-03T11:20  203.0.113.202   Discovery
OUT OF SCOPE         2026-12-04T16:30  198.51.100.7    Reconnaissance
TACTIC NOT PERMITTED 2026-12-04T17:10  203.0.113.30    Impact
OUTSIDE WINDOW       2026-12-06T03:05  203.0.113.10    Discovery
4 action(s) outside the rules of engagement
Terminal animation of roe_check.py: six logged agent actions are checked against the rules of engagement and four are flagged as excluded, out of scope, tactic not permitted or outside the window
The roe_check.py output from the article: two actions pass and four are flagged against the rules of engagement

It checks a log after the fact, so a real deployment should also run the check before an agent acts. It also trusts the log, so the log itself must come from your tooling, not from the agent’s own account of what it did.

Common Mistakes in AI Red Team Operations

Most failures in AI red team operations are process failures, not model failures. These are the ones I would watch for.

  • No rules of engagement before the agent starts. An agent with a broad instruction and no written limits will go where the task leads.
  • Treating AI output as findings. A model can state something false with confidence, so every claim needs evidence from the real system.
  • No objective. Without a defined goal, AI red team operations turn into a noisy pentest with a red team label on it.
  • Skipping the purple debrief. The value is in showing defenders what the activity looked like in their tools, and a report alone does not do that.
  • Forgetting the people side. Social engineering involves real employees, so scope, consent and stop conditions need human decisions, not an agent’s judgment.
  • Ignoring data handling. Anthropic requires data retention for its verification program, so know where your logs and client data go.

Where AI Red Team Operations Still Fall Apart

Long, messy operations remain the hard part of AI red team operations. The Incalmo baselines show it: long contexts, irrelevant commands and tasks implemented wrongly. AI also cannot walk into a building, read a room or judge when a phishing test has become unfair to staff.

AI red team operations do not replace the control team either. In TIBER-EU, a small group alone knows about the test and keeps it safe, and that role needs a person with authority. Our guide on AI vs human penetration testing covers where each side wins.

How XHack Handles AI Red Team Operations

I am biased, it is my company, and I think XHack is the strongest answer because it combines three things most providers sell separately: an autonomous AI agent, a continuous security platform and human testers.

Here is what I can state from our public pages about AI red team operations.

  • A real red team service. Our Red Team Operations service is objective-based, starts with documented rules of engagement, covers permitted techniques, excluded systems, testing windows, escalation and emergency stop, and offers a purple team option where we work alongside your defenders.
  • A report built for defenders. It includes a full attack narrative, a timeline correlating red team actions with blue team detection events, detection gaps and a debrief. Engagement data is encrypted and artifacts are destroyed after the report is delivered.
  • Scoped, not priced off a menu. Red teaming is scoped individually, so book a scoping call. Managed pentesting by OSCP-certified researchers starts at $2,500.
  • AI for planning in AI red team operations. Every individual plan lists red team operation planning, and the agent runs up to 12 sub-agents by default with three browsers. XHack does not store your agent chats.
  • Verified, not anonymous. The agent finds vulnerabilities from the start, and exploitation unlocks once your identity is verified, with a 7-day trial and one per person. Plans are $20, $49 and $150 a month, and company plans start at $560.

One honest limit: if you are in DORA TLPT scope, check Article 27 and your supervisor’s rules before choosing any provider, because I am not claiming that any provider, XHack included, satisfies them for you. See our AI VAPT services, the unrestricted AI agent guide and the pricing page. Brutal honesty is kind of our thing.

FAQ: AI Red Team Operations Questions Answered

What are AI red team operations?

AI red team operations are authorised, objective-based attack simulations that use AI to speed up planning, reconnaissance notes, logging and reporting. People still own the objectives, the rules of engagement and the decisions that carry risk. The goal is to test whether defenders detect and respond, not just to list vulnerabilities.

Can AI run red team operations by itself?

Not reliably. For AI red team operations, in the Carnegie Mellon and Anthropic study, leading LLM-assisted systems reached a critical asset in only 3 of 40 emulated networks, and 37 of 40 once a purpose-built abstraction layer was added. Even then the networks were emulated, so a human still has to own scope and stop conditions.

Are AI red team operations legal?

AI red team operations are legal when the work is authorised in writing and stays inside the agreed scope, and your AI provider’s rules allow it. Anthropic’s Red Team Access, for example, is for organisations and limits testing to systems they are authorised to test. This is not legal advice, so check your contract and local law.

How are AI red team operations different from a penetration test?

A penetration test is scope-limited and finds as many vulnerabilities as possible in a defined target. An AI red team operation is objective-based, pursues a goal such as a critical asset by any authorised path, and tests detection and response. The two answer different questions, so many organisations run both.

How often does DORA require red team testing?

DORA requires in-scope financial entities to run threat-led penetration testing at least every 3 years, on live production systems supporting critical or important functions. The competent authority can change the frequency, and Article 27 sets requirements for the testers. Check which entities are in scope with your regulator.

What should I log during an AI red team operation?

In AI red team operations, log every action with a timestamp, target and technique, so you can trace it against the rules of engagement and against the blue team’s detections. Generate the log from your tooling rather than the agent’s own summary, and keep it as evidence for the client and the debrief.

The Bottom Line

AI red team operations work best as an accelerator for planning, logging and reporting inside rules that humans wrote and enforce. The evidence says autonomy depends on the system around the model, regulators in finance already put responsibility on qualified testers, and the legal AI routes are organisation-only. Write the rules of engagement first, gate every action against them, verify every claim, and run the purple debrief. That is how you get the speed without the liability.

Put this into practice

Point the XHack agent at a target you are authorised to test. It hunts, proves and writes the report.

Start free for 7 days

Categories

Security

Next

Jailbroken AI for Security Research: What It Costs You and the Legal Route

On this page

What AI Red Team Operations Actually Are

The Phases of AI Red Team Operations and Where AI Helps

What the Evidence Says About Autonomy in AI Red Team Operations

Threat-Led Red Teaming in Finance: DORA and TIBER-EU

Using AI Legally in a Red Team Operation

A Rules-of-Engagement Gate for AI Red Team Operations

Common Mistakes in AI Red Team Operations

Where AI Red Team Operations Still Fall Apart

How XHack Handles AI Red Team Operations

FAQ: AI Red Team Operations Questions Answered

What are AI red team operations?

Can AI run red team operations by itself?

Are AI red team operations legal?

How are AI red team operations different from a penetration test?

How often does DORA require red team testing?

What should I log during an AI red team operation?

The Bottom Line

Related articles

Continue reading

Jailbroken AI for Security Research: What It Costs You and the Legal Route

Security

Jailbroken AI for Security Research: What It Costs You and the Legal Route

Jailbroken AI for security research: the five costs, what research shows on defences and the verified route to fewer ref...

Read article
Offensive AI: What the 2026 Evidence Shows and How to Use It Legally

Security

Offensive AI: What the 2026 Evidence Shows and How to Use It Legally

Offensive AI explained with 2026 evidence: what attackers actually do, where it still fails, how to use it lawfully and ...

Read article
NodeZero vs Pentera: Pricing, Scope and Who Each One Fits in 2026

Security

NodeZero vs Pentera: Pricing, Scope and Who Each One Fits in 2026

NodeZero vs Pentera compared from vendor pages and AWS listings: published per-asset prices, scope, certifications, revi...

Read article