AI Probe

AI Probe

This is the operator's playbook for the XHack AI Probe. Follow it from cold start to your first vulnerability finding. Every label and button below comes from the actual UI.

Step 1 — Register your AI target

Click AI Probe in the main sidebar, then click Targets in the AI Probe sub-navigation. The Targets page has a help section at the top with worked examples for OpenAI, Anthropic, DeepSeek, and custom RAG/chatbot APIs — read through it once to understand the field shapes.

You have two ways to add a target. The fast way is the AI-assisted form.

Fast way: AI-assisted setup

1. Click Add Target.

2. Click Use AI Assistant at the top of the modal.

3. Paste a description of your endpoint into the textarea, e.g.:

   "OpenAI GPT-4 chat completions endpoint at api.openai.com/v1/chat/completions, POST, with {{payload}} as the user message. Auth via Bearer token in Authorization header."

4. Click Generate Config. The platform fills in the body template, response path, and headers automatically.

5. Review the generated fields and click Create Target.

Manual way

1. Click Add Target.
2. Fill in:
   • Name — friendly label like "Production GPT-4"
   • Endpoint URL — full URL, e.g. https://api.openai.com/v1/chat/completions
   • Method — POST, GET, or PUT
   • Timeout (s) — 1 to 60, default 30
   • Body Template — the request body with {{payload}} placeholder for single-turn or {{messages}} for multi-turn. For OpenAI:

     {"model":"gpt-4","messages":[{"role":"user","content":"{{payload}}"}]}
     

   • Response JSONPath — dot-notation path to the model's text response, e.g. choices.0.message.content
   • Headers (JSON) — auth headers as JSON, e.g. {"Authorization":"Bearer sk-..."}
3. Click Create Target.

Headers are encrypted at rest with Fernet and never returned in API responses — even superadmins cannot read them after save. If you need to update them, paste the new value into the same field; leaving it empty keeps the existing value.

Step 2 — Test the connection

Before you run a scan, verify the platform can actually talk to your endpoint.

1. In the targets table, find your new target.
2. Click the Test button on the row.
3. The platform sends a benign greeting and tries to extract the response using your JSONPath.
4. The Last Test column shows a green OK chip on success or a red Fail chip with the error reason.

If the test fails, the most common causes are: wrong response path, missing auth header, expired API key, or {{payload}} placeholder not in the body template.

Step 3 — Start your first scan

Click New Scan in the AI Probe header (or + New Scan from the targets page).

1. Pick the target you just created.
2. Choose a Scan Mode:
   • Quick — about 20 payloads, 2-5 minutes, good for sanity checks
   • Standard — about 50 payloads, 10-15 minutes, the default for most cases
   • Comprehensive — 150+ payloads, 30-60 minutes, for production audits and reports
3. Optionally narrow the Categories to a subset (e.g. only LLM01 prompt injection + LLM07 system prompt leak). Leave empty to test all categories.
4. Click Start Scan.

The scan kicks off immediately. You'll be redirected to the scan detail page.

Step 4 — Watch the scan and read findings

The scan detail page streams findings live. The header shows progress, current category being tested, completed payload count, and estimated time remaining. You can leave the page — the scan continues in the background.

The findings table has three tabs:

• Vulnerable — the target failed the test
• Safe — the target correctly refused or handled the prompt
• Other — inconclusive results or errors

Each row shows: severity, category, payload name, attack mode (single-turn or multi-turn), and response time. Click any row to see:

• The prompt sent (or first prompt for multi-turn)
• The response received
• The AI judge reasoning — why the judge classified it the way it did
• A score between 0.0 (completely vulnerable) and 1.0 (perfectly safe)

The overall security score (0-100) and grade (A-F) appear at the top of the page when the scan finishes.

Step 5 — Escalate a finding

When the judge marks a finding as Vulnerable, you usually want to confirm it manually and try variants. That's what Escalate is for.

1. Open a vulnerable finding.
2. Click the Escalate button.
3. The Escalation page opens with the original prompt and response at the top, and a chat input at the bottom.
4. Type a follow-up probe in the input. The platform sends it to the target and shows the response inline.
5. Continue the conversation as long as you need. Every turn is recorded.
6. To get an AI-generated bypass payload variant, click Generate Bypass (rate-limited to 20/hour). The platform writes a fresh payload designed to break this specific defense — review it before sending.

Step 6 — Retest after fixes

After your team patches a vulnerability, you can retest just that finding instead of running the whole scan again.

1. Open the finding.
2. Click Retest.
3. The platform re-runs the same payload against the target and updates the finding status.

If you've fixed many findings at once, run a full scan instead — the retest is best for confirming a single fix.

Step 7 — Export the report

When you're done:

1. Click Export in the scan detail header.
2. Pick PDF or HTML.
3. Optionally filter by severity.
4. Click Generate and download.

The report includes the score and grade, severity breakdown, category breakdown, top 10 findings, and the AI judge reasoning for each.

Tips and best practices

• Always run a Quick scan first on a new target to confirm the connection and shape of responses before burning tokens on Comprehensive.
• Use the Test button before every Save when editing a target. Saving a broken target wastes scan time.
• Pin the API key out of band. Headers are write-only after save, so make sure your team has the key in a secrets manager somewhere.
• Don't escalate every vulnerable finding. The judge is good. Escalate only when severity is borderline or when you need a proof-of-concept for a stakeholder.
• Schedule re-scans of production LLM endpoints monthly. Models drift, prompts get tweaked, defenses regress.
• Set token quota alerts. Each scan burns AI tokens both on the target side (your bill) and on the platform AI judge side (your XHack quota). Watch usage in Settings → Billing.

Privacy

Targets, scans, findings, and conversation history are isolated per tenant. Headers are Fernet-encrypted at rest and never returned in API responses. Multi-turn conversation transcripts are visible to scan owners and platform superadmins only — not to other tenants.

Related

• Platform Overview
• Asset Management
• Settings & Administration