AI Application Pentest Methodology

Treat AI features like distributed systems with probabilistic decision points. The test target is not only the model: it is the prompt chain, retrieval layer, permission model, gateway, observability stack, and the business process that trusts the output.

Assessment Phases

AI application pentest methodology - the five assessment phases, with retest feeding back into abuse-case design

Scroll sideways to see the whole diagram

View full size View full size

Minimum Test Matrix

Client Deliverables

Setup and Prerequisites

• Scoped plan: create a scoped test plan in the planner above, then refine it against the client's architecture and rules of engagement.
• Fixture pack: the fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/) for controlled execution.
• Runtime: Docker + Docker Compose, or Python 3.11+ to run the app directly with python -m app.server.
• Working directory: run every command from inside the pack directory; the lab client is python scripts/lab.py.

docker compose up -d
python scripts/wait_for_health.py

Steps

1. Document AI features, model/provider, data classes, allowed test types, prohibited actions, and evidence handling in an AI RoE addendum.

   roe-addendum.md markdown

   # AI RoE addendum — fill every heading before testing.
   # Allowed models, providers, test classes, tool-use boundaries,
   # evidence restrictions, and emergency stop conditions.
   ## Allowed models
   ## Allowed providers
   ## Allowed test classes
   ## Prohibited actions
   ## Tool-use boundaries
   ## Evidence restrictions
   ## Emergency stop conditions

   Expected output

   Allowed models, providers, test classes, tool-use boundaries, evidence restrictions, and emergency stop conditions are all named before testing starts.

2. Map users, prompts, retrieval, memory, tools, model gateways, output sinks, logging, and approval gates onto the trust boundaries.

   attack-surface-map.md markdown

   # AI attack-surface map — trust boundaries for each area.
   ## Prompts
   ## Retrieval
   ## Tools
   ## Memory
   ## Gateways
   ## Human approval
   ## Logs
   ## Output consumers

   Expected output

   Trust boundaries exist for prompts, retrieval, tools, memory, gateways, human approval, logs, and output consumers.

3. Convert OWASP LLM, MITRE ATLAS, business flows, and tool permissions into testable hypotheses.

   abuse-case.md markdown

   # Abuse case — one testable hypothesis per file.
   surface: prompt-layer | retrieval | tools-agents | model-gateway | operations
   control: the control this case should trip
   expected: the observable result that proves impact
   evidence: prompt, context, response, refusal or bypass trace

   Expected output

   Each hypothesis names its surface, the control it should trip, the expected observable result, and the evidence to collect.

4. Run tests in low-risk order — prompt conflicts, retrieval boundaries, tool abuse, tenant isolation, output handling, and monitoring — starting with a fixture health check.

   $ ~/controlled-execution.sh bash

   python scripts/lab.py health

   Expected output

   The fixture answers healthy with marker OAI-TOYAPP-HEALTHY; higher-risk phases only start after the low-risk ones pass.

5. Preserve prompts, traces, screenshots, logs, control behavior, business impact, and fix validation criteria in a regression pack.

   regression-pack.json json

   {
     "prompts": ["reproducible prompt, pinned to a prompt version"],
     "fixtures": ["fixture name and hash"],
     "expected_outcomes": ["expected observable result per prompt"]
   }

   Expected output

   A small set of reproducible prompts, fixtures, and expected outcomes that validate fixes after model or prompt changes.

Verify It Worked

Evidence and Findings

Common Mistakes