AI AppSec
Advanced
AML.T0070 AML.T0051

RAG Security Testing

Retrieval-augmented generation systems fail differently from ordinary chatbots. The model is only one part of the target: documents, connectors, chunking, embedding search, access control, citations, and output rendering all become testable attack surface.

Authorized systems only

Test only systems, repositories, document stores, and connectors explicitly listed in the rules of engagement. RAG testing often touches sensitive internal knowledge, so evidence should minimize copied source content and preserve only the fields needed to prove impact.

What you'll be able to do

  • Prove whether the system treats retrieved content as data or as instruction
  • Confirm retrieval cannot cross customer, department, project, or document-level access boundaries
  • Compare cited sources against the actual answer and flag fabricated or laundered citations
  • Test whether secrets, HR records, customer data, or private tickets leak through summaries or multi-turn narrowing
  • Capture report-ready evidence: user context, retrieval trace, output proof, and control result

RAG abuse path - seed fixture, retrieval, answer influence, and captured evidence

The RAG abuse path from an approved poisoned or stale source through chunk retrieval and answer influence to captured evidence, with a dashed reproduce-after-fix edge returning from evidence to retrieval.

Scroll sideways to see the whole diagram

View full size

Trace every stage of the pipeline: ingestion, where files, crawlers, tickets, wiki pages, email, and SaaS connectors enter the index; retrieval, where embeddings, chunk ranking, filters, tenant boundaries, and ACL joins decide what context is returned; generation, where prompt assembly, citations, tool calls, refusals, and fallback behavior shape the answer; and evidence, where trace IDs, retrieved chunks, user role, prompt, output, and blocked controls prove what happened.

Core Test Areas

Document and chunk poisoning

Place benign test documents in approved locations that contain conflicting instructions, hidden text, metadata instructions, ambiguous citations, and malicious support links. Verify whether the system treats retrieved content as data or instruction.

Tenant and ACL boundaries

Create users with different roles and confirm retrieval cannot cross customer, department, project, or document-level access boundaries. Test deleted documents and stale index entries after permission changes.

Citation integrity

Compare cited sources against the actual answer. Flag fabricated citations, unsupported claims, citation laundering, hidden-source reliance, and answers that cite low-trust material over authoritative sources.

Sensitive data leakage

Test whether secrets, HR records, customer data, private tickets, API responses, or embeddings can be inferred through summaries, autocomplete, broad questions, or multi-turn narrowing.

Setup and Prerequisites

  • Fixture pack: the fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
  • Runtime: Docker + Docker Compose, or Python 3.11+ to run the app directly with python -m app.server.
  • Working directory: run every command from inside the pack directory; the lab client is python scripts/lab.py.
  • No model download: no model download and no provider key are needed — the fixture is offline and deterministic.
lab-start.sh 
docker compose up -d
python scripts/wait_for_health.py
python scripts/lab.py health

Without Docker, run python -m app.server directly (Python 3.11.9 is the verified version) and stop it with Ctrl-C when finished. The app was run end-to-end under Python 3.11.9 with all lab assertions passing; the Docker stack itself is authoring-derived and unverified.

The model in this fixture is a deterministic stub. The matrix shows what you actually get on each path.

Path Model Expected latency Token budget Privacy
Local CPU (default fixture) stub-embed-v1 (no weights) < 50 ms/request 0 tokens nothing leaves the host
Local GPU (not shipped) e.g. an Ollama-served ~30B coder ~1–5 s per stream operator-set prompts stay local
Hosted API (optional, off by default) operator-configured 0.5–3 s billed per token prompts leave the host
CI stub-embed-v1 < 50 ms/request 0 tokens in-runner only

Steps

  1. Confirm both tenants loaded.

    tenants.sh 
    python scripts/lab.py tenants

    Expected output

    The JSON lists tenant-a, tenant-b and support, each with a fixed non-secret api_key.

  2. Retrieve as tenant-a and read the trace.

    retrieve-tenant-a.sh 
    python scripts/lab.py retrieve --tenant tenant-a --query "quarterly canary token"

    Expected output

    Every returned chunk has tenant: "tenant-a", a chunk_id and a score; doc-canary-a is present with score 1.25.

  3. Turn the tenant ACL off and ask for another tenant's canary.

    retrieve-acl-off.sh 
    python scripts/lab.py retrieve --tenant tenant-a --query "globex tenant-b canary token" --acl-off

    Expected output

    The response contains a chunk whose tenant is tenant-b — the cross-tenant leak is reproduced.

  4. Ask with the weakened prompt so the poisoned document is followed.

    ask-weak-prompt.sh 
    python scripts/lab.py ask --tenant tenant-a --query "refund policy" --prompt weak

    Expected output

    injection_marker_present is true (the answer contains CANARY-INJECTED-POISON-A) and citation_laundering is true (the injected document cites doc-policy-a).

  5. Ask the same question with the shipped default prompt.

    ask-default-prompt.sh 
    python scripts/lab.py ask --tenant tenant-a --query "refund policy"

    Expected output

    injection_marker_present is false and citation_laundering is false — the default guard ignores the injected chunk.

Evidence Checklist

  • User context: role, tenant, group, and active permissions.
  • Source record: document ID, connector, timestamp, and classification.
  • Retrieval trace: chunk IDs, score, filters, and denied candidates where available.
  • Prompt assembly: system prompt version, retrieved context, and tool schema version.
  • Output proof: answer, citations, rendered view, and any downstream action.
  • Control result: blocked, allowed, warned, logged, escalated, or silently failed.

Verify It Worked

Check Expected Where to look
Retrieval trace carries chunk IDs, scores, and the tenant field Every chunk is tagged tenant-a with a chunk_id and a score, including doc-canary-a at score 1.25 The retrieve JSON response for tenant-a
Tenant boundary holds with the ACL on No chunk from another tenant is returned The default retrieve response, contrasted with the --acl-off response
Poisoned document is followed only under the weakened prompt injection_marker_present and citation_laundering are true with the weak prompt and false with the default prompt The ask answers for the same refund-policy query
Evidence stays minimal Role, tenant, document ID, chunk IDs, prompt version, output, and control result, with no extra source content The evidence bundle you retain

Best defenses to verify

Strong RAG systems separate instructions from retrieved data, apply ACL filters before retrieval, store retrieval traces, sanitize ingested content, show citations with confidence, and regression-test fixes whenever prompts, chunking, models, or connectors change.

Practice Lab

Runnable fixture

RAG retrieval, tenant boundaries and poisoned documents

Prove the toy support bot keeps retrieval inside a tenant boundary and that a poisoned document can change the answer and launder its citation.

Fixture path: /labs/offensive-ai

Runnable

Prerequisites

  • The fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
  • Docker + Docker Compose, or Python 3.11+ to run the app directly with `python -m app.server`.
  • Run every command from inside the pack directory; the lab client is `python scripts/lab.py`.
  • No model download and no provider key are needed — the fixture is offline and deterministic.

Steps

  1. 1

    Start the fixture stack.

    step-01.sh 
    docker compose up -d

    Expected output

    The `toyapp` container starts and publishes one host port, 127.0.0.1:8080.
  2. 2

    Wait until the app is healthy.

    step-02.sh 
    python scripts/wait_for_health.py

    Expected output

    Prints a healthy body whose `marker` is `OAI-TOYAPP-HEALTHY`.
  3. 3

    Confirm both tenants loaded.

    step-03.sh 
    python scripts/lab.py tenants

    Expected output

    The JSON lists `tenant-a`, `tenant-b` and `support`, each with a fixed non-secret `api_key`.
  4. 4

    Retrieve as tenant-a and read the trace.

    step-04.sh 
    python scripts/lab.py retrieve --tenant tenant-a --query "quarterly canary token"

    Expected output

    Every returned chunk has `tenant: "tenant-a"`, a `chunk_id` and a `score`; `doc-canary-a` is present with score 1.25.
  5. 5

    Turn the tenant ACL off and ask for another tenant's canary.

    step-05.sh 
    python scripts/lab.py retrieve --tenant tenant-a --query "globex tenant-b canary token" --acl-off

    Expected output

    The response contains a chunk whose `tenant` is `tenant-b` — the cross-tenant leak is reproduced.
  6. 6

    Ask with the weakened prompt so the poisoned document is followed.

    step-06.sh 
    python scripts/lab.py ask --tenant tenant-a --query "refund policy" --prompt weak

    Expected output

    `injection_marker_present` is true (the answer contains `CANARY-INJECTED-POISON-A`) and `citation_laundering` is true (the injected document cites `doc-policy-a`).
  7. 7

    Ask the same question with the shipped default prompt.

    step-07.sh 
    python scripts/lab.py ask --tenant tenant-a --query "refund policy"

    Expected output

    `injection_marker_present` is false and `citation_laundering` is false — the default guard ignores the injected chunk.

Teardown

teardown.sh 
docker compose down -v
If you ran the app directly, stop it with Ctrl-C (the fixture holds no persistent state).

Evidence to capture

  • The retrieval trace showing chunk IDs, scores and the `tenant` field.
  • The `--acl-off` response containing a tenant-b chunk.
  • The weak-prompt answer with `injection_marker_present` and `citation_laundering` both true.
  • The safe default answer for the same query, for contrast.

No Docker? Use Lakera Gandalf — A Lakera red-teaming game that practises the same instruction/data confusion without running the fixture.

Evidence and Findings

Finding Proof Control
Indirect Prompt Injection A retrieved document can override assistant behavior or redirect users because untrusted content is included without instruction/data separation. Separate instructions from retrieved data.
Cross-Tenant Retrieval A user can retrieve or infer content outside their authorization boundary through broad questions, stale index entries, or missing ACL filters. Apply ACL filters before retrieval.
Citation Laundering The answer cites an approved source while relying on untrusted or unrelated retrieved chunks, making unsupported advice look authoritative. Show citations with confidence.

Common Mistakes

Treating retrieved content as instructions

Place benign test documents in approved locations that contain conflicting instructions, hidden text, metadata instructions, ambiguous citations, and malicious support links. Verify whether the system treats retrieved content as data or instruction.

Trusting the index after permission changes

Create users with different roles and confirm retrieval cannot cross customer, department, project, or document-level access boundaries. Test deleted documents and stale index entries after permission changes.

Accepting citations at face value

Compare cited sources against the actual answer. Flag fabricated citations, unsupported claims, citation laundering, hidden-source reliance, and answers that cite low-trust material over authoritative sources.

Copying too much source content into evidence

RAG testing often touches sensitive internal knowledge, so evidence should minimize copied source content and preserve only the fields needed to prove impact.

Related Offensive AI Guides

AI App Pentest Methodology

End-to-end testing flow for AI-enabled applications.

AI Attack & Defense

Prompt injection, jailbreaks, and model-facing controls.

Agent Permission Model

Tool, memory, approval, and sandbox boundaries.

AI Incident Response

Detect, triage, and respond to retrieval poisoning you surface here.

AI Supply Chain

Model, dataset, registry, and artifact supply-chain risks.