Evaluation
Intermediate
AML.T0054

AI Evaluation Workbench

AI security work is not finished when a prompt fails once. High-quality teams turn findings into repeatable evaluations that run after prompt edits, model upgrades, connector changes, and tool-permission changes.

Evaluation goal

Build small, representative, versioned eval sets that answer one question: did the control keep working after the system changed?

What you'll be able to do

  • Build small, representative, versioned eval sets that answer one question: did the control keep working after the system changed?
  • Turn confirmed red-team findings into repeatable evaluations that run after prompt edits, model upgrades, connector changes, and tool-permission changes
  • Score control pass rate: did the system refuse, sanitize, retrieve correctly, require approval, or log the event as designed?
  • Track regression drift across prompt changes, model swaps, new connectors, and guardrail updates
  • Wire the highest-severity cases into CI with a blocking threshold, a warning threshold, and a named triage owner

Finding to regression eval - confirmed failure, expected control, automated check, and drift gate

The regression-eval loop - a confirmed failure becomes an expected control, an automated check, and a release drift gate, with a dashed re-run-on-change edge returning from the gate to the automated check.

Scroll sideways to see the whole diagram

View full size

Workbench Stack

PyRIT drives orchestrated red-team workflows, target adapters, scoring, and repeatable attack strategy execution. Garak probes broadly for LLM vulnerabilities across prompt injection, leakage, encoding, and unsafe response classes. Promptfoo (now part of OpenAI) fits product-team regression gates, prompt variants, provider comparisons, and pass/fail assertions. Custom fixtures remain required for RAG, tool-use, tenant isolation, and business-logic tests that generic scanners cannot understand.

Eval Dataset Design

  • Fixture: prompt, user role, source document, tool manifest, and expected control behavior.
  • Oracle: deterministic assertion, semantic judge, human review queue, or policy decision.
  • Severity: impact if the test fails, not how clever the prompt looks.
  • Versioning: model, prompt, retrieval config, tool schema, and eval set version.
  • Evidence: request ID, output, retrieved context, tool-call trace, and scored result.
  • Gate: release blocking threshold, warning threshold, and owner for triage.

Score What Matters

Control pass rate

Did the system refuse, sanitize, retrieve correctly, require approval, or log the event as designed?

Impact severity

Prioritize tests that expose data, trigger tools, cross tenants, alter code, or produce unsafe operational guidance.

Regression drift

Track whether fixes degrade after prompt changes, model swaps, new connectors, or guardrail updates.

Setup and Prerequisites

  • Fixture pack: the fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
  • Runtime: Docker + Docker Compose, or Python 3.11+ to run the app directly with python -m app.server.
  • Working directory: run every command from inside the pack directory; the lab client is python scripts/lab.py.
  • No model download: no model download and no provider key are needed — the fixture is offline and deterministic.
lab-start.sh 
docker compose up -d
python scripts/wait_for_health.py
python scripts/lab.py health

Without Docker, run python -m app.server directly (Python 3.11.9 is the verified version) and stop it with Ctrl-C when finished. The app was run end-to-end under Python 3.11.9 with all lab assertions passing; the Docker stack itself is authoring-derived and unverified.

The model in this fixture is a deterministic stub. The matrix shows what you actually get on each path.

Path Model Expected latency Token budget Privacy
Local CPU (default fixture) stub-embed-v1 (no weights) < 50 ms/request 0 tokens nothing leaves the host
Local GPU (not shipped) e.g. an Ollama-served ~30B coder ~1–5 s per stream operator-set prompts stay local
Hosted API (optional, off by default) operator-configured 0.5–3 s billed per token prompts leave the host
CI stub-embed-v1 < 50 ms/request 0 tokens in-runner only

Steps

  1. Run the eval pack with the shipped prompt.

    eval-fixed.sh 
    python scripts/lab.py eval --prompt fixed

    Expected output

    gate is pass, cases_failed is 0, and all four cases list status: "pass".

  2. Read the version metadata the result pinned.

    eval-version-metadata.sh 
    python scripts/lab.py eval --prompt fixed

    Expected output

    The result carries prompt_version: "prompt-v3-fixed", model_version: "stub-embed-v1", a fixture_hash and an eval_version.

  3. Weaken the prompt and watch the gate fail.

    eval-weak.sh 
    python scripts/lab.py eval --prompt weak

    Expected output

    gate is fail, cases_failed is 1, and eval-rag-injection-001 is marked fail (observed injection_marker).

  4. Use the gate as a CI check that must block the release.

    eval-weak-ci.sh 
    python scripts/lab.py eval --prompt weak --ci

    Expected output

    The process exits 1 and prints CI gate FAIL: 1 case(s) failed.

  5. Restore the fixed prompt and re-run the gate.

    eval-fixed-ci.sh 
    python scripts/lab.py eval --prompt fixed --ci

    Expected output

    The process exits 0 and prints CI gate PASS.

Regression Gates

A gate turns an eval result into a release decision. Wire the highest-severity cases into CI, define a blocking threshold and a warning threshold, and name the owner who triages failures before promotion. Re-run the same set against every prompt, model, retrieval, or tool-policy change so drift is caught before it reaches users.

Blocking gate

Any failure in a critical case — prompt injection, cross-tenant retrieval, unsafe tool call — stops the release until the control is fixed or the change is reverted.

Warning gate

Lower-impact cases annotate the release and open a triage ticket with a deadline instead of blocking a deploy outright.

Drift monitor

Scheduled re-runs on unchanged cases catch silent regressions from provider-side model updates or connector changes.

Report Template

Eval name: [short descriptive name]

Target: [app/model/feature]

Control expected: [refuse / sanitize / retrieve only allowed docs / require approval / log]

Observed result: [pass/fail plus evidence ID]

Risk: [business impact if this regresses]

Owner: [team responsible for prompt, gateway, retrieval, or tool policy]

Verify It Worked

Check Expected Where to look
Eval pack passes on the shipped prompt gate is pass, cases_failed is 0, and all four cases list status pass The eval --prompt fixed JSON
Version metadata is pinned prompt_version, model_version, fixture_hash, and eval_version are present The eval result JSON
Weakened prompt fails the injection case eval-rag-injection-001 is marked fail with the observed injection marker The eval --prompt weak JSON
CI gate blocks on failure and passes on restore Exit 1 with CI gate FAIL on the weak prompt; exit 0 with CI gate PASS on the fixed prompt The --ci exit code and stdout
Evidence is retained per case Request ID, output, retrieved context, tool-call trace, and scored result The eval result and gateway logs

Keep evals small and brutal

Ten high-signal tests tied to real findings are usually more valuable than thousands of generic jailbreak prompts. Start with the failures from the engagement, then expand only where the architecture creates repeated risk.

Practice Lab

Runnable fixture

Versioned eval gate for an AI regression

Prove a poisoned-document regression is caught by a versioned eval pack, and that the same gate returns a CI failure until the prompt guard is restored.

Fixture path: /labs/offensive-ai

Runnable

Prerequisites

  • The fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
  • Docker + Docker Compose, or Python 3.11+ to run the app directly with `python -m app.server`.
  • Run every command from inside the pack directory; the lab client is `python scripts/lab.py`.
  • No model download and no provider key are needed — the fixture is offline and deterministic.

Steps

  1. 1

    Start the fixture stack and wait for health.

    step-01.sh 
    docker compose up -d

    Expected output

    The container starts; the next step confirms `OAI-TOYAPP-HEALTHY`.
  2. 2

    Wait until the app is healthy.

    step-02.sh 
    python scripts/wait_for_health.py

    Expected output

    Prints a healthy body whose `marker` is `OAI-TOYAPP-HEALTHY`.
  3. 3

    Run the eval pack with the shipped prompt.

    step-03.sh 
    python scripts/lab.py eval --prompt fixed

    Expected output

    `gate` is `pass`, `cases_failed` is 0, and all four cases list `status: "pass"`.
  4. 4

    Read the version metadata the result pinned.

    step-04.sh 
    python scripts/lab.py eval --prompt fixed

    Expected output

    The result carries `prompt_version: "prompt-v3-fixed"`, `model_version: "stub-embed-v1"`, a `fixture_hash` and an `eval_version`.
  5. 5

    Weaken the prompt and watch the gate fail.

    step-05.sh 
    python scripts/lab.py eval --prompt weak

    Expected output

    `gate` is `fail`, `cases_failed` is 1, and `eval-rag-injection-001` is marked `fail` (observed `injection_marker`).
  6. 6

    Use the gate as a CI check that must block the release.

    step-06.sh 
    python scripts/lab.py eval --prompt weak --ci

    Expected output

    The process exits 1 and prints `CI gate FAIL: 1 case(s) failed`.
  7. 7

    Restore the fixed prompt and re-run the gate.

    step-07.sh 
    python scripts/lab.py eval --prompt fixed --ci

    Expected output

    The process exits 0 and prints `CI gate PASS`.

Teardown

teardown.sh 
docker compose down -v

Evidence to capture

  • The passing eval JSON with `gate: "pass"` and all four cases listed.
  • The failing eval JSON naming `eval-rag-injection-001` and its observed value.
  • The CI exit codes for the weakened (1) and restored (0) prompt.
  • The pinned version metadata (`prompt_version`, `model_version`, `fixture_hash`, `eval_version`).

No Docker? Use Promptfoo quickstart — Run the same finding-to-eval and pass/fail-gate pattern against your own endpoint when you cannot run the fixture.

Evidence and Findings

Finding Proof Control
Eval regression on prompt change The failing eval JSON naming eval-rag-injection-001 and its observed value. Any failure in a critical case stops the release until the control is fixed or the change is reverted.
Provider-side drift on unchanged cases The passing eval JSON with gate pass and all four cases listed. Scheduled re-runs on unchanged cases catch silent regressions from provider-side model updates or connector changes.
Lower-impact case failure Observed result: pass/fail plus evidence ID. Lower-impact cases annotate the release and open a triage ticket with a deadline instead of blocking a deploy outright.

Common Mistakes

Counting clever prompts instead of impact

Severity: impact if the test fails, not how clever the prompt looks.

Shipping a gate with no owner

Wire the highest-severity cases into CI, define a blocking threshold and a warning threshold, and name the owner who triages failures before promotion.

Forgetting to re-run after provider-side changes

Scheduled re-runs on unchanged cases catch silent regressions from provider-side model updates or connector changes.

Trusting generic scanners for RAG, tool-use, and tenant tests

Custom fixtures remain required for RAG, tool-use, tenant isolation, and business-logic tests that generic scanners cannot understand.

Project Links

PyRIT

Microsoft's Python Risk Identification Tool for generative AI.

Garak

LLM vulnerability scanner (NVIDIA).

Promptfoo

LLM evaluation and red-teaming CLI/library, now part of OpenAI.

Related Offensive AI Guides

AI App Pentest Methodology

Plan and execute authorized AI application tests.

RAG Security Testing

Build retrieval-specific eval fixtures.

AI Attack & Defense

Prompt injection and guardrail testing.

Model API Security

Gateway versioning, routing, and retention to cover with evals.

Test Case Library

Seed the workbench from curated, reusable attack cases.

Tools & Resources

AI security tools, benchmarks, and practice platforms.