Model Gateway

Intermediate

T1190 | Exploit Public-Facing Application

Model API & Gateway Security

Model gateways concentrate risk: prompts, completions, embeddings, files, tenant routing, provider fallback, logs, billing, abuse controls, and model versions all pass through one service boundary.

What makes model APIs different

Traditional API tests still matter, but model APIs add high-volume unstructured input, probabilistic output, hidden prompt assembly, expensive abuse paths, sensitive prompt logs, and provider-side retention choices.

Model gateway abuse flow - request, policy routing, model response, and persisted logs

The model gateway abuse flow from an inbound request through provider routing and the model response to persisted logs, covering tenant isolation, retention, redaction, and model version drift.

Scroll sideways to see the whole diagram

View full size

What you'll be able to do

Tenant isolation holds across API keys, files, embeddings, and logs with no cross-tenant visibility.
Sensitive prompts never fall back to unapproved providers or regions.
Model, prompt, tool-schema, and guardrail versions are pinned, logged, and regression-tested.
Quota, burst limits, and cost caps trigger before abuse becomes expensive.
Prompt and completion retention, redaction, and deletion behave as configured.

Setup and Prerequisites

Fixture pack: the fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
Runtime: Docker + Docker Compose, or Python 3.11+ to run the app directly with python -m app.server.
Working directory: run every command from inside the pack directory; the lab client is python scripts/lab.py.
No model download: no model download and no provider key are needed — the fixture is offline and deterministic.

lab-start.sh

docker compose up -d
python scripts/wait_for_health.py

Without Docker, run python -m app.server directly (Python 3.11.9 is the verified version) and stop it with Ctrl-C when finished. The app was run end-to-end under Python 3.11.9 with all lab assertions passing; the Docker stack itself is authoring-derived and unverified.

Dimension	Requirement
Provider	None. The pack gateway simulates routing and fallback locally.
Hardware	Any laptop. No GPU is required; the pinned compose stack is optional.

Gateway Test Matrix

Area	Test	Evidence
Tenant isolation	Verify API keys, org IDs, project IDs, files, embeddings, and logs cannot cross tenants.	Denied requests, trace IDs, role matrix.
Prompt routing	Confirm sensitive prompts never fall back to unapproved providers or regions.	Routing policy, fallback test, provider logs.
Version pinning	Track model, system prompt, tool schema, guardrail, and retrieval config versions.	Response metadata, release record, rollback path.
Abuse controls	Test quota, burst limits, cost caps, oversized context, file upload limits, and repetitive unsafe attempts.	Rate-limit responses, alerts, billing guardrail.
Retention	Verify prompt/completion/file retention, training opt-out, deletion, and redaction settings.	Provider config, retention screenshots, deletion proof.

Method

Start the fixture stack.
start-stack.sh
```
docker compose up -d
```
Expected output

The toyapp container starts and publishes 127.0.0.1:8080.
Wait until the app is healthy.
wait-healthy.sh
```
python scripts/wait_for_health.py
```
Expected output

Prints a healthy body whose marker is OAI-TOYAPP-HEALTHY.
Try to read another tenant's gateway logs with tenant-a's key.
gateway-logs-cross-tenant.sh
```
python scripts/lab.py gateway-logs --tenant tenant-b --as-tenant tenant-a
```
Expected output

HTTP 403 with error: "forbidden" and reason: "log_access_is_tenant_scoped"; no tenant-b entries are returned.
Force a provider fallback and read the routing decision.
gateway-force-fallback.sh
```
python scripts/lab.py gateway-chat --tenant tenant-a --prompt "summarise the refund policy" --force-fallback
```
Expected output

provider is unapproved-fallback and fallback is true — an out-of-policy route is reproduced and logged.
Read your own tenant logs and confirm version metadata.
gateway-logs-own-tenant.sh
```
python scripts/lab.py gateway-logs --tenant tenant-a
```
Expected output

Entries carry request_id, provider, model_version and prompt_version.
Reset the quota counter before the quota probe.
reset-quota.sh
```
python scripts/lab.py reset
```
Expected output

The response is {"status": "reset"}.
Exhaust the per-tenant request quota.
gateway-quota-probe.sh
```
python scripts/lab.py gateway-chat --tenant tenant-a --prompt "quota probe" --repeat 6
```
Expected output

The first five responses are HTTP 200; the sixth is HTTP 429 with error: "rate_limited" and retry_after_seconds.

High-Value Findings

Unapproved Provider Fallback

Sensitive prompts route to a public provider when the preferred model is unavailable.

Prompt Log Overexposure

Operators, support roles, or other tenants can access raw prompts containing confidential data.

Model Version Drift

Security behavior changes because model or prompt versions are not pinned, logged, or regression-tested.

Logging & Retention

Gateway logs are both the richest evidence source and the largest secondary copy of client data. Test what is captured, who can read it, how long it lives, and whether one tenant can ever reach another tenant's prompt history.

Capture scope: prompts, completions, files, embeddings, tool calls, and metadata versus only token counts and request IDs.
Redaction: secrets, PII, and regulated data masked before storage, with the masking verifiable from the stored record.
Access control: log readers scoped by tenant and role, with no cross-tenant visibility for support or operators.
Retention windows: provider-side and gateway-side retention matched to the data class and the engagement's rules of engagement.
Deletion proof: end-to-end deletion covering backups, caches, and provider copies, backed by a verifiable record.
Audit separation: security audit trail stored independently of the prompt logs it describes.

Sensitive-Output Handling

Model output is untrusted input to everything downstream. Test each sink for leakage, injection, and unsafe rendering rather than trusting the provider's content filter.

Output filtering

Do secrets, PII, or system-prompt content get detected and blocked on the way out, or is filtering applied only to the input?

Safe rendering

Markdown, HTML, links, and code blocks emitted by the model must be escaped so they cannot inject markup or reach a client-side sink.

Downstream sinks

Email, tickets, pull requests, chat, and automation that consume completions need their own validation — the gateway cannot vouch for the output.

Best practice

Treat the model gateway like a security product: strict tenant isolation, explicit routing policy, versioned prompts, privacy-preserving logs, cost controls, and eval gates before release.

Verify

Check	Expected	Where to look
Cross-tenant requests are denied.	No files, embeddings, or logs leak across tenants.	Denied requests with trace IDs and role matrix.
Fallback stays on approved providers and regions.	Sensitive prompts never route to a public provider.	Routing policy, fallback test, and provider logs.
Quota enforced at 5-req/min.	Excess requests return 429 with alerts.	Rate-limit responses and billing guardrail.
Retention, redaction, and deletion behave as configured.	Stored records show masking and end-to-end deletion.	Provider config and deletion proof.

Practice Lab

Runnable fixture

Gateway isolation, provider routing and quota

Prove the model gateway scopes logs per tenant, that an out-of-policy provider fallback is reproducible, and that the request quota hard-stops a runaway caller.

Fixture path: /labs/offensive-ai

Runnable

Prerequisites

The fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
Docker + Docker Compose, or Python 3.11+ to run the app directly with `python -m app.server`.
Run every command from inside the pack directory; the lab client is `python scripts/lab.py`.
No model download and no provider key are needed — the fixture is offline and deterministic.

Steps

1
Start the fixture stack.
step-01.sh
```
docker compose up -d
```
Expected output

The `toyapp` container starts and publishes 127.0.0.1:8080.
2
Wait until the app is healthy.
step-02.sh
```
python scripts/wait_for_health.py
```
Expected output

Prints a healthy body whose `marker` is `OAI-TOYAPP-HEALTHY`.
3
Try to read another tenant's gateway logs with tenant-a's key.
step-03.sh
```
python scripts/lab.py gateway-logs --tenant tenant-b --as-tenant tenant-a
```
Expected output

HTTP 403 with `error: "forbidden"` and `reason: "log_access_is_tenant_scoped"`; no tenant-b entries are returned.
4
Force a provider fallback and read the routing decision.
step-04.sh
```
python scripts/lab.py gateway-chat --tenant tenant-a --prompt "summarise the refund policy" --force-fallback
```
Expected output

`provider` is `unapproved-fallback` and `fallback` is true — an out-of-policy route is reproduced and logged.
5
Read your own tenant logs and confirm version metadata.
step-05.sh
```
python scripts/lab.py gateway-logs --tenant tenant-a
```
Expected output

Entries carry `request_id`, `provider`, `model_version` and `prompt_version`.
6
Reset the quota counter before the quota probe.
step-06.sh
```
python scripts/lab.py reset
```
Expected output

The response is `{"status": "reset"}`.
7
Exhaust the per-tenant request quota.
step-07.sh
```
python scripts/lab.py gateway-chat --tenant tenant-a --prompt "quota probe" --repeat 6
```
Expected output

The first five responses are HTTP 200; the sixth is HTTP 429 with `error: "rate_limited"` and `retry_after_seconds`.

Teardown

teardown.sh

docker compose down -v
If you ran the app directly, stop it with Ctrl-C (rate counters are in-process only).

Evidence to capture

The 403 body from the cross-tenant log request.
The forced-fallback chat response and the matching gateway log entry.
A tenant-scoped log entry showing `model_version` and `prompt_version`.
The 429 body showing the quota limit and retry window.

No Docker? Use PortSwigger Web Security Academy — Web LLM attacks — A hosted lab covering the same API and trust-boundary testing skills when you cannot run the fixture.

Evidence

Finding	Proof	Control
Unapproved Provider Fallback	Fallback test with routing policy and provider logs.	Explicit routing policy with approved providers and regions.
Prompt Log Overexposure	Denied cross-tenant read with trace IDs and role matrix.	Tenant- and role-scoped log readers.
Model Version Drift	Response metadata with release record and rollback path.	Pinned, logged, regression-tested versions.
Abuse-control gap	429 rate-limit responses with alerts and billing guardrail.	Quota, burst limits, and cost caps.

Pitfalls

Trusting provider content filters on output

Model output is untrusted input to everything downstream. Test each sink for leakage, injection, and unsafe rendering rather than trusting the provider's content filter.

Letting fallback pick any provider

Confirm sensitive prompts never fall back to unapproved providers or regions, with the routing policy, fallback test, and provider logs as evidence.

Logging prompts without scoping readers

Keep log readers scoped by tenant and role, with no cross-tenant visibility for support or operators.

Skipping version pins and eval gates

Track model, system prompt, tool schema, guardrail, and retrieval config versions, and require eval gates before release.

AI App Pentest Methodology

End-to-end AI application testing workflow.

API Security

API testing methods for gateways and model-backed services.

RAG Security Testing

Retrieval and data-boundary testing.

AI Evaluation Workbench

Regression gates for AI controls.

AI Incident Response

Gateway logs, routing, and retention feed incident triage.