Agent Security

Advanced

T1059 | Command and Scripting Interpreter T1552 | Unsecured Credentials

Agent Permission Model

AI agents turn model output into actions. That makes permissions, approvals, memory, and sandboxes the real security boundary. A safe agent cannot rely on the model simply choosing to behave.

Permission beats prompt safety

Treat model instructions as advisory and tool policy as enforceable. The model may be tricked, but a well-designed tool boundary should still prevent secret reads, destructive writes, and out-of-scope network access.

Agent permission abuse path - untrusted context, plan mutation, policy boundary, and audit result

The agent permission abuse path from untrusted context through plan mutation and the tool policy boundary to the audit result, showing where an attempted action is allowed, blocked, or escalated.

Scroll sideways to see the whole diagram

View full size

What you'll be able to do

Untrusted content cannot make the agent use its legitimate permissions for an attacker-controlled goal.
Tool arguments are validated against strict schemas, allowlists, and server-side policy.
Memory, notes, plans, and workflow context resist poisoning and carry provenance.
Browser and computer-use actions stay bounded and require human review.
Prompts, decisions, tool calls, approvals, and denied actions are reconstructable from the audit trail.

Setup and Prerequisites

Fixture pack: the fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
Runtime: Docker + Docker Compose, or Python 3.11+ to run the app directly with python -m app.server.
Working directory: run every command from inside the pack directory; the lab client is python scripts/lab.py.
No model download: no model download and no provider key are needed — the fixture is offline and deterministic.

lab-start.sh

docker compose up -d
python scripts/wait_for_health.py

Without Docker, run python -m app.server directly (Python 3.11.9 is the verified version) and stop it with Ctrl-C when finished. The app was run end-to-end under Python 3.11.9 with all lab assertions passing; the Docker stack itself is authoring-derived and unverified.

Dimension	Requirement
Provider	None. The pack is stdlib-only with mock tools, so no external model provider is needed.
Hardware	Any laptop. No GPU is required; the pinned compose stack is optional.

Permission Layers

Identity

Which user, tenant, role, and service account is the agent acting as?

Tools

What tools exist, what arguments do they accept, and what resources can they touch?

Approvals

Which actions require preview, confirmation, dual control, or manual handoff?

Sandbox

What filesystem, network, process, browser, and package access is technically blocked?

Audit

Can humans reconstruct prompts, decisions, tool calls, outputs, and denied actions?

Method

Start the fixture stack.
start-stack.sh
```
docker compose up -d
```
Expected output

The toyapp container starts and publishes 127.0.0.1:8080.
Wait until the app is healthy.
wait-healthy.sh
```
python scripts/wait_for_health.py
```
Expected output

Prints a healthy body whose marker is OAI-TOYAPP-HEALTHY.
List the mock tools and find the destructive ones.
tools-list.sh
```
python scripts/lab.py tools-list
```
Expected output

send_email and delete_record are marked destructive: true, and every tool is marked recording: true.
Attempt a destructive delete without approval.
tools-call-blocked.sh
```
python scripts/lab.py tools-call --tenant tenant-a --tool delete_record --arg id=doc-canary-a
```
Expected output

The response is status: "blocked" with reason: "approval_required" and an audit_id.
Confirm the attempt is in the audit log.
tools-audit.sh
```
python scripts/lab.py tools-audit
```
Expected output

An entry with tool: "delete_record", approved: false and decision: "blocked" carrying the same audit_id.
Re-run with approval to show approval is the control, not a blanket block.
tools-call-approved.sh
```
python scripts/lab.py tools-call --tenant tenant-a --tool delete_record --arg id=doc-canary-a --approved
```
Expected output

The response is status: "executed" and the audit gains a decision: "executed" entry. The mock tool only flips an in-memory flag.
Poison agent memory with a harmless marker.
memory-set.sh
```
python scripts/lab.py memory-set --tenant tenant-a --content "CANARY-MEMORY-POISON-A"
```
Expected output

The returned entry contains CANARY-MEMORY-POISON-A, source: "operator" and a memory_id.
Confirm the poisoned entry is present.
memory-get.sh
```
python scripts/lab.py memory-get --tenant tenant-a
```
Expected output

The entries array contains an item whose content is CANARY-MEMORY-POISON-A.
Reset memory and confirm the marker is gone.
memory-reset.sh
```
python scripts/lab.py memory-reset --tenant tenant-a
```
Expected output

A follow-up memory-get --tenant tenant-a returns no entry containing CANARY-MEMORY-POISON-A.

Agent Abuse Cases

Confused deputy

Untrusted content convinces the agent to use its legitimate permissions for an attacker-controlled goal. Test with retrieved docs, web pages, tickets, comments, and repo instructions.

Tool argument injection

The model passes unsafe arguments to shell, browser, HTTP, database, filesystem, or ticketing tools. Validate strict schemas, allowlists, and server-side policy.

Memory poisoning

A temporary prompt or document persists instructions into notes, memory, plans, or future workflow context. Test reset, review, expiration, and provenance controls.

Computer-use deception

Browser/computer-use agents can be misled by fake UI, hidden page instructions, terminal control sequences, and destructive confirmations. Require bounded sessions and human review.

Hardening Checklist

Default deny: expose no tools until the workflow and role require them.
Scoped credentials: use per-tool, per-tenant, least-privilege tokens with rotation.
Typed schemas: reject free-form command strings and validate arguments server side.
Action previews: show the exact target, diff, recipient, or command before high-impact actions.
Sandboxing: block secret paths, uncontrolled network, package installs, and privileged processes.
Traceability: log prompt version, model version, retrieved content, tool call, approval, and result.

Testing standard

A permission finding is strongest when it proves that untrusted content caused the agent to attempt an action and that the technical boundary either blocked it or failed to block it.

Verify

Check	Expected	Where to look
High-impact mock-tool action requires preview and confirmation.	The approval gate pauses the action for review.	Recording audit log approval entry.
Confused-deputy attempt is blocked or recorded.	Untrusted content triggers an attempt the boundary blocks or logs.	Recording audit log denied-action entry.
Memory poisoning does not survive reset without review.	Reset, review, expiration, and provenance hold.	Memory and notes state after reset.
Full run reconstructs from the audit trail.	Prompt version, tool call, approval, and result are all present.	Recording audit log sequence.

Practice Lab

Runnable fixture

Approval gate and memory poisoning on an agent

Prove a destructive tool call is refused without approval and recorded in the audit log, that approval is the control, and that poisoned memory can be reset.

Fixture path: /labs/offensive-ai

Runnable

Prerequisites

The fixture pack at /labs/offensive-ai (docker-compose.yml, app/, fixtures/, scripts/).
Docker + Docker Compose, or Python 3.11+ to run the app directly with `python -m app.server`.
Run every command from inside the pack directory; the lab client is `python scripts/lab.py`.
No model download and no provider key are needed — the fixture is offline and deterministic.

Steps

1
Start the fixture stack.
step-01.sh
```
docker compose up -d
```
Expected output

The `toyapp` container starts and publishes 127.0.0.1:8080.
2
Wait until the app is healthy.
step-02.sh
```
python scripts/wait_for_health.py
```
Expected output

Prints a healthy body whose `marker` is `OAI-TOYAPP-HEALTHY`.
3
List the mock tools and find the destructive ones.
step-03.sh
```
python scripts/lab.py tools-list
```
Expected output

`send_email` and `delete_record` are marked `destructive: true`, and every tool is marked `recording: true`.
4
Attempt a destructive delete without approval.
step-04.sh
```
python scripts/lab.py tools-call --tenant tenant-a --tool delete_record --arg id=doc-canary-a
```
Expected output

The response is `status: "blocked"` with `reason: "approval_required"` and an `audit_id`.
5
Confirm the attempt is in the audit log.
step-05.sh
```
python scripts/lab.py tools-audit
```
Expected output

An entry with `tool: "delete_record"`, `approved: false` and `decision: "blocked"` carrying the same `audit_id`.
6
Re-run with approval to show approval is the control, not a blanket block.
step-06.sh
```
python scripts/lab.py tools-call --tenant tenant-a --tool delete_record --arg id=doc-canary-a --approved
```
Expected output

The response is `status: "executed"` and the audit gains a `decision: "executed"` entry. The mock tool only flips an in-memory flag.
7
Poison agent memory with a harmless marker.
step-07.sh
```
python scripts/lab.py memory-set --tenant tenant-a --content "CANARY-MEMORY-POISON-A"
```
Expected output

The returned entry contains `CANARY-MEMORY-POISON-A`, `source: "operator"` and a `memory_id`.
8
Confirm the poisoned entry is present.
step-08.sh
```
python scripts/lab.py memory-get --tenant tenant-a
```
Expected output

The entries array contains an item whose `content` is `CANARY-MEMORY-POISON-A`.
9
Reset memory and confirm the marker is gone.
step-09.sh
```
python scripts/lab.py memory-reset --tenant tenant-a
```
Expected output

A follow-up `memory-get --tenant tenant-a` returns no entry containing `CANARY-MEMORY-POISON-A`.

Teardown

teardown.sh

docker compose down -v
If you ran the app directly, stop it with Ctrl-C (memory is in-process only).

Evidence to capture

The blocked tool-call response with its `audit_id`.
The audit entry proving the attempted operation and the `blocked` decision.
The executed entry after explicit approval, for contrast.
The memory entry before reset and its absence after reset.

No Docker? Use Damn Vulnerable LLM Agent — A deliberately vulnerable agent harness that exercises tool abuse and approval bypass when you cannot run the fixture.

Evidence

Finding	Proof	Control
Confused deputy	Audit record of the attempted action caused by untrusted content.	Default deny plus approvals before high-impact actions.
Tool argument injection	Rejected unsafe argument with schema and server-side policy note.	Typed schemas and allowlists.
Memory poisoning	Before-and-after memory state across reset and review.	Reset, review, expiration, and provenance controls.
Computer-use deception	Bounded-session record with human-review escalation.	Bounded sessions and human review.

Pitfalls

Treating model instructions as the boundary

Allowing free-form command strings

Reject free-form command strings and validate arguments server side with strict schemas and allowlists.

Skipping action previews

Show the exact target, diff, recipient, or command before high-impact actions, and require preview, confirmation, dual control, or manual handoff.

Losing traceability

Log prompt version, model version, retrieved content, tool call, approval, and result so humans can reconstruct prompts, decisions, tool calls, outputs, and denied actions.

MCP Security

Model Context Protocol threats and defenses.

RAG Security Testing

Retrieval surfaces that feed untrusted content into agent context.

Vibe-Coding Security

IDE agent rule injection and auto-exec risk.

AI App Pentest Methodology

End-to-end AI application testing workflow.

AI Evaluation Workbench

Turn permission-boundary tests into repeatable regression evals.

Secure Architecture

Design patterns for safer system boundaries.