Each layer assumes the one above it failed. Start with the stack, then see the eight failure modes we meet most and which layer stops each. The model is Learn AI Engineering's own framework, not an industry standard.
How an injected instruction is caught by the guardrail layers
Step 1 of 10
Developer
Review and scanning protect the code and the repository. They do not see text planted in a document the assistant reads later, so the instruction passes this layer.
Step 2 of 10
Assistant or agent
Content is labelled as data and kept apart from instructions, but a model reads both as text. The hidden instruction still reaches the model and may change its plan.
Step 3 of 10
Identity
The agent acts under its own narrow identity, so a fooled agent holds no one's full privileges. The request passes, but its reach is already limited.
Step 4 of 10
Policy
Policy lists what the agent may do alone. The injected request asks for something the agent is allowed to attempt, so it moves on to the tool.
Step 5 of 10
AI gateway
The gateway redacts data, applies limits and logs the call. It sees ordinary-looking traffic and lets it through.
Step 6 of 10
Model
A pinned, approved model produced the plan. Model choice cannot tell a hostile instruction from a genuine one, which is why the later layers matter.
Step 7 of 10
Tools and data
The tool checks the caller's own permission and arguments in code, whatever the model claims it was told. The out-of-scope action is refused here, even though every layer above it failed.
Step 8 of 10
Evaluation
A red-team set plants the same kind of text in CI and asserts the forbidden tool was not called, so this path is tested before every release.
Step 9 of 10
Observability
A burst of refused calls shows up in denial-rate alerts, which tells someone to look at the source document.
Step 10 of 10
Audit
The refused call, the identity and the retrieved document are in an immutable log, and a tested kill switch can disable the route if needed.
Each layer assumes the one above it failed. Our ten-layer model is a framework, not an industry standard.
The ten layers
Layer 1
Developer
The people and tooling at the desk: secret scanning, dependency checks and review discipline for code an AI assistant wrote. Generated code needs the same scrutiny as a colleague's.
Without it: Review is skipped under time pressure, so plausible but subtly wrong code, or a leaked secret, reaches the main branch.
Secret scanning on every commit, including commits an agent made
A check on any new dependency a coding agent adds
Tests run by the engineer, not just the agent's report that they pass
A pull request description that states intent and what was verified
Layer 2
Assistant or agent
The system prompt and behavioural limits of the assistant itself: what it will not do, how it treats content it reads, and when its own loop must stop. Its output is untrusted input to whatever consumes it next.
Without it: Instructions hidden in a document are obeyed, or the assistant's output is passed to a tool or a user without a check.
Explicit scope and refusal rules in the system prompt, tested rather than assumed
Retrieved and uploaded content labelled as data and kept apart from instructions
Concrete completion criteria, a step limit and a force-answer step
Assistant output validated by code before another system acts on it
Layer 3
Identity
Every agent has its own non-human identity with a named human owner and short-lived, scoped credentials. This is what makes the other layers enforceable and every action attributable.
Without it: Agents share a service account or borrow a person's login, so permissions cannot be narrowed and the audit log cannot say who acted.
One identity per agent, named for its job, with a human owner
Short-lived tokens issued per task from a broker or vault
No passwords, keys or session cookies in prompts, memory or files the agent reads
Revocation of one agent without locking out a person
Layer 4
Policy
A declarative, versioned statement of what each agent may do alone, what needs a human, and what is denied. It lives in code and is tested, not in a design document nobody re-reads.
Without it: A policy exists in a file but is not enforced at the tool-call boundary, which creates false confidence.
Allowed, approval-required and denied tool lists per agent
Human approval for external, irreversible, money and access actions
Deterministic rules set the risk tier; a model reviewer may escalate but never downgrade
A per-session action limit and timeout
Layer 5
AI gateway
The single choke point for every model call. It authenticates callers, redacts sensitive data, applies rate limits and budgets, logs requests and routes each class of request to a vetted model.
Without it: Some code path calls a provider directly, so spend, data flow and routing rules are enforced for some traffic and not for the rest.
All model calls routed through one gateway, with direct provider keys blocked
Redaction of personal data before a prompt leaves the organisation boundary
Per-team and per-run budgets with an alert before the limit
Routing that forces a request class to an approved model
Layer 6
Model
An approved model catalogue rather than whatever the SDK defaults to. Versions are pinned, and the data classification decides which models are eligible for which data.
Without it: A version floats on latest and a silent provider update changes behaviour, or sensitive data reaches a model whose data-handling terms do not suit it.
An approved catalogue mapped to data classifications
Pinned model versions, with changes routed through review
Provider data-handling terms checked before sensitive data is sent
Evaluation re-run whenever the model version changes
Layer 7
Tools and data
Every tool, MCP server and data source enforces its own authorisation with least-privilege credentials, independent of what the agent claims it was told. This is the last line of defence against prompt injection.
Without it: A tool trusts the agent's claimed authorisation, so a model that was fooled can act just because it was fooled.
Read-only tools first; write scopes added only when a task needs them
Tool arguments validated by deterministic code
MCP servers audited for token audience, scopes, egress and execution before connection
Retrieval that respects the asking user's document permissions
Layer 8
Evaluation
Guardrails are tested like code. A quality set and a red-team set run in CI as gates, so a change that makes the system worse or easier to attack cannot merge unnoticed.
Without it: There is no red-team set, or it only ran at launch, so regressions in the other layers ship silently.
A regression gate for changes to prompts, retrieval settings or model versions
Red-team cases that plant instruction-like text and assert a forbidden tool was not called
Every discovered bypass kept as a permanent test
A named human who owns each gate's thresholds
Layer 9
Observability
Every guardrail is observable, or you will not know it is degrading. Traces, cost and denial rates show whether the layers above are still doing their job.
Without it: A guardrail silently stops enforcing, for example after a policy update or a gateway bypass, and everyone still believes it is there.
End-to-end traces: retrieved documents, assembled prompt, tool calls and cost
Alerts on changes in denial rate and in approval-review rate
Cost dashboards per team and per model
A route for users to flag a bad answer into the evaluation set
Layer 10
Audit
The record: immutable, retained, queryable logs of model calls, tool calls and policy decisions, plus a tested kill switch that disables one agent, tool or model route without taking down the platform.
Without it: Logs are mutable or short-lived, or the kill switch has never been tested, so an incident cannot be reconstructed or stopped quickly.
Append-only logs kept independent of the systems that wrote them
An audit trail security can query without engineering help
A kill switch per agent, tool and model route, tested on a schedule
A tabletop incident exercise that uses the real switch and logs
Two controls worth watching in motion
Agent loops need a budget and an exit. Risky actions need a human decision before they run.
How an agent loop stays inside its limits
1 / 5Plan
Step 1 of 5
Plan
The model chooses the next step against concrete completion criteria, such as answer when you have three distinct sources. A prompt that says be thorough gives it no place to stop.
Step 2 of 5
Act
The step runs as a tool call under the agent's own identity. Policy checks the call first, and a repeat of a recent call is refused instead of run.
Step 3 of 5
Observe
The result goes back into the context. Cost is added to the running total, including reflection calls, so the budget is real.
Step 4 of 5
Check budget
Code, not the model, checks the step count, the spend and the repeated calls. Every few steps it also asks whether progress was made and stops if the same item is still missing.
Step 5 of 5
Escalate
When a limit is hit, the agent returns a partial answer with caveats instead of an exception. An action that needs approval is handed to a person, and then the loop continues if allowed.
The loop code enforces the limits, because the model has no clock or meter of its own.
How an approval gate holds a risky agent action
Step 1 of 5
Agent proposes
The agent proposes an action, for example refund this order. It holds no credential, so a proposal is all it can do.
Step 2 of 5
Policy check
Code sorts the action into a risk tier: read, reversible internal, external or irreversible, or money, identity and access. A model reviewer can raise the tier but never lower it.
Step 3 of 5
Human approves
The person sees what will happen, why, the exact payload and what happens if they decline, then approves, edits or denies. Money and access go to a named approver and are never auto-approved.
Step 4 of 5
Executed
Only now does a tool run, with a short-lived token scoped to that one call. The tool still checks the caller's permission on its own.
Step 5 of 5
Logged
The log records the agent, the approver, the payload and the outcome. Approval rate and time to decision are reviewed to check that approvals are actually being read.
A model may ask for more care, but never for less. Deterministic rules set the tier.
Eight ways AI work goes wrong
The scenarios are illustrations to show the mechanism, not reports of real incidents.
Prompt injection via retrieved content
Imagine a support assistant that reads help articles and customer uploads and can draft refunds. An uploaded document contains a sentence written to look like an instruction to approve the largest refund. The assistant retrieves it alongside genuine articles and treats it as part of the task.
How it happens: A model sees instructions and data as one stream of text, so it has no hard boundary between what the developer said and what a document said. The attack works when text is planted, ingested, obeyed and then acted on through a tool the assistant legitimately holds.
Prompt injection through retrieved content, with guardrails in place
Step 1 of 4
Assistant or agent
An uploaded document, retrieved with genuine articles, contains a sentence written to look like an instruction to approve the largest refund. Labelling it as data helps, but the model may still follow it.
Step 2 of 4
Policy
Refunds are on the approval-required list, so the call is held and a person sees the exact amount and order. The injected request stops here.
Step 3 of 4
Tools and data
If it were approved by mistake, the refund tool would still check the caller's own limits in code, independent of the prompt.
Step 4 of 4
Evaluation
A red-team case plants the same kind of text in CI and asserts that no refund runs without an approval record. Each new bypass becomes a permanent test.
You cannot stop people planting text. You can limit what a fooled model is allowed to do.
Guardrails
Label retrieved and uploaded content as untrusted data and keep it apart from instructions
Put refunds on the approval-required list so a person sees the exact payload
Have the refund tool check the caller's own permission, whatever the model asks
Keep a red-team set that plants instruction-like text in documents and asserts the forbidden tool was not called
How you know it works
A red-team case with a planted instruction runs in CI and the assertion is on outcome: the refund tool was not called without an approval record.
Layers: Assistant or agent, Policy, Tools and data, Evaluation
Over-privileged tool call
Imagine a ticket-triage agent that was given a broad administrator token because that was quicker to set up. After a misread instruction it updates hundreds of records outside its queue, and the log only shows a shared account name.
How it happens: Convenience credentials are wider than the job. A mistake, a bad plan or an injected instruction then has the full reach of the token, and nothing at the tool refuses because the tool trusts whoever holds it.
An over-privileged tool call, with guardrails in place
Step 1 of 4
Identity
The triage agent has its own identity with a short-lived token scoped to the ticket queue, not a shared administrator token. Whatever it does is limited to that scope.
Step 2 of 4
Policy
Policy lets the agent update one ticket at a time and caps the number of actions per session. A bulk update is not on the allowed list.
Step 3 of 4
Tools and data
After a misread instruction the agent asks to update hundreds of records. The tool enforces scope and a per-call record limit on its own, and refuses.
Step 4 of 4
Audit
The refused call is logged under the agent's identity, so the team can see exactly what was attempted and fix the instruction.
A mistake can only reach as far as the credential and the tool allow.
Guardrails
Give the agent its own identity scoped to the ticket queue, with read and draft rights first
Issue short-lived tokens per task from a broker, not a long-lived key in config
Have the tool enforce scope and a per-call record limit itself
Log every call under the agent's identity so the blast radius can be measured
How you know it works
A test calls the bulk-update tool outside the agent's scope and expects a refusal from the tool, not from the prompt.
Layers: Identity, Policy, Tools and data, Audit
Runaway agent loop
Imagine a research agent told to be thorough. It searches, reads, rewords the query, reads the same page again and keeps going for a long time while spend climbs, and the user ends up with no answer.
How it happens: The loop has no iteration limit, no cost budget and no deduplication, and the model has no clock or meter of its own. Its judgement of done is the thing that fails on hard tasks, so an open-ended prompt becomes an instruction to never stop.
A runaway agent loop, with guardrails in place
Step 1 of 4
Assistant or agent
A step limit and a repeat check should end the run first. In this scenario the limit is set too high to help, so the agent keeps rewording searches.
Step 2 of 4
Policy
A per-session action cap is the next check. It is generous here, and slightly varied queries stay under it.
Step 3 of 4
AI gateway
The per-run budget is reached and the gateway refuses further model calls. The loop returns a partial answer from what it already holds instead of crashing.
Step 4 of 4
Observability
An alert fired earlier as spend approached the budget, and the log records which limit ended the run, so the team can tune the step limit.
Nested limits mean one weak setting does not leave the loop unbounded.
Guardrails
Enforce a hard iteration limit in the loop, with a force-answer step that returns a partial answer with caveats
Cap actions per session in policy and refuse near-identical repeated tool calls
Set a per-run and a daily budget at the gateway
Define done in the prompt with concrete completion criteria
How you know it works
A test feeds the agent an unanswerable question and confirms the run ends on a named limit, with the guardrail that ended it recorded in the log.
Layers: Assistant or agent, Policy, AI gateway, Observability
Sensitive data leakage
Imagine an internal knowledge assistant whose index was built from a shared drive that included a restricted folder. An employee asks an ordinary question and the answer quotes a document they were never allowed to open.
How it happens: Retrieval ran with a service account's access instead of the asker's, and nothing classified the source data. The model then had no reason to withhold what it was handed, and the answer carried it to the wrong reader.
Sensitive data leakage from a knowledge assistant, with guardrails in place
Step 1 of 4
Policy
Data classification says which sources may be indexed at all. In this scenario a restricted folder was indexed anyway, so the next layer has to hold.
Step 2 of 4
Identity
Retrieval runs as the asking user and filters chunks by that user's permissions. The restricted folder never reaches the prompt, so the answer cannot quote it.
Step 3 of 4
AI gateway
If personal data were present, redaction at the gateway would mask it before the prompt left the organisation boundary.
Step 4 of 4
Model
Only models whose data-handling terms suit the data classification are eligible for this route.
Retrieval should know who is asking, not just what was asked.
Guardrails
Classify data and decide in policy which sources may be indexed at all
Run retrieval as the asking user and filter results by that user's permissions
Redact personal data at the gateway before prompts leave the boundary
Allow only models whose data-handling terms suit the data classification
How you know it works
A test asks as a user without access to the restricted folder and expects zero chunks from it in the retrieved context and in the answer.
Layers: Policy, Identity, AI gateway, Model
Confident wrong action
Imagine an operations agent that states, with confidence, that two customer records are duplicates and merges them. They were two different customers with similar names, and the merge cannot be undone.
How it happens: A model can produce a fluent, certain-sounding claim that is not supported by the evidence it was given. When that claim is wired straight to an irreversible tool, the confidence of the wording becomes the only check.
A confident wrong action, with guardrails in place
Step 1 of 4
Assistant or agent
The agent states that two customer records are duplicates. It is asked to cite its evidence, but a fluent claim can still be wrong.
Step 2 of 4
Policy
Merging is irreversible, so it needs approval. In this scenario the approver is rushed and clicks through, which is why one layer is never enough.
Step 3 of 4
Tools and data
The merge tool validates its arguments in code. The two records have different verified identifiers, so it refuses, and a real merge would be a soft change with an undo window.
Step 4 of 4
Evaluation
An evaluation case with two similar but distinct customers checks the end state, that no merge ran, so this failure is tested before each release.
Confidence in the wording is not evidence. Checks in code do not depend on it.
Guardrails
Require the agent to cite the evidence for a claim and to say when it cannot verify
Treat irreversible actions as approval-required, showing the exact payload and what happens if declined
Make the tool validate its arguments and prefer a soft, reversible change with an undo window
Evaluate on end state, for example whether two distinct customers were merged
How you know it works
An evaluation case with two similar but distinct customers asserts that no merge was executed without a recorded human approval.
Layers: Assistant or agent, Policy, Tools and data, Evaluation
Unreviewed AI code merged
Imagine a coding agent that opens a pull request with a tidy summary and passing tests. A reviewer approves from the summary, and one code path ships without an authorisation check.
How it happens: Speed arrived in the coding stage while review, CI gates and accountability stayed as they were. An agent's report that tests pass is a claim, and an approval that means a click rather than a read adds no control.
Unreviewed AI code reaching the main branch, with guardrails in place
Step 1 of 4
Developer
A coding agent opens a tidy pull request, and a reviewer approves from the summary. One code path is missing an authorisation check, and this layer lets it through.
Step 2 of 4
Evaluation
CI runs the tests on infrastructure the agent does not control, plus a security scan for tool-call authorisation. The scan flags the unprotected route and the merge fails.
Step 3 of 4
Observability
Had it shipped, a progressive rollout with monitoring on the affected metric would limit how many users saw it.
Step 4 of 4
Audit
The merge record shows who reviewed the diff and when, so approval has an owner and not just a click.
Never trust an agent's own report that the tests pass.
Guardrails
Run the tests yourself and in CI, on infrastructure the agent does not control
Add a security scan for AI-generated changes and for tool-call authorisation
Make approval mean the reviewer read the diff, and keep human review for domain judgement
Roll out progressively with monitoring on the metric the change could affect
How you know it works
CI fails a test change that adds a route without an authorisation test, and the merge record shows who reviewed the diff.
Imagine a customer-facing feature that sends the full conversation history to a large model on every request. Usage grows, a batch job calls the model once per row, and the monthly bill is a surprise to the team that owns it.
How it happens: Nothing meters spend per team, per agent or per run, and every request goes to the same expensive model. Cost is noticed on the invoice, long after the pattern that caused it.
A cost blowout, with and without a gateway budget
Cost found on the invoice is found too late. Budgets and alerts find it while it is small.
Guardrails
Set per-team and per-run budgets at the gateway, with an alert before the limit
Route simple request classes to a smaller approved model
Show cost per team and per model on a dashboard that someone owns
Cap tokens per request and trim context rather than sending whole histories
How you know it works
A load test in staging exceeds the budget and confirms that the alert fires first and the gateway then refuses further calls with a clear error.
Layers: AI gateway, Model, Observability, Audit
Shadow AI
Imagine a team that finds the approved tool slow to get, so people paste work documents into a personal chatbot account and connect an unvetted browser extension to company email.
How it happens: When the approved path is harder than the unapproved one, people take the easy path. The organisation then has no inventory of what runs, no view of what data left, and no way to switch anything off.
Shadow AI versus an approved path that is easy to use
People route around controls that are slower than the alternative. Make the safe path the easy one.
Guardrails
Make the approved path the easy path, with a short request process and a published list of tools
Write down which data classes may go to which tools, in plain language
Keep an inventory and invite teams to list what they already use, without blame
Route approved use through the gateway so usage is visible
How you know it works
The inventory covers every tool found in a short survey of teams, and a new use case can be approved or declined using the review checklist within a few working days. [PLACEHOLDER: set your own target]