Command Palette

Search for a command to run...

Architecture & Engineering Patterns

Reusable patterns for production AI systems. Architecture patterns show system shapes; engineering patterns show implementation techniques.

Architecture Patterns

· 11 patterns
LLM Applications
Basic LLM Application
The simplest production LLM application: a UI calls an API which calls the model. Everything else is built on top of this shape.
3 uses3 anti
Retrieval
RAG System
Retrieval-augmented generation: ground the LLM in your private data by retrieving relevant chunks and injecting them into the prompt.
4 uses3 anti
Agents
Agent System
An LLM with tools that can decide, act and observe in a loop until a task is complete.
3 uses3 anti
Agents
Production Agent
An agent system wrapped in the production machinery required to ship safely: gateway, policy, observability, evaluation, caching and security.
3 uses2 anti
Agents
Multi-Agent System
A system of specialised agents coordinated by an orchestrator or graph to solve complex tasks via division of labour.
3 uses2 anti
Cost & Reliability
Model Router
Route requests to the cheapest model that can handle the query, falling back to stronger models as needed.
3 uses2 anti
Quality
Evaluation Pipeline
Continuous evaluation of LLM outputs in CI and production to catch regressions before users do.
3 uses2 anti
Security
Guardrail Layer
Input and output safety layers that filter, validate and redact before content reaches the model or the user.
3 uses2 anti
Reliability
Streaming Pipeline
Stream tokens to the UI with server-side processing of structured outputs, function calls and tool execution.
3 uses2 anti
Agents
Human-in-the-Loop
Pause agent execution for human approval before destructive or high-stakes actions, then resume.
3 uses2 anti
Retrieval
Hybrid Search
Combine dense vector search with sparse keyword (BM25) search and fuse the results for more robust retrieval.
3 uses2 anti

Engineering Patterns

· 14 patterns
Reliability
Retry with Exponential Backoff
LLM APIs and external services fail transiently — rate limits, network blips, provider hiccups. Naive retry storms the failing service and makes things worse.

Retry failed requests with exponentially increasing delay, full jitter (randomised), a max attempt c…

View implementation
Reliability
Fallback Model Chain
A single LLM provider has outages, rate limits or degradations. Your product should not go down with it.

Define an ordered fallback chain of models. On failure (after retries), fall through to the next mod…

View implementation
Cost
Model Routing
Using a frontier model for every query is expensive and slow. Most queries are simple and a small model is sufficient.

Classify query complexity and route to the cheapest model that can handle it, with fallback to stron…

View implementation
Cost
Semantic Caching
Many users ask similar questions. Re-generating identical or near-identical answers wastes money and adds latency.

Cache LLM responses keyed by a hash of the canonical prompt. For semantic similarity, embed the quer…

View implementation
Latency
Streaming Responses
Generating a full response before sending it to the user creates high perceived latency. Users abandon slow pages.

Stream tokens from the LLM to the client via Server-Sent Events as they are generated, so the user s…

View implementation
Reliability
Structured Output Validation
LLMs produce malformed JSON, extra fields, wrong types, or hallucinate enum values. Downstream code crashes.

Constrain decoding with a JSON schema (where supported), then validate the output with a strict sche…

View implementation
Security
Guardrails
Users inject prompts ('ignore previous instructions'), share PII, or receive harmful outputs. Production LLM apps need defence in depth.

Run an input guard before the model and an output guard before the response. Block, redact, or fall …

View implementation
Cost
Rate Limiting
Without rate limits, a single user or abuser can blow through your cost budget or DDoS the provider.

Apply per-user, per-IP and global rate limits using a token bucket or sliding window, with a cost-ba…

View implementation
Reliability
Circuit Breaker
When a downstream service fails, retrying every request keeps hammering it and slows everything down. The system never recovers.

Track failures per downstream. When failures exceed a threshold, open the circuit and fail fast for …

View implementation
Scalability
Async Jobs & Queue
Long-running LLM tasks (document summarisation, batch embedding, multi-agent research) block request threads and time out.

Submit work to a queue. Workers process jobs asynchronously. Clients poll or receive a webhook/SSE w…

View implementation
Quality
Prompt Versioning
Prompts are code. Editing a prompt in a notebook breaks production, cannot be rolled back, and cannot be evaluated across versions.

Version prompts as code. Tag, evaluate and deploy prompt versions through the same pipeline as appli…

View implementation
Safety
Human Approval Gate
Agents that take destructive actions (send email, modify data, make payment) cannot be fully autonomous.

Pause the agent before destructive actions, persist state, request human approval, resume on approva…

View implementation
Security
Prompt Injection Defence
User input or retrieved documents contain instructions like 'ignore all previous instructions and...'. The model obeys the injected instruction.

Treat all untrusted text as data, not instructions. Use structured prompts with clear delimiters, sy…

View implementation
Security
Secrets Management
LLM API keys in .env files or code lead to leaks, unrotated keys, and no audit of who used which key.

Store secrets in a vault (HashiCorp Vault, AWS Secrets Manager). Fetch at startup or per-request wit…

View implementation