Prototyping & Productionisation
From POC to production — the hardest journey in FDE work.
POC to production
A POC proves the concept works. Productionisation makes it work reliably at scale, under constraints, for real users. The gap between them is enormous: POC has no auth, no RBAC, no audit, no observability, no eval, no rollback. Production has all of these. The FDE's job is to bridge that gap without losing the POC's value.
Why productionisation is hard
POCs impress in demos; production systems serve users. The 80% of work that turns a POC into production — auth, RBAC, audit, observability, eval, cost controls, rollback, runbooks, on-call — is invisible in the demo but determines whether the system survives contact with real users. Skipping it = a demo that dies in production.
Productionisation pipeline
POC → harden (auth, RBAC, input/output guards) → observe (tracing, metrics, logs) → eval (golden set + online) → cost-control (cache, routing, budgets) → rollout (pilot → measure → expand) → operate (runbooks, on-call, incident response) → iterate (feedback → improvements).
Productionisation checklist as code
from dataclasses import dataclass
from enum import Enum
class Stage(Enum):
POC = "poc"
HARDENED = "hardened"
OBSERVABLE = "observable"
EVALUATED = "evaluated"
COST_CONTROLLED = "cost_controlled"
READY_FOR_PILOT = "ready_for_pilot"
IN_PILOT = "in_pilot"
IN_PRODUCTION = "in_production"
@dataclass
class Gate:
stage: Stage
checks: list[str]
blocker_if_failed: bool
productionisation_pipeline = [
Gate(Stage.HARDENED, [
"Authentication on every endpoint",
"RBAC enforced at query, not display",
"Input guard (injection, PII, length)",
"Output guard (secrets, harmful, leaks)",
"Rate limiting per user",
"Secrets from vault, not env",
], blocker_if_failed=True),
Gate(Stage.OBSERVABLE, [
"OpenTelemetry traces on every stage",
"Metrics: latency P50/P95/P99, error rate, cost",
"Logs: structured, queryable, retained",
"Dashboards: latency, cost, quality, errors",
"Alerts: anomaly on latency/cost/quality",
], blocker_if_failed=True),
Gate(Stage.EVALUATED, [
"Golden set curated from real traffic",
"Eval runs in CI, gates deploy",
"Online eval samples 1% of traffic",
"Drift alert with auto-rollback",
], blocker_if_failed=True),
Gate(Stage.COST_CONTROLLED, [
"Semantic cache with TTL",
"Model routing with fallback",
"Per-user cost budget",
"Cost dashboard + anomaly alert",
], blocker_if_failed=True),
Gate(Stage.READY_FOR_PILOT, [
"Security review passed",
"Infra review passed",
"Runbook written",
"On-call rotation set",
"Rollback procedure tested",
"Customer sign-off",
], blocker_if_failed=True),
]
def assess(system: dict) -> Stage:
for gate in productionisation_pipeline:
if not all(system.get(c) for c in gate.checks):
return Stage.POC if gate.stage == Stage.HARDENED else gate.stage
return Stage.IN_PRODUCTIONExperiment: rollout strategy
Compare rollout strategies and their risk profiles.
What to observe
Big-bang rollouts of AI systems are reckless — wrong answers at scale cause reputational damage and are slow to roll back. Shadow mode (run alongside, not serving) is the safest first step. Pilot → expand team-by-team balances value and safety. Canary works for homogeneous user bases. Always have a tested rollback.
Production FDE work
Production FDE: harden → observe → eval → cost-control → pilot → expand → operate. Each stage has a gate. Rollout is gradual (shadow → pilot → expand). Rollback is tested BEFORE going live. Runbooks + on-call are set before pilot. The customer is informed at every stage. Iterate based on real usage data.
Challenge
Your POC impressed the customer. They want it in production next week for all 18k users. Push back constructively and propose a safe rollout plan that delivers value fast without risking a failed launch.
Production checklist
Production checklist
0 of 12 checked
Knowledge check
The customer wants big-bang rollout to 18k users next week. What's the right response?
Complete
You can now take a POC to production safely. This completes the FDE Fundamentals series — you're ready for FDE work.
Mark this chapter as complete
Track your progress and unlock the next chapter.