Your AI feature costs $4,800/day. How do you cut costs by 80% without hurting quality?
Think first
Think about: caching, model routing, prompt compression, budget enforcement.
Strong answer
Three levers that compound: (1) Semantic caching — cache responses keyed by prompt hash + semantic similarity (0.95 threshold). Typical hit rate 40-60%, instantly cutting cost. (2) Model routing — classify query complexity, route simple queries to a cheap model (gpt-4o-mini is ~17x cheaper than gpt-4o), fall back to frontier only on low confidence. 80% of traffic can use the small model. (3) Prompt/context compression — summarise conversation history instead of sending full history; a 450-token prompt growing to 8,200 tokens is an 18x cost multiplier. Combined: cache (0.6 saving) × routing (0.25 saving on remaining) × compression (0.5 on remaining) = ~90% total reduction. Add per-user cost budgets and anomaly alerts to prevent regression.
Weak-answer indicators
- Only suggests 'use a cheaper model'
- Doesn't mention caching
- Ignores prompt/context size as a cost driver
- Doesn't quantify the savings
Likely follow-up questions
- How do you handle cache invalidation?
- What's a good similarity threshold for semantic caching?
- How do you route queries without adding latency?