Cost

Semantic Caching

Problem

Many users ask similar questions. Re-generating identical or near-identical answers wastes money and adds latency.

Pattern

Cache LLM responses keyed by a hash of the canonical prompt. For semantic similarity, embed the query and match against cached queries above a similarity threshold.

Implementation

semantic_cache.pypython

import hashlib
from dataclasses import dataclass

@dataclass
class CacheEntry:
    query_hash: str
    query_embedding: list[float]
    response: str
    model: str
    created_at: float

def get_or_call(messages, model, similarity_threshold=0.95):
    query = json.dumps(messages)
    qhash = hashlib.sha256(query.encode()).read()

    # Exact hit
    if entry := cache.get(qhash):
        return entry.response

    # Semantic hit
    qemb = embed(query)
    for entry in cache.recent_entries(model, limit=50):
        sim = cosine(qemb, entry.query_embedding)
        if sim >= similarity_threshold:
            return entry.response

    # Miss — call the model
    response = call_llm(messages, model)
    cache.set(qhash, qemb, response, model)
    return response

Trade-offs

Massive cost reduction on repeated queries
Semantic cache can return wrong answer on near-misses
Cache invalidation on prompt or data changes
Storage cost for embeddings

Production checklist

TTL per cache entry
Version cache by prompt template version
Invalidate on data ingestion changes
Log cache hit rate per endpoint
Bypass cache for personalised / user-specific data
Tune similarity threshold carefully