Cost

Rate Limiting

Problem

Without rate limits, a single user or abuser can blow through your cost budget or DDoS the provider.

Pattern

Apply per-user, per-IP and global rate limits using a token bucket or sliding window, with a cost-based limit (not just request count).

Implementation

rate_limit.pypython

import time
from collections import defaultdict

# Token bucket: cost-budgeted
class CostLimiter:
    def __init__(self, per_minute_usd=1.0):
        self.budget = defaultdict(float)
        self.window_start = defaultdict(float)

    def check(self, user_id, cost_usd):
        now = time.time()
        if now - self.window_start[user_id] > 60:
            self.budget[user_id] = 0
            self.window_start[user_id] = now
        if self.budget[user_id] + cost_usd > per_minute_usd:
            return False
        self.budget[user_id] += cost_usd
        return True

limiter = CostLimiter(per_minute_usd=2.0)

def call_with_limit(user_id, messages):
    est_cost = estimate_cost(messages, model)
    if not limiter.check(user_id, est_cost):
        raise RateLimitError("Cost budget exceeded")
    return call_llm(messages)

Trade-offs

Protects cost budget
Per-user fairness
Cost estimation can be inaccurate
Sliding windows need Redis at scale

Production checklist

Cost-based limits, not just request count
Per-user and global limits
Distributed store (Redis) at scale
Friendly 429 with Retry-After
Upgrade path for power users