Scalability

Async Jobs & Queue

Problem

Long-running LLM tasks (document summarisation, batch embedding, multi-agent research) block request threads and time out.

Pattern

Submit work to a queue. Workers process jobs asynchronously. Clients poll or receive a webhook/SSE when done.

Implementation

queue.pypython

import redis
from rq import Queue

q = Queue(connection=redis.from_url("redis://localhost:6379"))

@app.post("/api/summarise")
def summarise(req):
    job = q.enqueue(summarise_document, req.doc_id, timeout=600)
    return {"job_id": job.id, "status": "queued"}

@app.get("/api/jobs/{job_id}")
def job_status(job_id):
    job = q.fetch_job(job_id)
    if job.is_finished:
        return {"status": "done", "result": job.result}
    if job.is_failed:
        return {"status": "failed", "error": str(job.exc_info)}
    return {"status": "running"}

Trade-offs

Unblocks request threads
Handles long jobs gracefully
Adds infra (queue, workers)
UX needs polling or push

Production checklist

Idempotent jobs
Per-job timeout and retry
Dead letter queue
Worker autoscaling
Webhook or SSE for completion
Cost accounting per job