Local & Open-Source Models
Ollama, LM Studio, Hugging Face, Llama, Mistral, Gemma, Qwen — when and how to self-host models.
Local & open-source models
Open-source models (Llama, Mistral, Gemma, Qwen, Phi) can be downloaded and run locally or on your own infrastructure. Tools like Ollama (one-command local model runner), LM Studio (GUI), vLLM (high-throughput serving), and Hugging Face Transformers (programmatic) make this accessible.
The trade-off: open-source models are typically 6-18 months behind frontier closed models (GPT-4o, Claude 3.5 Sonnet) on capability, but offer: data privacy (no data leaves your env), cost predictability (no per-token fees), customisation (fine-tuning), and air-gapped deployment. The gap is closing — Llama 3.1 405B rivals GPT-4 on many benchmarks.
Why local models matter
Three scenarios demand local models: (1) Compliance — HIPAA, FedRAMP, EU data residency may prohibit sending data to OpenAI. (2) Cost — at high volume (>1M tokens/day), self-hosted is cheaper. (3) Latency/autonomy — local models have no network dependency and no rate limits. The downside: you operate the infra (GPU provisioning, model serving, updates, monitoring).
Local model stack
Model weights (Hugging Face) → serving runtime (vLLM / TGI / Ollama) → OpenAI-compatible API → your app. vLLM is the production serving choice (PagedAttention, continuous batching, high throughput). Ollama is the dev/laptop choice. TGI (Text Generation Inference, by Hugging Face) is the middle ground. All expose OpenAI-compatible APIs so your app code doesn't change.
Running local models
# Ollama — one command to run locally
# Terminal: ollama run llama3.1:8b
# Then use the OpenAI-compatible endpoint:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "What is RAG?"}],
)
print(r.choices[0].message.content)
# Works great for dev — but Ollama isn't built for production throughputExperiment: local vs cloud model choice
See when local wins and when cloud wins.
What to observe
Local wins on: data privacy, air-gapped, latency, and cost at very high volume (>5M tokens/day). Cloud wins on: capability (frontier models are 6-18 months ahead), cost at low-medium volume, and zero ops. Most production systems use BOTH — local for privacy/latency-sensitive traffic, cloud for complex reasoning. The OpenAI-compatible API means your code is identical.
Production local deployment
Production local: vLLM with continuous batching, GPU autoscaling, model versioning (pin weights), monitoring (throughput, latency, GPU utilisation), OpenAI-compatible API for code portability, and a cloud fallback for reliability. Expect 6-18 months behind frontier on capability — evaluate Llama 3.1 70B against your eval set before committing.
Challenge
Your startup needs to process 2M tokens/day of customer support transcripts. GPT-4o-mini costs $60/day. Is self-hosting Llama 3.1 70B cheaper? What's the break-even and what hidden costs exist? (Hint: GPU rental + ops + eval gap + fallback.)
Production checklist
Production checklist
0 of 10 checked
Knowledge check
When is self-hosting Llama 3.1 70B cheaper than GPT-4o-mini?
Complete
You can now choose and deploy local/open-source models. This completes the Building LLM Applications series.
Mark this chapter as complete
Track your progress and unlock the next chapter.