LLM Fundamentals
Tokens, context windows, model APIs, pricing and the mental model every AI engineer needs.
What is an LLM?
A Large Language Model is a neural network trained to predict the next token given a sequence of tokens. Through next-token prediction at massive scale, models learn language, facts, reasoning patterns and instruction-following.
As an engineer, you don't train LLMs — you use them via APIs. The key concepts: tokens (the unit of text the model sees), context window (max tokens the model can attend to), parameters (temperature, top_p, max_tokens), and pricing (per input and output token).
Why fundamentals matter
Engineers who skip fundamentals burn budget on oversized prompts, hit context-window errors in production, and cannot reason about latency or cost. Understanding tokens, context windows and pricing is the difference between a $0.02/query feature and a $2/query disaster.
Where the LLM sits
Your app → API call (messages + params) → LLM provider (inference) → token stream → your app. The LLM is stateless — every call sends the full conversation. State is YOUR responsibility. Token cost = input tokens (prompt) + output tokens (generation). Both are priced.
A complete LLM call
from openai import OpenAI
import tiktoken
client = OpenAI()
def count_tokens(text: str, model: str = "gpt-4o-mini") -> int:
enc = tiktoken.encoding_for_model(model)
return len(enc.encode(text))
def call_llm(messages: list[dict], model: str = "gpt-4o-mini", temperature: float = 0.7, max_tokens: int = 500) -> dict:
"""Call the LLM with token counting and cost estimation."""
# Count input tokens
input_text = " ".join(m["content"] for m in messages)
input_tokens = count_tokens(input_text, model)
response = client.chat.completions.create(
model=model,
messages=messages,
temperature=temperature,
max_tokens=max_tokens,
)
output = response.choices[0].message.content
output_tokens = response.usage.completion_tokens
# Cost (per 1M tokens, illustrative)
pricing = {
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
"gpt-4o": {"input": 2.50, "output": 10.00},
}
cost = (input_tokens * pricing[model]["input"] + output_tokens * pricing[model]["output"]) / 1_000_000
return {
"output": output,
"input_tokens": input_tokens,
"output_tokens": output_tokens,
"cost_usd": cost,
"model": model,
"latency_ms": response.model_dump().get("latency_ms", 0),
}
result = call_llm([
{"role": "system", "content": "You are a helpful assistant. Be concise."},
{"role": "user", "content": "What is RAG in one sentence?"},
])
print(result["output"])
print(f"Cost: " + str(result['cost_usd']) + " (USD)")Experiment: model & temperature
See how model choice and temperature affect output, cost and latency.
What to observe
gpt-4o-mini is ~17x cheaper and ~2x faster than gpt-4o. Most queries can use mini. Temperature 0 = deterministic (good for extraction and tests); higher = more variance (creative only). Cost scales with BOTH input and output tokens — long prompts are expensive.
Production considerations
Always count tokens and estimate cost per call. Pin model versions (not 'latest'). Default to the cheapest model that works, with fallback. Set max_tokens to prevent runaway generations. Stream for UX. Monitor cost per user and per feature — set budgets.
Challenge
Your feature costs $200/day on 10k calls. Average prompt is 2,000 tokens, average output 800 tokens, using gpt-4o. Cut cost by 80% without hurting quality. (Hint: model routing + prompt compression + caching.)
Production checklist
Production checklist
0 of 8 checked
Knowledge check
Your chatbot sends the full 20-message conversation history on every turn. Cost is climbing. What's the most effective first fix?
Complete
You now understand tokens, context windows, pricing and the LLM call. Next: prompt engineering — getting reliable, structured outputs.
Mark this chapter as complete
Track your progress and unlock the next chapter.