Vision & Image Understanding
How vision-language models see images: tokens, patches, resolution, OCR, chart reading and the failure modes of VLMs.
What is vision-language understanding?
Vision-Language Models (VLMs) extend LLMs to process images alongside text. The model converts an image into a sequence of visual tokens (patches or embeddings) that are interleaved with text tokens, allowing the model to reason about both. Modern VLMs (GPT-4o, Claude 3.5 Sonnet, Gemini) can describe images, answer questions about them, extract text (OCR), read charts and reason about diagrams.
The key engineering insight: images are NOT 'understood' the way humans understand them. The model samples the image at a fixed resolution, converts regions to tokens, and reasons over those tokens. Small text, fine details, and precise numeric values are often misread — this is the #1 production failure mode.
Why vision matters for AI engineers
A huge fraction of enterprise documents are images: scanned PDFs, screenshots, diagrams, charts, photos of forms. A text-only RAG system cannot answer questions about these. Vision unlocks document AI (insurance claims, medical imaging triage, manufacturing defect detection, retail product photos). But VLMs are confidently wrong on numbers — engineers who treat them as magic get burned in production.
Where VLMs fit
Image → resize/crop to model's native resolution → encode to visual tokens → interleave with text prompt → VLM → structured output (description, extracted fields, answer). In production: pre-process (rotate, denoise, OCR fallback for small text), VLM call, output validation, and a verification pass for numeric values.
Structured image understanding
def describe_image(image_url: str) -> str:
r = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": [
{"type": "text", "text": "What does this image show?"},
{"type": "image_url", "image_url": {"url": image_url}},
]}],
)
return r.choices[0].message.content
# Problems: free-text output, no validation, no numeric verification, no fallback for small textExperiment: VLM failure modes
See how VLMs fail on different image types and what mitigations help.
What to observe
VLMs excel at visual reasoning (diagrams, layout, object identification) but are unreliable on precise numbers and small text. Mitigations: 'detail: high' for resolution, two-pass verification for numbers, OCR fallback for small text. For high-stakes extraction, never trust a single VLM pass — verify or use OCR.
Production VLM systems
Production vision: pre-process images (rotate, denoise, upscale), use 'detail: high' for documents, two-pass verification for numeric values, OCR fallback when VLM confidence is low, structured output with confidence scores, and human review for low-confidence extractions. Cost: vision tokens are expensive — cache aggressively and resize images before sending.
Challenge
Your VLM-powered invoice extractor works 92% of the time but the 8% failures are on invoices from a specific vendor with unusual formatting. How do you detect and handle this without retraining? (See the 'Vision Model Misreads Charts' challenge.)
Production checklist
Production checklist
0 of 8 checked
Knowledge check
Your VLM reads $4.2M as $42M from a chart. What's the most effective fix?
Complete
You can now build reliable vision systems with VLMs. Next: audio processing — STT, TTS and voice-first applications.
Mark this chapter as complete
Track your progress and unlock the next chapter.