VLM (Vision-Language Model)
An LLM extended to process images alongside text — GPT-4o, Claude 3.5, Gemini.
Simple explanation
A VLM is an LLM that can see images. It converts images into visual tokens and reasons over them with text. VLMs can describe images, do OCR, read charts and answer questions about visual content.
Engineering explanation
VLMs encode images into patches/visual tokens interleaved with text tokens. Production failure modes: imprecise on numbers (misreads 4.2M as 42M), bad at small text (use OCR fallback), bad at handwriting. Mitigations: detail:'high' for documents, two-pass numeric verification, OCR fallback, confidence scoring with human review.
Example
Invoice image → VLM extracts {invoice_number, total_amount, line_items} as structured JSON. Verify total_amount by re-asking.
When to use it
Document AI (invoices, claims, forms), chart reading, diagram understanding, multimodal assistants.
Common misconception
VLMs are NOT reliable on numbers. They confidently misread charts, decimals and large numbers. Always verify numerics with a second pass or OCR.