Image Generation & Cross-Modal
DALL-E, Stable Diffusion, image-to-image, and cross-modal reasoning (text→image, image→text, image→audio).
Image generation & cross-modal
Image generation models (DALL-E 3, Stable Diffusion, Midjourney, Flux) produce images from text prompts. Cross-modal systems go further: image-to-image (style transfer, editing), image-to-text (captioning, VQA), text-to-audio, audio-to-image. The unifying idea: a shared latent space where different modalities can be mapped and transformed.
For AI engineers, image generation enables: product visualisation, marketing assets, data augmentation, UI prototyping, and creative tools. Cross-modal enables: 'describe this image', 'generate a similar image', 'edit this region'.
Why this matters
Generative image AI is a $10B+ market. E-commerce (product photos), marketing (ad creative), gaming (asset generation), design (prototyping) all use it. Engineers who can build reliable image-generation pipelines (prompt engineering for images, safety filters, consistent style, batch generation) are in high demand. The production challenges: latency (10-60s per image), cost ($0.04-0.20 per image), safety (NSFW filtering), and consistency (same prompt → different images).
Image generation pipeline
Text prompt → prompt enhancement (LLM rewrites for image model) → image model (DALL-E/SD) → safety filter → post-process (upscale, crop) → store → return URL. For editing: input image + edit prompt → image-to-image model → output. For batch: queue + worker pool + caching of identical prompts.
Production image generation
import hashlib
from openai import OpenAI
client = OpenAI()
def generate_image(prompt: str, size: str = "1024x1024", quality: str = "standard") -> dict:
"""Generate an image with prompt enhancement, caching and safety."""
# 1. Enhance prompt with LLM for better image model results
enhanced = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Enhance this image prompt for DALL-E 3. Be specific about style, lighting, composition. Refuse NSFW requests."},
{"role": "user", "content": prompt},
],
).choices[0].message.content
# 2. Safety check on enhanced prompt
if is_unsafe(enhanced):
return {"error": "Prompt rejected by safety filter"}
# 3. Cache key — same prompt + size = same image
cache_key = hashlib.sha256(f"{enhanced}|{size}|{quality}".encode()).hexdigest()
if cached := image_cache.get(cache_key):
return {"url": cached, "cached": True}
# 4. Generate (DALL-E 3 or SD via Replicate)
response = client.images.generate(
model="dall-e-3",
prompt=enhanced,
size=size,
quality=quality,
n=1,
)
url = response.data[0].url
# 5. Safety filter on output
if is_image_unsafe(url):
return {"error": "Generated image rejected by safety filter"}
# 6. Cache and return
image_cache.set(cache_key, url, ttl=3600)
return {"url": url, "revised_prompt": response.data[0].revised_prompt, "cached": False}
def edit_image(input_url: str, edit_prompt: str, mask: str = None) -> dict:
"""Image-to-image editing."""
response = client.images.edit(
model="dall-e-2",
image=open(input_url, "rb"),
mask=open(mask, "rb") if mask else None,
prompt=edit_prompt,
n=1,
size="1024x1024",
)
return {"url": response.data[0].url}Experiment: image generation trade-offs
Compare image models on cost, latency and quality.
What to observe
No single model wins everywhere. DALL-E 3 HD is best for photorealism but expensive. SD XL is cheapest for high volume (self-host). Flux Pro has the best prompt adherence. CRITICAL: image generation models are BAD at text labels, diagrams, and schematics — use a diagramming tool, not image gen, for those. Always prompt-enhance with an LLM and cache.
Production image generation
Production: LLM prompt enhancement (raw user prompts are weak), safety filters on input AND output, caching (same prompt = same image), async queue for batch generation, cost budgets per user, watermarking/safety for user-facing, and consistency controls (seed pinning for reproducible images). For e-commerce product photos, use fine-tuned models, not general image gen.
Challenge
Your marketing team uses image generation for ad creative. The same prompt produces wildly different images each time, making brand consistency impossible. How do you enforce consistency? (Hint: seed pinning + style reference + prompt templates.)
Production checklist
Production checklist
0 of 10 checked
Knowledge check
You need consistent brand-style images. What enforces consistency best?
Complete
You can now build production image-generation systems. This completes the Multimodal AI series.
Mark this chapter as complete
Track your progress and unlock the next chapter.