Audio: Speech-to-Text & Text-to-Speech
STT (Whisper, streaming ASR), TTS, voice-first applications and production audio pipelines.
Audio AI: STT and TTS
Speech-to-Text (STT) converts audio to text; Text-to-Speech (TTS) converts text to audio. Modern STT (Whisper, Deepgram, AssemblyAI) handles multiple languages, accents and background noise. Modern TTS (OpenAI TTS, ElevenLabs, PlayHT) produces near-human voices with emotion and pace control.
Voice-first applications (voice assistants, meeting transcription, accessibility, IVR) combine STT → LLM → TTS in a pipeline. Latency, accuracy in noisy environments, and voice consistency are the production challenges.
Why audio matters
Voice is the most natural human interface. Voice-first AI assistants (customer support, healthcare intake, in-car, accessibility) are a massive market. Meeting transcription, podcast indexing, and real-time translation all depend on STT. Engineers who only know text LLMs miss this entire surface area.
Voice pipeline
Audio in → VAD (voice activity detection) → STT (streaming or batch) → text → LLM → response text → TTS → audio out. For real-time: stream STT chunks, send to LLM as they arrive, stream TTS as response generates. Latency budget: STT 200-500ms, LLM 300-800ms, TTS 200-500ms — total 700-1800ms for a turn.
Streaming voice assistant
import asyncio
from openai import OpenAI
client = OpenAI()
async def voice_turn(audio_stream):
"""Streaming STT → LLM → TTS for a voice assistant turn."""
# 1. Stream audio to STT
transcript = await transcribe_streaming(audio_stream)
# 2. Send to LLM with conversation history
response_stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": transcript}],
stream=True,
)
# 3. Stream text to TTS as it generates
async def tts_stream():
buffer = ""
async for chunk in response_stream:
delta = chunk.choices[0].delta.content
if delta:
buffer += delta
# Sentence boundary — send to TTS
if any(p in buffer for p in [".", "!", "?"]):
audio = await synthesize(buffer)
yield audio
buffer = ""
if buffer:
yield await synthesize(buffer)
async for audio_chunk in tts_stream():
yield audio_chunk
async def transcribe_streaming(audio):
"""Streaming STT with VAD."""
# Whisper streaming or Deepgram streaming
result = await whisper.transcribe(audio, language="en", stream=True)
return result.text
async def synthesize(text):
"""TTS with voice selection."""
response = client.audio.speech.create(
model="tts-1",
voice="alloy",
input=text,
)
return response.contentExperiment: STT accuracy
See how STT accuracy varies with audio conditions and models.
What to observe
STT accuracy is highly environment-dependent. Clean audio = any model works. Noisy environments need large or domain-tuned models. Latency matters for real-time: Deepgram streams faster than Whisper. For production voice apps, domain-tuning on your actual audio (accent, noise, vocabulary) is the highest-accuracy, lowest-latency option.
Production voice systems
Production voice: VAD to detect speech boundaries, streaming STT for low latency, domain-tuned models for accuracy, noise suppression pre-processing, voice activity endpoints to know when the user finished speaking, TTS with consistent voice identity, and barge-in (interrupt TTS when user speaks again). Cache common responses. Measure latency end-to-end.
Challenge
Your voice assistant works great in testing but in production has 2.5s latency — too slow for natural conversation. Where do you cut 1s? (Hint: streaming STT + streaming TTS + sentence-boundary TTS chunking.)
Production checklist
Production checklist
0 of 10 checked
Knowledge check
Your voice assistant has 2.5s latency. What gives the biggest reduction?
Complete
You can now build voice-first AI applications. Next: cross-modal and image generation.
Mark this chapter as complete
Track your progress and unlock the next chapter.