AI Engineer AcademyAIA

Hands-on Labs

Learn by building. Each lab has a scenario, starter code, solution hints and validation steps.

Build a Production HTTP API

Implement a POST /api/echo endpoint that validates input, returns structured output, and handles errors with proper status codes.

You're building the service that will later host your LLM endpoints. Before any AI, build a robust, validated, well-error-handled HTTP API.

HTTPAPIsJSONValidation+1

Call an LLM with Structured Output

Implement call_structured() that returns a validated Pydantic model from an LLM, with retry on validation failure.

Build the function that every downstream feature will use: a reliable, validated, retried structured LLM call.

LLM APIsStructured OutputsPydanticRetries

Document Chunking Strategies

Implement three chunking strategies and measure their impact on a sample retrieval eval.

Your RAG retrieval is poor. Before touching embeddings, experiment with chunking strategies — the #1 cause of bad RAG.

ChunkingRAGEvaluation

Embeddings & Vector Search

Implement a VectorStore with embed, insert, and search(filter) methods.

Build the retrieval layer: embed documents, store in pgvector, and query with metadata filters for access control.

EmbeddingsVector SearchpgvectorMetadata Filtering

Reranking for Retrieval Precision

Retrieve top-20 with vector search, rerank to top-5, and measure precision@5 improvement.

Your vector search recall is good but precision is poor — too many irrelevant chunks in top-5. Add a reranker.

RerankingRetrievalEvaluation

Build a ReAct Agent Loop

Implement run_agent(task, tools, max_iter) that loops reasoning + tool calling until final answer or max_iter.

Build the agent loop from scratch — no framework. Understand exactly how agents work by implementing one.

AgentsReActTool CallingAgent Loops

Add Tracing & Observability

Instrument call_llm, retrieve and tool_call with spans. Export to a trace backend.

Your LLM app is a black box in production. Add tracing so you can see every step, latency and token cost.

ObservabilityTracingOpenTelemetry

Build an Evaluation Pipeline

Implement run_eval(golden_set, prompt_version) -> Report with faithfulness and answer_relevance scores.

You're shipping prompt changes weekly and regressions keep slipping through. Build an eval pipeline that gates deploys.

EvaluationLLM-as-JudgeCI/CD