Building RAG Systems

Retrieval-augmented generation (RAG) lets a language model answer questions using your own documents — product manuals, policies, wikis, tickets — instead of only what it memorised during training. This course builds a RAG system piece by piece, explains every design decision, and finishes with a working “chat with your documents” application that cites its sources.

Last updated: 01 Nov, 2026

What is RAG?

Definition · Retrieval-augmented generation (RAG)

An architecture that, for each question, first retrieves the most relevant passages from a knowledge base and then asks a language model to generate an answer using those passages as context — ideally citing them.

LLMs are powerful but have three gaps that matter in business: they do not know your private data, their knowledge stops at a training cutoff, and they hallucinate when they do not know. RAG closes all three by putting the right facts in front of the model at the moment it answers. It is the most widely deployed LLM architecture in industry today.

The two pipelines

RAG architecture

INDEXING (offline, whenever documents change)
  documents ──► load & clean ──► chunk ──► embed ──► vector store (+ metadata)

QUERYING (online, for every question)
  question ──► embed ──► search vector store ──► top-k chunks
                                                    │
                     prompt = instructions + chunks + question
                                                    │
                                                    ▼
                                 LLM ──► answer with citations

What you will learn

Lesson	What you learn
What is RAG	Why RAG exists, how it compares with fine-tuning and long context, and a complete minimal RAG in plain Python.
Loading and chunking	Loading files, cleaning text, chunking strategies, sizes and overlap, and metadata.
Embeddings and similarity	What embeddings are, embedding APIs and local models, cosine similarity, and choosing a model.
Vector stores	Building an in-memory vector store, approximate nearest-neighbour indexes, Chroma, pgvector, and FAISS.
Retrieval patterns	Top-k and thresholds, hybrid keyword + vector search, reranking, query rewriting, MMR, and retrieval metrics.
Generation with sources	Grounded prompts, citations, “I don't know” behaviour, context budgets, and faithfulness checks.
Project: chat with documents	A complete, runnable document Q&A application tying everything together.

Prerequisites

Python with lists, dicts, files, and NumPy basics (Python for AI Developers).
Tokens, messages, and API calls (LLM Fundamentals).
Helpful: system prompts and structured outputs (Prompt Engineering).

Terminal

python -m pip install numpy openai python-dotenv
# optional, used in later lessons
python -m pip install chromadb sentence-transformers rank-bm25

Note

To keep every example runnable offline and reproducible, the course uses a small toy embedding function (a hashed bag of words). It matches words rather than meaning, so it is weaker than a real model — but the rest of the pipeline is identical, and each lesson shows the one-line swap to a real embedding model.

Key terms

Term	Meaning
Corpus / knowledge base	The collection of documents the system can search.
Chunk	A small passage of a document, the unit that is embedded and retrieved.
Embedding	A vector of numbers representing a chunk's meaning.
Vector store	A database that stores embeddings and finds the most similar ones quickly.
Top-k	The number of chunks retrieved per question.
Hybrid search	Combining keyword search (BM25) with vector search.
Reranker	A model that re-scores retrieved chunks for relevance to the question.
Grounding / faithfulness	Whether the answer is supported by the retrieved context.

Course syllabus

Work through the lessons in order — each one adds a component to the pipeline. Your progress is saved in this browser.

Syllabus