Retrieval patterns
Retrieval is where RAG systems win or lose. Basic vector search is a good start, but real questions are short, vague, full of product codes, or phrased differently from the documents. This lesson covers the patterns production systems use to retrieve the right chunks — hybrid search, reranking, query rewriting, and diversity — and how to measure retrieval quality so you know which patterns actually help.
Top-k and similarity thresholds
The number of chunks returned for each query. Too small and you miss the answer; too large and you add noise, cost, and latency to the generation step. Typical values are 3–10 after reranking, often 20–50 before it.
A minimum similarity threshold drops weak matches. If nothing passes it, the system should say it does not know rather than answer from irrelevant text. Thresholds are model-specific, so set them by looking at scores for relevant and irrelevant queries in your evaluation set.
Keyword search: BM25
The classic ranking function used by search engines. It scores a document higher when it contains the query's words, especially rare words (inverse document frequency), with diminishing returns for repetition and a correction for document length.
Vector search understands paraphrases; keyword search nails exact terms — error codes, product SKUs, names, acronyms — that embeddings often blur. Here is a compact BM25 implementation:
import math
import re
from collections import Counter
DOCS = {
"d1": "Error E-504 means the backup server timed out. Retry after five minutes.",
"d2": "If a backup fails, check your internet connection and retry.",
"d3": "Annual plans can be refunded within 30 days of purchase.",
"d4": "Restore files from the dashboard under Backups and Restore.",
}
def tokenize(text: str) -> list[str]:
return re.findall(r"[a-z0-9-]+", text.lower())
class BM25:
def __init__(self, docs: dict[str, str], k1: float = 1.5, b: float = 0.75):
self.ids = list(docs)
self.tokens = [tokenize(docs[i]) for i in self.ids]
self.avg_len = sum(map(len, self.tokens)) / len(self.tokens)
self.df = Counter(term for toks in self.tokens for term in set(toks))
self.n, self.k1, self.b = len(self.ids), k1, b
def idf(self, term: str) -> float:
df = self.df.get(term, 0)
return math.log(1 + (self.n - df + 0.5) / (df + 0.5))
def search(self, query: str, k: int = 3) -> list[tuple[str, float]]:
scores = []
for doc_id, toks in zip(self.ids, self.tokens):
tf = Counter(toks)
s = 0.0
for term in tokenize(query):
if term in tf:
num = tf[term] * (self.k1 + 1)
den = tf[term] + self.k1 * (1 - self.b + self.b * len(toks) / self.avg_len)
s += self.idf(term) * num / den
if s > 0:
scores.append((doc_id, round(s, 3)))
return sorted(scores, key=lambda x: -x[1])[:k]
bm25 = BM25(DOCS)
print(bm25.search("error e-504"))
print(bm25.search("backup failed retry"))The exact code e-504 is rare, so BM25 ranks d1 first with a large margin. In production, use a library such as rank-bm25, or the keyword search built into Postgres, Elasticsearch, or your vector database.
Hybrid search with reciprocal rank fusion
Running keyword search and vector search for the same query and merging the two result lists, getting the precision of exact terms and the recall of semantic matching.
A simple, robust way to merge ranked lists: each document scores 1 / (k + rank) in each list where it appears (k is usually 60), and the scores are summed. It uses only ranks, so it needs no tuning of incompatible score scales.
def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
scores: dict[str, float] = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1 / (k + rank)
return sorted(((d, round(s, 4)) for d, s in scores.items()), key=lambda x: -x[1])
keyword_results = ["d1", "d2", "d4"] # from BM25
vector_results = ["d2", "d4", "d1"] # from embeddings
print(rrf([keyword_results, vector_results]))d2 ranks highly in both lists, so it wins overall; d1, first in keyword search, stays near the top. Hybrid search is one of the most reliable single improvements you can make to a RAG system.
Reranking
A model that reads the query and one candidate chunk together and outputs a relevance score. Because it sees both texts at once, it is far more accurate than comparing two separately computed embeddings — but too slow to run over the whole corpus, so it re-scores only the top 20–50 candidates.
question ──► hybrid search ──► top 30 candidates ──► reranker ──► top 5 ──► LLM
(fast, broad recall) (slow, precise)from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
question = "How do I fix error E-504?"
candidates = [
"If a backup fails, check your internet connection and retry.",
"Error E-504 means the backup server timed out. Retry after five minutes.",
"Restore files from the dashboard under Backups and Restore.",
]
scores = reranker.predict([(question, c) for c in candidates])
for score, text in sorted(zip(scores, candidates), reverse=True):
print(f"{score:6.2f} {text}")Hosted rerankers (for example Cohere Rerank, Voyage, or Jina) offer the same capability via an API. Scores are on the model's own scale; only their order matters.
Query transformation
| Technique | How it works | Helps when |
|---|---|---|
| Query rewriting | An LLM rewrites a conversational question into a standalone search query using the chat history. | Follow-ups like “and for monthly plans?” |
| Multi-query | Generate 3–5 differently phrased queries, retrieve for each, merge with RRF. | Vague or ambiguous questions |
| HyDE | Ask an LLM to write a hypothetical answer, embed that, and search with it. | Short questions vs long technical passages |
| Decomposition | Split a complex question into sub-questions, retrieve for each. | “Compare the refund policies of Vault and Sync” |
Rewrite the user's latest message as a standalone search query.
Use the conversation for context. Output only the query.
<conversation>
user: What's the refund window for annual plans?
assistant: 30 days from purchase.
user: and for monthly ones?
</conversation>
Standalone query: -> "refund policy for monthly plans"Never embed a raw follow-up such as “and for monthly ones?” — it contains almost no searchable meaning. Query rewriting is essential for any multi-turn RAG chat.
Diversity with MMR
A selection method that picks chunks one at a time, balancing relevance to the query against similarity to chunks already chosen. It avoids filling the context with five near-duplicates of the same passage.
import hashlib
import re
import numpy as np
STOPWORDS = {"a", "an", "the", "is", "are", "be", "can", "do", "does", "i", "my", "to",
"of", "in", "on", "for", "and", "or", "what", "how", "it", "you", "your", "at"}
def words(text: str) -> list[str]:
"""Lowercase, drop stopwords, and strip a few suffixes (refunded -> refund)."""
out = []
for w in re.findall(r"[a-z0-9-]+", text.lower()):
if w in STOPWORDS:
continue
for suffix in ("ing", "ed", "s"):
if w.endswith(suffix) and len(w) > len(suffix) + 3:
w = w[: -len(suffix)]
break
out.append(w)
return out
def embed(text: str, dim: int = 256) -> np.ndarray:
"""Toy embedding: hashed bag of words, L2-normalised. Real models capture meaning."""
vec = np.zeros(dim, dtype=np.float32)
for word in words(text):
bucket = int(hashlib.md5(word.encode()).hexdigest(), 16) % dim
vec[bucket] += 1.0
norm = np.linalg.norm(vec)
return vec / norm if norm else vec
def mmr(query: str, texts: list[str], k: int = 2, lam: float = 0.5) -> list[int]:
q = embed(query)
vecs = np.stack([embed(t) for t in texts])
relevance = vecs @ q
chosen: list[int] = []
while len(chosen) < k:
best_i, best_score = None, -1e9
for i in range(len(texts)):
if i in chosen:
continue
redundancy = max((float(vecs[i] @ vecs[j]) for j in chosen), default=0.0)
score = lam * relevance[i] - (1 - lam) * redundancy
if score > best_score:
best_i, best_score = i, score
chosen.append(best_i)
return chosen
texts = [
"Backups run every night at 2 AM.",
"Backups run every night at 2 AM UTC.", # near-duplicate
"Backups are kept for 90 days.",
]
query = "backups run every night, kept how long"
top_by_relevance = list(np.argsort(-(np.stack([embed(t) for t in texts]) @ embed(query)))[:2])
print("plain top-2:", [int(i) for i in top_by_relevance])
print("MMR top-2: ", mmr(query, texts))Plain top-2 returns the two near-identical schedule chunks. MMR keeps the best one, then swaps the duplicate for the retention chunk, so the context can answer both halves of the question. A lam of 1.0 is pure relevance; lower values push harder for diversity.
Measuring retrieval
Evaluate retrieval separately from generation. Build a set of questions, each labelled with the chunk id(s) that contain the answer, and compute:
The fraction of questions for which at least one relevant chunk appears in the top k results. If the right chunk is not retrieved, the LLM cannot answer correctly.
The average of 1 / rank of the first relevant chunk (0 if absent). It rewards putting the right chunk near the top.
EVAL = [
{"q": "fix error E-504", "relevant": {"d1"}, "retrieved": ["d1", "d2", "d4"]},
{"q": "refund annual plan", "relevant": {"d3"}, "retrieved": ["d2", "d3", "d1"]},
{"q": "restore a file", "relevant": {"d4"}, "retrieved": ["d2", "d1", "d3"]},
{"q": "backup failed", "relevant": {"d2", "d1"}, "retrieved": ["d2", "d4", "d1"]},
]
def recall_at_k(eval_set, k):
return sum(bool(set(e["retrieved"][:k]) & e["relevant"]) for e in eval_set) / len(eval_set)
def mrr(eval_set):
total = 0.0
for e in eval_set:
for rank, doc_id in enumerate(e["retrieved"], start=1):
if doc_id in e["relevant"]:
total += 1 / rank
break
return total / len(eval_set)
for k in (1, 3):
print(f"recall@{k}: {recall_at_k(EVAL, k):.2f}")
print(f"MRR: {mrr(EVAL):.2f}")Recall@3 of 0.75 tells you one question in four never reaches the LLM with the right context — no prompt can fix that. Use these metrics to compare chunk sizes, embedding models, hybrid search, and reranking objectively.
- Write 15 questions about your own documents and label the relevant chunk ids.
- Measure recall@5 and MRR for vector-only search.
- Add BM25 with RRF and re-measure; then add a reranker. Keep only the changes that improve the metrics.
- Tune top-k and use a similarity threshold to support “I don't know”.
- Hybrid search (BM25 + vectors, merged with RRF) captures both exact terms and meaning.
- Two-stage retrieval — broad search, then a cross-encoder reranker — gives the best precision.
- Rewrite conversational questions into standalone queries; consider multi-query and decomposition.
- Use MMR to avoid redundant chunks, and measure retrieval with recall@k and MRR.