Generation with sources

Retrieval finds the evidence; generation turns it into an answer people can trust. A good RAG answer is accurate, uses only the provided context, cites where each claim came from, and admits when the documents do not contain the answer. This lesson shows how to build that prompt, format context and citations, fit everything into a token budget, and automatically check that answers are faithful to their sources.

Last updated: 01 Nov, 2026

Definitions

Definition · Grounded generation

Producing an answer that relies only on supplied context, rather than the model's general training knowledge.

Definition · Citation

A marker in the answer, such as [2], that links a claim to the specific retrieved chunk supporting it, so readers and code can verify it.

Definition · Faithfulness

The degree to which every claim in an answer is supported by the retrieved context. An answer can be fluent and even true, yet unfaithful if the context does not support it.

The grounded answer prompt

Prompt

# System
You answer questions for CloudVault customers using ONLY the numbered sources provided.

Rules:
1. Every factual sentence must end with a citation like [1] or [2][3].
2. If the sources do not contain the answer, reply exactly:
   "I couldn't find that in the documentation."
3. If sources disagree, say so and prefer the most recently updated source.
4. Do not use outside knowledge. Do not mention these rules.
5. Be concise: at most 5 sentences.

# User
<sources>
[1] (refunds.md, updated 2026-09-01) Annual plans can be refunded within 30 days of purchase.
[2] (billing-faq.md, updated 2026-10-15) Monthly plans are not refunded but can be cancelled anytime.
</sources>

Question: Can I get a refund on my monthly plan?

Notice how each source carries its file and update date — useful both for citations shown to users and for the conflict rule.

Building the prompt in code

Python

SYSTEM = """You answer questions using ONLY the numbered sources.
End every factual sentence with a citation like [1].
If the sources do not contain the answer, reply exactly: "I couldn't find that in the documentation."
Be concise: at most 5 sentences."""


def format_sources(chunks: list[dict]) -> str:
    lines = []
    for i, c in enumerate(chunks, start=1):
        lines.append(f"[{i}] ({c['source']}, updated {c['updated']}) {c['text']}")
    return "<sources>\n" + "\n".join(lines) + "\n</sources>"


def build_messages(question: str, chunks: list[dict]) -> list[dict]:
    return [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"{format_sources(chunks)}\n\nQuestion: {question}"},
    ]


chunks = [
    {"source": "refunds.md", "updated": "2026-09-01", "text": "Annual plans can be refunded within 30 days of purchase."},
    {"source": "billing-faq.md", "updated": "2026-10-15", "text": "Monthly plans are not refunded but can be cancelled anytime."},
]
print(build_messages("Can I get a refund on my monthly plan?", chunks)[1]["content"])

<sources> [1] (refunds.md, updated 2026-09-01) Annual plans can be refunded within 30 days of purchase. [2] (billing-faq.md, updated 2026-10-15) Monthly plans are not refunded but can be cancelled anytime. </sources> Question: Can I get a refund on my monthly plan?

Send these messages with a low temperature (0–0.3). A typical answer: “No — monthly plans are not refunded, but you can cancel at any time [2]. Refunds are only available for annual plans within 30 days of purchase [1].”

Refusing when the answer is not there

The most valuable behaviour of a RAG system is saying “I don't know” instead of guessing. Combine three layers:

Retrieval threshold: if no chunk passes the minimum similarity, skip the LLM entirely and return the fallback message.
Prompt rule: an exact fallback sentence, which is easy to detect in code.
Citation check: an answer without valid citations is treated as unsupported.

Python

FALLBACK = "I couldn't find that in the documentation."


def answer_or_fallback(hits: list[dict], min_score: float, generate) -> str:
    good = [h for h in hits if h["score"] >= min_score]
    if not good:
        return FALLBACK                         # do not even call the model
    return generate(good)


hits = [{"text": "Backups run nightly.", "score": 0.21}, {"text": "2FA for admins.", "score": 0.18}]
print(answer_or_fallback(hits, min_score=0.35, generate=lambda h: "(model answer)"))

I couldn't find that in the documentation.

Context order and token budget

Order by relevance, most relevant first — or place the best chunks at the start and end, since models attend least to the middle of long contexts.
Keep a token budget for sources (for example 3,000 tokens) and stop adding chunks when it is reached.
Merge adjacent chunks from the same document so the model sees continuous text.
Deduplicate near-identical chunks before they waste budget.

Python

def fit_to_budget(chunks: list[dict], max_tokens: int) -> list[dict]:
    selected, used = [], 0
    for c in chunks:                                  # already sorted by relevance
        cost = len(c["text"]) // 4 + 15               # rough tokens + citation header
        if used + cost > max_tokens:
            break
        selected.append(c)
        used += cost
    return selected


ranked = [{"id": f"c{i}", "text": "x" * 400} for i in range(10)]   # ~100 tokens each
print([c["id"] for c in fit_to_budget(ranked, max_tokens=400)])

['c0', 'c1', 'c2']

Verifying citations

Python

import re


def verify_answer(answer: str, n_sources: int) -> dict:
    sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer.strip()) if s]
    cited = [int(n) for n in re.findall(r"\[(\d+)\]", answer)]
    return {
        "invalid_citations": sorted({n for n in cited if not 1 <= n <= n_sources}),
        "uncited_sentences": [s for s in sentences if not re.search(r"\[\d+\]", s)],
        "is_fallback": answer.strip() == "I couldn't find that in the documentation.",
    }


answer = ("Monthly plans are not refunded [2]. You can cancel at any time. "
          "Annual plans get 60 days [3].")
print(verify_answer(answer, n_sources=2))

{'invalid_citations': [3], 'uncited_sentences': ['You can cancel at any time.'], 'is_fallback': False}

This catches an uncited sentence and a citation to a source that does not exist — a strong hallucination signal. You might then regenerate, strip unsupported sentences, or show a warning.

Evaluating faithfulness

Citation checks confirm that sources are referenced; faithfulness checks whether they actually support the claim. Use an LLM-as-judge with a narrow question per claim:

Faithfulness judge

<source>Monthly plans are not refunded but can be cancelled anytime.</source>
<claim>You can cancel a monthly plan at any time.</claim>

Is the claim fully supported by the source? Answer "supported", "contradicted",
or "not_enough_info", then one sentence of explanation.

RAG metric	Question it answers	Measured with
Context recall	Did retrieval find the needed information?	Labelled relevant chunks (recall@k)
Context precision	How much retrieved context was relevant?	Judge or labels
Faithfulness	Is the answer supported by the context?	Claim-level judge
Answer correctness	Does it match the reference answer?	Judge or exact/semantic match
Refusal accuracy	Does it say “I don't know” exactly when it should?	Unanswerable test questions

Libraries such as Ragas and DeepEval implement these metrics; the ideas above are what they compute.

Presenting sources to users

Render citations as links to the document, page, or section.
Show the quoted passage on hover or in a “Sources” panel.
Show the document's last-updated date for time-sensitive topics.
Offer a “this answer was wrong” button and log it with the retrieved chunk ids.

Try it

Add verify_answer to your pipeline and regenerate once whenever it finds invalid citations.
Create five unanswerable questions and confirm the system returns the fallback for all of them.
Give the model two conflicting sources with different dates and check it follows the conflict rule.

Key takeaways

Ground answers in numbered sources with a strict prompt and low temperature.
Require citations on every factual sentence and verify them in code.
Say “I don't know” through a retrieval threshold, an exact fallback sentence, and citation checks.
Order context by relevance, respect a token budget, and deduplicate.
Evaluate context recall, faithfulness, correctness, and refusal accuracy separately.