RAG with Azure AI Search
Export PDFMost enterprise AI requests boil down to “answer from our documents, and show where the answer came from”. That is retrieval-augmented generation (RAG). This lesson explains the two ways to do RAG in Foundry — managed file search with vector stores, and Azure AI Search for production-scale knowledge — and covers indexes, hybrid search, semantic ranking, connecting search to a project, and producing cited answers.
RAG in one picture
A pattern where, for every question, the system first retrieves relevant passages from a knowledge source and then asks the model to generate an answer using only those passages, ideally with citations. It gives the model private, up-to-date facts without retraining it.
INGEST (when documents change)
PDFs, Word, HTML, SharePoint ─► extract text ─► chunk ─► embed ─► index
QUERY (every question)
question ─► (rewrite) ─► search index: keyword + vector ─► semantic rerank ─► top passages
│
instructions + passages + question ─► model ─► answer [cited]The Building RAG Systems course builds each step by hand. Here we focus on the managed Azure pieces that do the heavy lifting.
Two ways to do RAG in Foundry
| File search + vector stores | Azure AI Search | |
|---|---|---|
| What it is | A managed tool: upload files, Foundry chunks, embeds, and indexes them for you | A separate search service you control: index schema, chunking, scoring, security |
| Setup effort | Minutes | Hours to days |
| Scale | Good for hundreds to thousands of files per store | Millions of documents, many indexes |
| Control | Few knobs (chunk size, max results, filters) | Full control: fields, analyzers, vector profiles, scoring, semantic ranker |
| Security trimming | Per vector store | Document-level filters (for example, by user group) |
| Use for | Prototypes, per-user uploads, small knowledge bases | Production enterprise knowledge |
Option A: file search with a vector store
A managed container in your project that holds files and their embedded chunks. You attach it to an agent through the file search tool, and the agent searches it whenever a question needs it.
This example calls your Azure resources, so it needs a deployed project and a signed-in identity. Output will differ on every run.
from pathlib import Path
with project.get_openai_client() as client:
store = client.vector_stores.create(name="support-kb")
for path in Path("kb").glob("*.md"): # refund-policy.md, shipping-faq.md, ...
with path.open("rb") as f:
client.vector_stores.files.upload_and_poll(vector_store_id=store.id, file=f)
print(store.id, client.vector_stores.retrieve(store.id).file_counts)Behind the scenes Foundry extracts text, splits it into overlapping chunks, embeds them, and builds a hybrid (keyword + vector) index. You attach the store to an agent in the next lessons with FileSearchTool(vector_store_ids=[store.id]).
Option B: Azure AI Search
Azure's managed search engine (formerly Azure Cognitive Search). It stores indexes of documents and supports full-text keyword search (BM25), vector search, hybrid search, filters, facets, and a semantic ranker that reorders results with a language model.
Core concepts
| Concept | Meaning |
|---|---|
| Index | The searchable collection, with a schema of fields (for example id, title, content, content_vector, url, group_ids). |
| Field attributes | Whether a field is searchable, filterable, sortable, facetable, retrievable, or a vector field. |
| Data source + indexer | A connection to Blob Storage, SharePoint, SQL, or Cosmos DB, and a scheduled crawler that pulls documents into the index. |
| Skillset | Enrichment steps applied during indexing: OCR, chunking (text split), and embedding (integrated vectorization). |
| Vectorizer | Configuration that embeds the query at search time with the same model used for documents. |
| Semantic configuration | Tells the semantic ranker which fields hold the title and content. |
The easiest path to a working index is the portal's Import and vectorize data wizard: point it at a Blob container, pick your embedding deployment, and it creates the data source, skillset (chunk + embed), index, and indexer for you.
Keyword, vector, hybrid, semantic
| Search mode | How it matches | Strength | Weakness |
|---|---|---|---|
| Keyword (BM25) | Exact terms, weighted by rarity | Product codes, names, error numbers | Misses synonyms and paraphrases |
| Vector | Nearest embeddings by meaning | Paraphrases, natural questions | Can miss exact identifiers |
| Hybrid | Runs both and merges ranks with Reciprocal Rank Fusion | Best of both; the recommended default | Slightly more compute |
| Hybrid + semantic ranker | Reranks the top hybrid results with a cross-encoder language model | Highest relevance in Microsoft's benchmarks | Extra cost per query |
A way to merge several ranked lists without comparing their incompatible scores. Each document gets 1 / (k + rank) from every list it appears in (Azure AI Search uses k = 60), and the sums are sorted. Documents ranked well by both keyword and vector search rise to the top.
def rrf(rankings: dict[str, list[str]], k: int = 60) -> list[tuple[str, float]]:
"""Reciprocal Rank Fusion: score(doc) = sum over lists of 1 / (k + rank)."""
scores: dict[str, float] = {}
for ranked_ids in rankings.values():
for rank, doc_id in enumerate(ranked_ids, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1 / (k + rank)
return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
keyword_hits = ["refund-policy", "shipping-faq", "warranty", "returns-portal"]
vector_hits = ["returns-portal", "refund-policy", "store-credit", "warranty"]
for doc_id, score in rrf({"bm25": keyword_hits, "vector": vector_hits}):
print(f"{doc_id:<15} {score:.5f}")returns-portal was only fourth in keyword search but first in vector search, so fusion lifts it to second. store-credit, found only by vector search, still makes the list. This is exactly why hybrid search beats either method alone.
Query an index from Python
This example calls your Azure resources, so it needs a deployed project and a signed-in identity. Output will differ on every run.
import os
from azure.identity import DefaultAzureCredential
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery
search = SearchClient(
endpoint=os.environ["SEARCH_ENDPOINT"], # https://<name>.search.windows.net
index_name="support-kb",
credential=DefaultAzureCredential(), # needs Search Index Data Reader
)
question = "Can I get a refund if my parcel arrived damaged?"
results = search.search(
search_text=question, # keyword half
vector_queries=[VectorizableTextQuery(text=question, # vector half (index vectorizer embeds it)
k_nearest_neighbors=50,
fields="content_vector")],
query_type="semantic", # semantic reranking
semantic_configuration_name="default",
select=["title", "content", "url"],
top=5,
)
passages = [{"title": r["title"], "url": r["url"], "content": r["content"],
"score": r["@search.reranker_score"]} for r in results]Generate a cited answer
context = "\n\n".join(
f"[{i}] {p['title']} ({p['url']})\n{p['content']}" for i, p in enumerate(passages, 1)
)
answer = client.responses.create(
model=deployment,
instructions=(
"Answer only from the numbered sources. Cite sources like [1] after each claim. "
"If the sources do not contain the answer, say you don't know and suggest contacting support."
),
input=f"Sources:\n{context}\n\nQuestion: {question}",
)
print(answer.output_text)Connecting search to your project and agents
To let an agent use an index, create a connection from the project to the search service (portal: Management center → Connected resources → Azure AI Search), and give the project's managed identity Search Index Data Reader on the search service. The agent then uses the Azure AI Search tool:
from azure.ai.projects.models import (
AISearchIndexResource,
AzureAISearchTool,
AzureAISearchToolResource,
)
connection = project.connections.get("search-support") # the connection name you created
search_tool = AzureAISearchTool(
azure_ai_search=AzureAISearchToolResource(
indexes=[AISearchIndexResource(
project_connection_id=connection.id,
index_name="support-kb",
query_type="vector_semantic_hybrid",
top_k=5,
)]
)
)
# pass tools=[search_tool] when creating the agent (next lessons)Microsoft also offers Foundry IQ, a managed knowledge layer built on Azure AI Search that lets several agents share knowledge bases spanning multiple sources (Blob, SharePoint, OneLake, the web) with agentic retrieval — the system plans sub-queries, runs them in parallel, and merges results. It is a good next step once a single index is not enough.
RAG quality checklist
- Chunking: chunks of a few hundred tokens with some overlap usually work; keep headings with their text.
- Metadata: store title, URL, date, and access groups with every chunk, for citations and filtering.
- Security trimming: filter by the user's group IDs so people only retrieve what they may read.
- Freshness: schedule indexers, and delete chunks when source documents are removed.
- Grounding instructions: tell the model to answer only from sources and to admit when it does not know.
- Measure: evaluate retrieval and groundedness on a test set (lesson 9).
- Add a third list
"title_match"to the RRF example and see how rankings change. - Change
kfrom 60 to 1 and observe how much more the top rank dominates. - Create a vector store from three Markdown files of your own and note the chunk counts in the portal.
- RAG retrieves relevant passages first, then generates an answer from them with citations.
- File search with vector stores is managed and quick; Azure AI Search gives production control and scale.
- Hybrid search (BM25 + vector, merged with RRF) plus semantic ranking is the recommended default.
- Connect search to a project with a connection and give the project's identity Search Index Data Reader.
- Instruct the model to answer only from sources, cite them, and say when it doesn't know.