How RAG Actually Works on Azure: Architecture, Phases, and the Prompt That Stops Hallucination
Retrieval Augmented Generation (RAG) is now a standard pattern for enterprise AI applications — but most explanations skip the details that actually matter when you implement it. This post covers the two phases of a RAG system, the five steps inside each phase, and the one system prompt design decision that determines whether your RAG system hallucinates or not. All examples use Azure-native services.
The Two Phases of RAG
Every RAG system has two distinct operational phases: indexing and retrieval and generation. Confusing these two phases is the most common cause of RAG implementation failures.
Phase 1: Indexing (The Offline Phase)
Indexing happens before any user query is processed. Its job is to prepare your knowledge corpus for fast, accurate retrieval.
- Step 1 — Document ingestion: Source documents (PDFs, Word files, SharePoint pages, database records) are loaded into the pipeline. On Azure this typically uses Azure Data Factory or the Foundry ingestion SDK.
- Step 2 — Chunking: Documents are split into overlapping chunks. Chunk size and overlap are critical tuning parameters — too small loses context, too large dilutes relevance. Typical starting point: 512 tokens with 10% overlap.
- Step 3 — Embedding generation: Each chunk is converted to a vector embedding using Azure OpenAI's embedding models (text-embedding-3-large is the current standard). The embedding captures semantic meaning, not just keywords.
- Step 4 — Index storage: Embeddings and original chunk text are stored in Azure AI Search with vector search enabled. AI Search handles both traditional keyword search and vector similarity search.
- Step 5 — Metadata enrichment: Each chunk is tagged with metadata (source document, section, date, access level) to enable filtered retrieval and access control at query time.
Phase 2: Retrieval and Generation (The Online Phase)
This phase runs on every user query.
- Step 1 — Query embedding: The user's question is converted to an embedding using the same model used during indexing. Consistency is critical — different embedding models produce incompatible vector spaces.
- Step 2 — Hybrid search: Azure AI Search runs both vector similarity search and keyword search simultaneously, then merges results using Reciprocal Rank Fusion (RRF). Hybrid almost always outperforms pure vector or pure keyword search.
- Step 3 — Reranking: Retrieved chunks are reranked using Azure AI Search's semantic ranker. This applies a cross-encoder model to reorder results by true semantic relevance rather than initial retrieval score.
- Step 4 — Context assembly: Top-ranked chunks are assembled into the prompt context. Include source metadata with each chunk — the model needs this for citation generation.
- Step 5 — Generation with grounded prompt: The assembled context and user query are sent to Azure OpenAI with a carefully designed system prompt.
The System Prompt That Stops Hallucination
The single most important hallucination control in a RAG system is the system prompt instruction pattern. The critical element: explicitly instruct the model to answer only from the provided context, and to say it does not know if the context does not contain the answer. Without this instruction, frontier models will supplement retrieved context with their parametric knowledge — which is exactly the hallucination risk RAG is supposed to eliminate.
A production system prompt for RAG should include: a grounding instruction (answer only from context), a citation instruction (cite source metadata for each claim), and a fallback instruction (acknowledge when context is insufficient rather than inferring).
The Azure Foundry RAG Pattern
Azure AI Foundry now ships a native RAG pattern that wires Azure AI Search, Azure OpenAI, and the orchestration layer together with built-in evaluation tooling. For new projects, starting with the Foundry RAG pattern and customising from there is faster than building the pipeline from scratch.
RAG quality is determined by indexing quality, retrieval precision, and prompt grounding discipline — not by model capability alone. Get the architecture right first.
