What Is RAG?
Retrieval-Augmented Generation (RAG) is an architecture where an AI system retrieves relevant information from a knowledge source before generating an answer. Instead of relying solely on the model's training data, the system grounds its response in your approved content.
This matters because most business AI use cases involve private, changing, or domain-specific knowledge that no model was trained on.
The Production RAG Pipeline
A production RAG system is not just "embed documents and query." It is a multi-stage pipeline where each stage affects quality:
1. Document Ingestion
Documents come in many formats — PDFs, wikis, policies, databases. The ingestion pipeline must handle parsing, structure extraction, and metadata preservation. Poor parsing at this stage degrades everything downstream.
2. Chunking Strategy
Splitting documents into chunks is one of the most impactful decisions. Too small and you lose context. Too large and retrieval precision drops. Strategies include:
- Fixed-size chunks with overlap
- Sentence or paragraph-based splitting
- Section-aware splitting that respects document structure
- Metadata-tagged chunks (source, section, access level)
3. Embeddings and Indexing
Each chunk is converted into a vector embedding and stored in a vector database. The embedding model choice affects semantic matching quality. Common options include OpenAI embeddings, open-source models, and specialized domain models.
4. Retrieval
When a user asks a question, the system converts the query into an embedding and retrieves the most similar chunks. But pure semantic retrieval has weaknesses — it can miss exact matches or struggle with rare terms.
5. Hybrid Search
Production systems combine semantic search with keyword (BM25) search. This hybrid approach catches both meaning-based matches and exact keyword matches, improving recall across query types.
6. Reranking
Retrieved candidates are reranked by a cross-encoder or reranking model. This improves precision by re-evaluating the top results with a more expensive but accurate model.
7. Context Construction
The reranked chunks are assembled into a context window for the LLM. This must fit within the model's context limit while preserving the most relevant information.
8. Answer Generation
The LLM generates an answer grounded in the retrieved context. The prompt instructs the model to use only the provided context and to cite sources.
9. Citations
The response includes references to the source material. This allows users to verify answers and builds trust in the system.
Evaluation
Production RAG needs evaluation at two levels:
- Retrieval evaluation: Are the right documents being found?
- Generation evaluation: Is the answer correct and grounded?
Without evaluation, you cannot know if your RAG system is improving or degrading over time.
Access Control
For enterprise deployments, users should only receive answers grounded in documents they are authorized to access. Access control is applied at the retrieval level — before the LLM sees the context.
When RAG Is Not the Right Choice
If a traditional database query, search, or deterministic software can solve the problem more reliably, RAG may be unnecessary. RAG adds complexity — use it when the problem demands semantic understanding of unstructured knowledge.