Retrieval-Augmented Generation (RAG)
Definition. An architecture in which a language model's answer is conditioned on documents retrieved at query time rather than on parametric memory alone. A RAG pipeline has an offline ingestion path (parse, chunk, embed, index) and an online path (embed the query, retrieve candidates, rerank, assemble a prompt, generate, cite). It converts a closed-book model into an open-book one, which is the single most effective way to reduce hallucination on private or fast-changing knowledge.
How it works. Ingest documents with structure preserved, chunk semantically at heading boundaries, embed and index with metadata. At query time, rewrite the user question into a search query, retrieve a wide candidate set, rerank with a cross-encoder, and assemble a prompt that instructs the model to answer only from the provided passages and to cite them. Add a refusal path for when retrieval returns nothing above a similarity threshold.
Example. An internal policy assistant indexes 12,000 HR documents. A user asks whether parental leave can be taken in two blocks. The retriever returns four passages, the reranker promotes the one from the current leave policy over an archived 2019 version because the metadata filter excludes superseded documents, and the model answers with the exact clause and a link. Without the date filter the model would have cited the outdated policy, which is the most common RAG failure in practice.
Use case. RAG is the default choice for question answering over private corpora, customer support deflection, contract and compliance review, and any assistant that must cite sources. It is preferable to fine-tuning whenever the knowledge changes frequently, because updating an index is minutes of work while retraining is days.
Warnings. RAG does not eliminate hallucination; it only gives the model better material to hallucinate around. Always require citations and verify that cited passages actually support the claim. Never index documents without access control metadata, or the retriever will happily surface content the user is not entitled to see.
Limitations. RAG is only as good as retrieval. Chunking that splits a table from its header, PDFs parsed into scrambled columns, and queries that need aggregation across many documents all break it. It also cannot reason over the whole corpus at once, and multi-hop questions requiring chaining several documents need query decomposition or an agentic loop.
| Pros | Cons |
|---|---|
| Answers against a corpus that can be updated as source material changes. | Poor parsing, chunking, or retrieval can undermine the answer before generation. |