Learn / Systems
Retrieval-augmented generation
Giving a language model the right documents at the right moment: the full pipeline from ingestion to evaluation.
On this page
Intuition
A language model knows what it saw in training and nothing about your internal policies, last week’s price list or a customer’s contract. Retrieval-augmented generation (RAG) fixes that without retraining: find the passages that are relevant to the question, paste them into the prompt, and ask the model to answer from those passages only, citing them.
The model supplies the language. Your documents supply the facts. Most of the quality and most of the failures come from the retrieval half.
What the data looks like
A RAG system works on a table of chunks: small pieces of your documents, each with its source and an embedding.
| rowchunk | inputsource | inputsection | inputtext | outputembedding |
|---|---|---|---|---|
| 1 | faq.md | Cards and limits | Every debit card has a daily ATM limit of 5,000 lei… | [0.02, -0.11, …] |
| 2 | faq.md | Transfers | International transfers settle within two business days… | [-0.08, 0.04, …] |
| 3 | faq.md | Branches | The Orbit City branch is open on weekdays from 09:00… | [0.05, 0.09, …] |
Documents are split into chunks, each stored with its source and an embedding. A question is embedded the same way and matched against this table.
Example rows are invented for illustration.
The pipeline
- Stage 01
Ingest
Collect the source documents: PDFs, wiki pages, policies, tickets. Record where each came from and who may see it.
- Stage 02
Parse & clean
Extract plain text, keep headings and tables, drop navigation, footers and repeated boilerplate.
- Stage 03
Chunk
Split documents into passages. Fixed-size windows are simple; recursive and sentence-aware splitting respect the structure. Overlap keeps sentences that straddle a boundary retrievable.
- Stage 04
Embed
Turn every chunk into a vector with an embedding model, so meaning can be compared numerically.
- Stage 05
Index
Store vectors in a vector index and the text in a keyword index, with metadata for filtering.
- Stage 06
Query transformation
Optionally rewrite the question, expand it into several, or generate a hypothetical answer (HyDE) and search with that.
- Stage 07
Retrieve
Dense search finds chunks with similar meaning; BM25 finds exact terms such as product codes. Hybrid search fuses both rankings with reciprocal rank fusion.
- Stage 08
Rerank
A cross-encoder reads the question and each candidate together and re-orders the top results far more precisely than retrieval can.
- Stage 09
Assemble the prompt
Put the instructions, the best chunks with their ids, and the question into one prompt that fits the token budget.
- Stage 10
Generate
The language model answers using only the supplied context and cites the chunks it used.
- Stage 11
Evaluate
Measure retrieval (recall@k, MRR) and answers (faithfulness to the context, relevance) on a fixed question set, before and after every change.
How the data is prepared
- Parsing decides everything downstream: a table flattened into word soup cannot be retrieved.
- Chunk size is a trade-off. Small chunks are precise but lose context; large chunks keep context but dilute the embedding and waste the token budget.
- Overlap (10-20%) protects sentences that straddle a boundary.
- Metadata (source, section, date, permissions) enables filtering (“only current policies”) and citations.
- Deduplicate so five copies of one paragraph do not fill the whole top k.
The maths
BM25 scores a chunk for a query by rewarding rare terms that occur often in a short chunk:
Reciprocal rank fusion merges rankings from different retrievers without needing comparable scores:
Retrieval metrics over a set of questions , where is the position of the first relevant chunk:
Real-world examples
- Internal policy assistant: employees ask in plain language and get an answer with a link to the paragraph it came from.
- Branch and support assistant: front-line staff look up procedures, fees and limits while the customer is in front of them.
- Docs Q&A: product and API documentation that answers questions instead of returning ten links.
Lab
Walk through the pipeline one step at a time, or press Run all. Chunking, embedding, retrieval and reranking run in your browser. The last step, generation, calls a small language model on my own machine; when it is offline you get a clearly labelled extractive answer instead.
In Python
from langchain_text_splitters import RecursiveCharacterTextSplitter
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder, SentenceTransformer
chunks = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=40).split_text(document)
embedder = SentenceTransformer("all-MiniLM-L6-v2")
E = embedder.encode(chunks, normalize_embeddings=True)
bm25 = BM25Okapi([c.lower().split() for c in chunks])
def retrieve(question, k=5):
dense = (E @ embedder.encode(question, normalize_embeddings=True)).argsort()[::-1]
sparse = bm25.get_scores(question.lower().split()).argsort()[::-1]
fused = {}
for ranking in (dense, sparse): # reciprocal rank fusion
for pos, i in enumerate(ranking):
fused[i] = fused.get(i, 0) + 1 / (60 + pos + 1)
return sorted(fused, key=fused.get, reverse=True)[:k]
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
top = retrieve(question)
order = reranker.predict([(question, chunks[i]) for i in top]).argsort()[::-1]
context = [chunks[top[i]] for i in order[:3]]
Pitfalls
- Tuning the prompt when retrieval is the problem. If the right chunk is not in the context, no prompt will fix the answer. Measure recall@k first.
- No evaluation set. Twenty real questions with known answers beat any amount of eyeballing.
- Dense search alone. It misses exact identifiers such as product codes, article numbers and names. Hybrid search covers both.
- Stuffing the context. More chunks means more noise and cost; rerank and keep the best few.
- Answers without citations. If users cannot check the source, they cannot trust the answer. Neither can you.
- Ignoring permissions. Retrieval must respect who is allowed to read each document.