Skip to content

Learn / Systems

Retrieval-augmented generation

Giving a language model the right documents at the right moment: the full pipeline from ingestion to evaluation.

Jump to the lab

On this page
  1. Intuition
  2. What the data looks like
  3. The pipeline
  4. How the data is prepared
  5. The maths
  6. Real-world examples
  7. Lab
  8. In Python
  9. Pitfalls

Intuition

A language model knows what it saw in training and nothing about your internal policies, last week’s price list or a customer’s contract. Retrieval-augmented generation (RAG) fixes that without retraining: find the passages that are relevant to the question, paste them into the prompt, and ask the model to answer from those passages only, citing them.

The model supplies the language. Your documents supply the facts. Most of the quality and most of the failures come from the retrieval half.

What the data looks like

A RAG system works on a table of chunks: small pieces of your documents, each with its source and an embedding.

Knowledge base: one row per chunk One row = one chunk of a document. Answer: none; at question time the system ranks chunks by relevance.
rowchunkinputsourceinputsectioninputtextoutputembedding
1faq.mdCards and limitsEvery debit card has a daily ATM limit of 5,000 lei…[0.02, -0.11, …]
2faq.mdTransfersInternational transfers settle within two business days…[-0.08, 0.04, …]
3faq.mdBranchesThe Orbit City branch is open on weekdays from 09:00…[0.05, 0.09, …]

Documents are split into chunks, each stored with its source and an embedding. A question is embedded the same way and matched against this table.

Example rows are invented for illustration.

The pipeline

  1. Stage 01

    Ingest

    Collect the source documents: PDFs, wiki pages, policies, tickets. Record where each came from and who may see it.

  2. Stage 02

    Parse & clean

    Extract plain text, keep headings and tables, drop navigation, footers and repeated boilerplate.

  3. Stage 03

    Chunk

    Split documents into passages. Fixed-size windows are simple; recursive and sentence-aware splitting respect the structure. Overlap keeps sentences that straddle a boundary retrievable.

  4. Stage 04

    Embed

    Turn every chunk into a vector with an embedding model, so meaning can be compared numerically.

  5. Stage 05

    Index

    Store vectors in a vector index and the text in a keyword index, with metadata for filtering.

  6. Stage 06

    Query transformation

    Optionally rewrite the question, expand it into several, or generate a hypothetical answer (HyDE) and search with that.

  7. Stage 07

    Retrieve

    Dense search finds chunks with similar meaning; BM25 finds exact terms such as product codes. Hybrid search fuses both rankings with reciprocal rank fusion.

  8. Stage 08

    Rerank

    A cross-encoder reads the question and each candidate together and re-orders the top results far more precisely than retrieval can.

  9. Stage 09

    Assemble the prompt

    Put the instructions, the best chunks with their ids, and the question into one prompt that fits the token budget.

  10. Stage 10

    Generate

    The language model answers using only the supplied context and cites the chunks it used.

  11. Stage 11

    Evaluate

    Measure retrieval (recall@k, MRR) and answers (faithfulness to the context, relevance) on a fixed question set, before and after every change.

How the data is prepared

  • Parsing decides everything downstream: a table flattened into word soup cannot be retrieved.
  • Chunk size is a trade-off. Small chunks are precise but lose context; large chunks keep context but dilute the embedding and waste the token budget.
  • Overlap (10-20%) protects sentences that straddle a boundary.
  • Metadata (source, section, date, permissions) enables filtering (“only current policies”) and citations.
  • Deduplicate so five copies of one paragraph do not fill the whole top k.

The maths

BM25 scores a chunk dd for a query qq by rewarding rare terms that occur often in a short chunk:

BM25(q,d)=∑t∈qIDF(t)⋅f(t,d) (k1+1)f(t,d)+k1(1−b+b ∣d∣avgdl)\mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{f(t, d)\,(k_1 + 1)}{f(t, d) + k_1 \left(1 - b + b \, \frac{|d|}{\mathrm{avgdl}}\right)}

Reciprocal rank fusion merges rankings from different retrievers without needing comparable scores:

RRF(d)=∑r∈rankers1k+rankr(d),k=60\mathrm{RRF}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \mathrm{rank}_r(d)}, \qquad k = 60

Retrieval metrics over a set of questions QQ, where ranki\mathrm{rank}_i is the position of the first relevant chunk:

recall@k=∣{ i:ranki≤k }∣∣Q∣MRR=1∣Q∣∑i1ranki\text{recall@}k = \frac{|\{\, i : \mathrm{rank}_i \le k \,\}|}{|Q|} \qquad \mathrm{MRR} = \frac{1}{|Q|} \sum_{i} \frac{1}{\mathrm{rank}_i}

Real-world examples

  • Internal policy assistant: employees ask in plain language and get an answer with a link to the paragraph it came from.
  • Branch and support assistant: front-line staff look up procedures, fees and limits while the customer is in front of them.
  • Docs Q&A: product and API documentation that answers questions instead of returning ten links.

Lab

Walk through the pipeline one step at a time, or press Run all. Chunking, embedding, retrieval and reranking run in your browser. The last step, generation, calls a small language model on my own machine; when it is offline you get a clearly labelled extractive answer instead.

1 · Ingest

2,069 characters · about 518 tokens. Real pipelines also parse PDFs/HTML and strip boilerplate here.

In Python

from langchain_text_splitters import RecursiveCharacterTextSplitter
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder, SentenceTransformer

chunks = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=40).split_text(document)

embedder = SentenceTransformer("all-MiniLM-L6-v2")
E = embedder.encode(chunks, normalize_embeddings=True)
bm25 = BM25Okapi([c.lower().split() for c in chunks])

def retrieve(question, k=5):
    dense = (E @ embedder.encode(question, normalize_embeddings=True)).argsort()[::-1]
    sparse = bm25.get_scores(question.lower().split()).argsort()[::-1]
    fused = {}
    for ranking in (dense, sparse):                      # reciprocal rank fusion
        for pos, i in enumerate(ranking):
            fused[i] = fused.get(i, 0) + 1 / (60 + pos + 1)
    return sorted(fused, key=fused.get, reverse=True)[:k]

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
top = retrieve(question)
order = reranker.predict([(question, chunks[i]) for i in top]).argsort()[::-1]
context = [chunks[top[i]] for i in order[:3]]

Pitfalls

  • Tuning the prompt when retrieval is the problem. If the right chunk is not in the context, no prompt will fix the answer. Measure recall@k first.
  • No evaluation set. Twenty real questions with known answers beat any amount of eyeballing.
  • Dense search alone. It misses exact identifiers such as product codes, article numbers and names. Hybrid search covers both.
  • Stuffing the context. More chunks means more noise and cost; rerank and keep the best few.
  • Answers without citations. If users cannot check the source, they cannot trust the answer. Neither can you.
  • Ignoring permissions. Retrieval must respect who is allowed to read each document.