Skip to content

Learn / Representation

Embeddings

Turning text into vectors so that similar meaning becomes small distance. The foundation of semantic search and RAG.

Jump to the lab

On this page
  1. Intuition
  2. What the data looks like
  3. Word vs sentence embeddings
  4. How a transformer produces one vector
  5. How the data is prepared
  6. The maths
  7. Real-world examples
  8. Lab
  9. In Python
  10. Pitfalls

Intuition

An embedding model reads a piece of text and returns a list of numbers: a point in space. It is trained so that texts with similar meaning land close together: “I lost my card” ends up next to “my card was stolen” even though they share almost no words. Once meaning is geometry, search, clustering and deduplication all become distance calculations.

What the data looks like

The input is text, one piece per row. The output is a fixed-length list of numbers per row.

Support messages: one row per text One row = one piece of text. Answer: none; the model outputs a list of 384 numbers (the embedding).
inputmessageoutputdim_1outputdim_2outputdim_3output…outputdim_384
How do I block my card?0.021-0.1130.064…0.007
My card was stolen0.034-0.0980.071…0.011
What is the transfer fee?-0.0870.045-0.012…0.093

The input is raw text. The output is a vector; similar meanings get similar numbers, which is why the first two rows look alike.

Example rows are invented for illustration.

Word vs sentence embeddings

  • Word embeddings (word2vec, GloVe) give one fixed vector per word. “Bank” has the same vector in “river bank” and “bank account”.
  • Sentence embeddings come from a transformer that reads the whole text, so each word’s representation depends on its context. The model’s per-token vectors are then combined into one vector for the sentence.

How a transformer produces one vector

  1. The text is split into tokens.
  2. The transformer outputs one contextual vector per token.
  3. Mean pooling averages the token vectors (ignoring padding).
  4. The result is normalised to length 1.

The model in the lab below, all-MiniLM-L6-v2, does exactly this and returns 384 numbers per sentence.

How the data is prepared

  • Clean the text: strip markup, signatures and boilerplate that would dominate the vector.
  • Mind the length limit: text beyond the model’s window (256 tokens here) is silently cut off. Long documents must be chunked.
  • Use the same model for documents and queries. Vectors from different models are not comparable.
  • Check the language: an English-only model scatters translations. Try the three-languages preset in the lab.

The maths

Similarity is measured between vectors aa and bb:

cosine(a,b)=a⋅b∥a∥ ∥b∥dot(a,b)=∑iaibiL2(a,b)=∥a−b∥\text{cosine}(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert} \qquad \text{dot}(a, b) = \sum_i a_i b_i \qquad \text{L2}(a, b) = \lVert a - b \rVert

When vectors are normalised to unit length the three agree on ranking: cosine equals the dot product, and ∥a−b∥2=2−2 a⋅b\lVert a - b \rVert^2 = 2 - 2\,a \cdot b. That is why most vector databases store normalised vectors and use the cheap dot product.

Mean pooling over TT token vectors hth_t with attention mask mtm_t:

e=∑t=1Tmt ht∑t=1Tmte = \frac{\sum_{t=1}^{T} m_t \, h_t}{\sum_{t=1}^{T} m_t}

Real-world examples

  • Semantic search over policies and procedures, where users do not know the exact wording.
  • Deduplication of near-identical tickets or documents.
  • Clustering support tickets into themes without labels.
  • Retrieval for RAG: the step that finds what the language model should read.

Lab

Type your own sentences. Everything runs in your browser; the model is downloaded once (about 23 MB) when you press the button.

Sentences (one per line)

The model runs in your browser. Loading it downloads files from Hugging Face.

Load the model to see your sentences in space
Close points have similar meaning. Lines join each sentence to its nearest neighbour.

In Python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
docs = ["How do I block my card?", "My card was stolen", "What is the transfer fee?"]

E = model.encode(docs, normalize_embeddings=True)          # shape (3, 384)
q = model.encode("I lost my bank card", normalize_embeddings=True)

scores = E @ q                                             # cosine similarity, since vectors are unit length
print(docs[scores.argmax()], scores.round(3))

Pitfalls

  • Comparing scores across models. 0.7 is “very similar” for one model and “unrelated” for another.
  • Assuming similarity means relevance. A question is often most similar to another question, not to its answer.
  • Ignoring truncation. The end of a long text may not be in the vector at all.
  • Trusting the 2-D map. A projection from 384 dimensions to 2 always distorts; use it to explore, not to decide.