Skip to content

Learn / Systems

Recommender systems

Predicting what each person will want next, from what they and people like them did before.

Jump to the lab

On this page
  1. Intuition
  2. What the data looks like
  3. How the data is prepared
  4. Algorithms
  5. The maths
  6. Real-world examples
  7. Lab
  8. In Python
  9. Pitfalls

Intuition

A recommender fills in a huge, mostly empty table: users down the side, items across the top, a rating or interaction in the few cells where a user met an item. The task is to guess the empty cells well enough to put the right handful of items in front of each person.

There are two sources of evidence. Content: this item resembles things you liked. Collaboration: people whose taste matched yours liked this. Real systems combine both.

What the data looks like

Recommender data is a big, mostly empty grid: who rated or used what.

Ratings: users down the side, titles across the top One row = one user; each cell is that user’s rating of one title. Answer: the empty cells: the model predicts them.
rowuserinputNebula RisinginputVault JobinputParis LettersinputDeep OceansinputMidnight Score
U-015·2·4
U-02·5·15
U-034··2·
U-04·41·?

Most cells are empty (·): people rate only a few titles. Recommending means predicting the empty cells, for example the ? for U-04.

Example rows are invented for illustration.

How the data is prepared

  • Interaction matrix. Usually over 99% empty, so it is stored as a list of (user, item, value) triples.
  • Explicit vs implicit feedback. Ratings are clear but rare. Clicks, views and purchases are plentiful but ambiguous: a missing interaction mostly means “never saw it”, not “disliked it”.
  • Negative sampling. For implicit data, sample unseen items as negatives during training.
  • Time-based split. Train on the past, evaluate on what each user did next.
  • Cold start. New users and new items have no history; plan a fallback (popularity, content features, a short onboarding question).

Algorithms

Popularity baseline

Recommend what most people like.

Cost
O(ratings)
Strength
Works for brand-new users.
Weakness
Not personalised at all.
Used for
cold start, trending lists
Open in the playground

Content-based

Recommend items similar to what this user liked, using item features.

Cost
O(items)
Strength
Needs no other users; handles new items.
Weakness
Stays inside the user’s bubble.
Used for
new product launches
Open in the playground

Collaborative filtering

People who agreed in the past will agree again: use similar users or similar items.

Cost
O(items²) item-item
Strength
No item features needed; finds surprising links.
Weakness
Cold start; popularity bias.
Used for
“customers also bought”
Open in the playground

Matrix factorisation

Learn a short vector for every user and item so their dot product predicts the rating.

Cost
O(epochs·ratings·k)
Strength
Compact, accurate, scalable.
Weakness
Cold start; needs retraining for new users.
Used for
next-best-offer, movie ratings
Open in the playground

Hybrid

Blend collaborative and content signals.

Cost
depends
Strength
Covers each method’s weak spots.
Weakness
More moving parts to maintain.
Used for
production recommenders
Open in the playground

The maths

Matrix factorisation gives each user uu a vector pup_u and each item ii a vector qiq_i, and predicts

r^ui=μ+bu+bi+pu⊤qi\hat{r}_{ui} = \mu + b_u + b_i + p_u^\top q_i

where μ\mu is the global mean and bub_u, bib_i capture “this user rates high” and “this item is liked by everyone”. The parameters minimise the regularised squared error over the observed ratings K\mathcal{K}:

min⁡∑(u,i)∈K(rui−r^ui)2+λ(∥pu∥2+∥qi∥2+bu2+bi2)\min \sum_{(u,i) \in \mathcal{K}} \big(r_{ui} - \hat{r}_{ui}\big)^2 + \lambda\big(\lVert p_u \rVert^2 + \lVert q_i \rVert^2 + b_u^2 + b_i^2\big)

Stochastic gradient descent visits one rating at a time, with error eui=rui−r^uie_{ui} = r_{ui} - \hat{r}_{ui}:

pu←pu+η (eui qi−λ pu)qi←qi+η (eui pu−λ qi)p_u \leftarrow p_u + \eta\,(e_{ui}\, q_i - \lambda\, p_u) \qquad q_i \leftarrow q_i + \eta\,(e_{ui}\, p_u - \lambda\, q_i)

Quality is judged on the list, not the rating: precision@k\text{precision@}k is the share of the top kk recommendations the user actually liked.

Real-world examples

  • Product offers and next-best-action in banking: which product, if any, to show this customer in the app today.
  • Films, music and shopping: the systems everyone has met.
  • Content feeds: where implicit feedback and freshness dominate.

Lab

Sixty invented titles in four hidden genres, forty synthetic users. Rate a few titles (or use a shortcut button), press Train, and watch matrix factorisation pull you toward the titles you would like in a latent space it learned without ever being told the genres.

Your ratings (0)
Nebula Rising
Vault Job
Paris Letters
Deep Oceans
Orbit Rising
Midnight Job
Summer Letters
Wild Oceans
Starfall Rising
Silent Job
Lost Letters
Hidden Oceans
Void Rising
Double Job
Second Letters
Frozen Oceans
Nebula Protocol
Vault Score
Paris Promise
Deep Planet
Orbit Protocol
Midnight Score
Summer Promise
Wild Planet
Starfall Protocol
Silent Score
Lost Promise
Hidden Planet
Void Protocol
Double Score
Second Promise
Frozen Planet
Nebula Legacy
Vault Crew
Paris Hearts
Deep Cities
Orbit Legacy
Midnight Crew
Summer Hearts
Wild Cities
Starfall Legacy
Silent Crew
Lost Hearts
Hidden Cities
Void Legacy
Double Crew
Second Hearts
Frozen Cities
Nebula Frontier
Vault Cross
Paris Chance
Deep Minds
Orbit Frontier
Midnight Cross
Summer Chance
Wild Minds
Starfall Frontier
Silent Cross
Lost Chance
Hidden Minds

Space opera Heist Romance Documentary · the genres are hidden from collaborative filtering and matrix factorisation

Latent space · 2 learned dimensions

● items, coloured by hidden genre · small dots = the 40 other users · white = you · outlined = rated by you

In Python

from surprise import SVD, Dataset, Reader
from surprise.model_selection import cross_validate

data = Dataset.load_from_df(ratings[["user_id", "item_id", "rating"]], Reader(rating_scale=(1, 5)))

algo = SVD(n_factors=50, lr_all=0.005, reg_all=0.05)   # biased matrix factorisation, trained with SGD
cross_validate(algo, data, measures=["RMSE"], cv=5, verbose=True)

algo.fit(data.build_full_trainset())
print(algo.predict(uid="u42", iid="i7").est)

Pitfalls

  • Optimising RMSE when the product is a ranked list. A model can predict ratings well and still order the top five badly.
  • Popularity bias. Recommending what is already popular makes it more popular; measure coverage and diversity too.
  • Random train/test splits that let the model see a user’s future.
  • Forgetting cold start until launch day.
  • Feedback loops. Users can only click what you show them, so tomorrow’s training data is shaped by today’s model.