Learn / Systems
Recommender systems
Predicting what each person will want next, from what they and people like them did before.
On this page
Intuition
A recommender fills in a huge, mostly empty table: users down the side, items across the top, a rating or interaction in the few cells where a user met an item. The task is to guess the empty cells well enough to put the right handful of items in front of each person.
There are two sources of evidence. Content: this item resembles things you liked. Collaboration: people whose taste matched yours liked this. Real systems combine both.
What the data looks like
Recommender data is a big, mostly empty grid: who rated or used what.
| rowuser | inputNebula Rising | inputVault Job | inputParis Letters | inputDeep Oceans | inputMidnight Score |
|---|---|---|---|---|---|
| U-01 | 5 | · | 2 | · | 4 |
| U-02 | · | 5 | · | 1 | 5 |
| U-03 | 4 | · | · | 2 | · |
| U-04 | · | 4 | 1 | · | ? |
Most cells are empty (·): people rate only a few titles. Recommending means predicting the empty cells, for example the ? for U-04.
Example rows are invented for illustration.
How the data is prepared
- Interaction matrix. Usually over 99% empty, so it is stored as a list of (user, item, value) triples.
- Explicit vs implicit feedback. Ratings are clear but rare. Clicks, views and purchases are plentiful but ambiguous: a missing interaction mostly means “never saw it”, not “disliked it”.
- Negative sampling. For implicit data, sample unseen items as negatives during training.
- Time-based split. Train on the past, evaluate on what each user did next.
- Cold start. New users and new items have no history; plan a fallback (popularity, content features, a short onboarding question).
Algorithms
Popularity baseline
Recommend what most people like.
- Cost
- O(ratings)
- Strength
- Works for brand-new users.
- Weakness
- Not personalised at all.
- Used for
- cold start, trending lists
Content-based
Recommend items similar to what this user liked, using item features.
- Cost
- O(items)
- Strength
- Needs no other users; handles new items.
- Weakness
- Stays inside the user’s bubble.
- Used for
- new product launches
Collaborative filtering
People who agreed in the past will agree again: use similar users or similar items.
- Cost
- O(items²) item-item
- Strength
- No item features needed; finds surprising links.
- Weakness
- Cold start; popularity bias.
- Used for
- “customers also bought”
Matrix factorisation
Learn a short vector for every user and item so their dot product predicts the rating.
- Cost
- O(epochs·ratings·k)
- Strength
- Compact, accurate, scalable.
- Weakness
- Cold start; needs retraining for new users.
- Used for
- next-best-offer, movie ratings
Hybrid
Blend collaborative and content signals.
- Cost
- depends
- Strength
- Covers each method’s weak spots.
- Weakness
- More moving parts to maintain.
- Used for
- production recommenders
The maths
Matrix factorisation gives each user a vector and each item a vector , and predicts
where is the global mean and , capture “this user rates high” and “this item is liked by everyone”. The parameters minimise the regularised squared error over the observed ratings :
Stochastic gradient descent visits one rating at a time, with error :
Quality is judged on the list, not the rating: is the share of the top recommendations the user actually liked.
Real-world examples
- Product offers and next-best-action in banking: which product, if any, to show this customer in the app today.
- Films, music and shopping: the systems everyone has met.
- Content feeds: where implicit feedback and freshness dominate.
Lab
Sixty invented titles in four hidden genres, forty synthetic users. Rate a few titles (or use a shortcut button), press Train, and watch matrix factorisation pull you toward the titles you would like in a latent space it learned without ever being told the genres.
● items, coloured by hidden genre · small dots = the 40 other users · white = you · outlined = rated by you
In Python
from surprise import SVD, Dataset, Reader
from surprise.model_selection import cross_validate
data = Dataset.load_from_df(ratings[["user_id", "item_id", "rating"]], Reader(rating_scale=(1, 5)))
algo = SVD(n_factors=50, lr_all=0.005, reg_all=0.05) # biased matrix factorisation, trained with SGD
cross_validate(algo, data, measures=["RMSE"], cv=5, verbose=True)
algo.fit(data.build_full_trainset())
print(algo.predict(uid="u42", iid="i7").est)
Pitfalls
- Optimising RMSE when the product is a ranked list. A model can predict ratings well and still order the top five badly.
- Popularity bias. Recommending what is already popular makes it more popular; measure coverage and diversity too.
- Random train/test splits that let the model see a user’s future.
- Forgetting cold start until launch day.
- Feedback loops. Users can only click what you show them, so tomorrow’s training data is shaped by today’s model.