Skip to content

Learn / Supervised

Regression

Predicting a number. From a straight line to boosted trees, and how to tell a good fit from a memorised one.

Jump to the playground

On this page
  1. Intuition
  2. What the data looks like
  3. How the data is prepared
  4. Algorithms
  5. The maths
  6. Real-world examples
  7. Playground
  8. In Python
  9. Pitfalls

Intuition

Regression draws a curve through a cloud of points so that, given a new xx, you can read off a sensible yy. The whole subject is a negotiation between two failure modes: a curve too stiff to follow the pattern, and a curve so flexible it chases every noisy point.

What the data looks like

Each row is one example, the inputs describe it, and the answer is a number.

Transaction volume forecasting: one row per day One row = one day on one platform. Answer: a number: tomorrow’s transaction count.
rowdateinputweekdayinputpaydayinputholidayinputvolume_last_weekanswervolume
2026-03-02Monnono4120042850
2026-03-05Thuyesno4410061930
2026-03-08Sunnoyes2390021740
2026-03-09Monnono4285043310

The answer is a number, so this is regression. In the playground each dot is one row with a single input (x) and the answer (y).

Example rows are invented for illustration.

How the data is prepared

  • Split into train and test before looking at relationships; for forecasting, split by time.
  • Scale features for anything using gradients, distances or penalties (linear models with regularisation, kNN, neural nets). Trees and boosting do not need it.
  • Look at the target: a long right tail (amounts, volumes) is often easier to model after a log transform.
  • Treat outliers deliberately: squared error lets a single extreme point dominate the fit.

Algorithms

Linear regression

Fit the straight line that minimises squared error, either in closed form (OLS) or by gradient descent.

Cost
O(n·d²)
Strength
Fast, interpretable coefficients, a strong baseline.
Weakness
Only captures linear relationships; sensitive to outliers.
Used for
credit limit estimation, price elasticity
Open in the playground

Polynomial regression

Linear regression on powers of x, so the line can bend.

Cost
O(n·d²)
Strength
Captures curvature with a simple model.
Weakness
High degrees oscillate wildly between points.
Used for
growth curves, calibration
Open in the playground

Ridge (L2)

Penalise large weights so a flexible model stays smooth.

Cost
O(n·d²)
Strength
Tames overfitting and collinearity; closed form.
Weakness
Keeps every feature; λ must be tuned.
Used for
many correlated features, forecasting with lags
Open in the playground

Lasso (L1)

Penalise absolute weights, which drives some exactly to zero.

Cost
O(n·d) per pass
Strength
Built-in feature selection; sparse models.
Weakness
Unstable among correlated features.
Used for
feature selection, sparse signals
Open in the playground

kNN regressor

Predict the average of the k closest training points.

Cost
O(n) per query
Strength
No training, no assumptions about shape.
Weakness
Slow at query time; needs scaled features; poor in high dimensions.
Used for
local price estimates, imputation
Open in the playground

Decision tree

Split the feature space into boxes and predict the mean of each box.

Cost
O(n·d·log n)
Strength
Handles non-linearities and mixed types; readable rules.
Weakness
Piecewise-constant and high variance.
Used for
pricing rules, segmented forecasts
Open in the playground

Gradient boosting

Add small trees one at a time, each fitting the errors left by the ensemble so far.

Cost
O(rounds·n·d)
Strength
State of the art on tabular data.
Weakness
Many knobs; overfits with too many rounds or a high learning rate.
Used for
transaction volume forecasting, demand prediction
Open in the playground

Neural net (MLP)

Stack linear layers with non-linear activations and train by backpropagation.

Cost
O(epochs·n·weights)
Strength
A universal approximator; scales to huge data.
Weakness
Needs tuning and more data; hard to interpret.
Used for
complex non-linear signals
Open in the playground

The maths

Linear regression minimises the mean squared error

MSE(w,b)=1n∑i=1n(yi−(w⊤xi+b))2\mathrm{MSE}(w, b) = \frac{1}{n}\sum_{i=1}^{n} \big(y_i - (w^\top x_i + b)\big)^2

which has the closed-form solution w^=(X⊤X)−1X⊤y\hat{w} = (X^\top X)^{-1} X^\top y (ordinary least squares), or can be reached step by step with gradient descent: w←w−η ∇wMSEw \leftarrow w - \eta \, \nabla_w \mathrm{MSE}.

Regularisation adds a price for large weights:

Ridge: MSE+λ∥w∥22Lasso: MSE+λ∥w∥1\text{Ridge: } \mathrm{MSE} + \lambda \lVert w \rVert_2^2 \qquad\qquad \text{Lasso: } \mathrm{MSE} + \lambda \lVert w \rVert_1

Ridge shrinks all weights smoothly; lasso pushes some to exactly zero. Fit quality is usually reported as R2=1−SSres/SStotR^2 = 1 - \mathrm{SS}_{res} / \mathrm{SS}_{tot}: the share of the target’s variance the model explains.

Real-world examples

  • Transaction volume forecasting: predicting daily volumes per platform so capacity and liquidity can be planned.
  • Credit limit estimation: a first limit proposal from income and behaviour features.
  • House prices and demand: the classic tabular regression problems.

Playground

Start with a polynomial, ridge and gradient boosting on the sine wave: the polynomial and ridge share the same flexibility, but the penalty keeps ridge from wiggling, while boosting builds the curve from small steps. Then try a deep tree on few points and watch test error climb.

Data
More data options
Model A
Model B
Model C
A · Polynomial regression

Click the plot to add a point.

B · Ridge (L2)

Press Train to compare.

C · Gradient boosting

Press Train to compare.

MSE per step (axis fitted to the range)

Train a model to see its loss curve.

Step -train ━test ┅

● train ○ test

Press Train. The verdict appears here.

In Python

from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import RidgeCV
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, shuffle=False)  # time-ordered

baseline = RidgeCV(alphas=[0.1, 1, 10]).fit(X_tr, y_tr)
model = HistGradientBoostingRegressor(max_depth=6, learning_rate=0.05).fit(X_tr, y_tr)

for name, m in [("ridge", baseline), ("boosting", model)]:
    pred = m.predict(X_te)
    print(name, mean_absolute_error(y_te, pred), r2_score(y_te, pred))

Pitfalls

  • Extrapolation. Trees predict a constant outside the training range; polynomials explode. Know which you are using before forecasting beyond the data.
  • A high R2R^2 on the training set says nothing. Compare train and test.
  • Reporting only RMSE. One bad day can dominate it; look at MAE and at the residuals.
  • Heteroscedastic noise. If errors grow with the target, a single error bar is wrong. Try the heteroscedastic dataset above.