Learn / Supervised
Regression
Predicting a number. From a straight line to boosted trees, and how to tell a good fit from a memorised one.
On this page
Intuition
Regression draws a curve through a cloud of points so that, given a new , you can read off a sensible . The whole subject is a negotiation between two failure modes: a curve too stiff to follow the pattern, and a curve so flexible it chases every noisy point.
What the data looks like
Each row is one example, the inputs describe it, and the answer is a number.
| rowdate | inputweekday | inputpayday | inputholiday | inputvolume_last_week | answervolume |
|---|---|---|---|---|---|
| 2026-03-02 | Mon | no | no | 41200 | 42850 |
| 2026-03-05 | Thu | yes | no | 44100 | 61930 |
| 2026-03-08 | Sun | no | yes | 23900 | 21740 |
| 2026-03-09 | Mon | no | no | 42850 | 43310 |
The answer is a number, so this is regression. In the playground each dot is one row with a single input (x) and the answer (y).
Example rows are invented for illustration.
How the data is prepared
- Split into train and test before looking at relationships; for forecasting, split by time.
- Scale features for anything using gradients, distances or penalties (linear models with regularisation, kNN, neural nets). Trees and boosting do not need it.
- Look at the target: a long right tail (amounts, volumes) is often easier to model after a log transform.
- Treat outliers deliberately: squared error lets a single extreme point dominate the fit.
Algorithms
Linear regression
Fit the straight line that minimises squared error, either in closed form (OLS) or by gradient descent.
- Cost
- O(n·d²)
- Strength
- Fast, interpretable coefficients, a strong baseline.
- Weakness
- Only captures linear relationships; sensitive to outliers.
- Used for
- credit limit estimation, price elasticity
Polynomial regression
Linear regression on powers of x, so the line can bend.
- Cost
- O(n·d²)
- Strength
- Captures curvature with a simple model.
- Weakness
- High degrees oscillate wildly between points.
- Used for
- growth curves, calibration
Ridge (L2)
Penalise large weights so a flexible model stays smooth.
- Cost
- O(n·d²)
- Strength
- Tames overfitting and collinearity; closed form.
- Weakness
- Keeps every feature; λ must be tuned.
- Used for
- many correlated features, forecasting with lags
Lasso (L1)
Penalise absolute weights, which drives some exactly to zero.
- Cost
- O(n·d) per pass
- Strength
- Built-in feature selection; sparse models.
- Weakness
- Unstable among correlated features.
- Used for
- feature selection, sparse signals
kNN regressor
Predict the average of the k closest training points.
- Cost
- O(n) per query
- Strength
- No training, no assumptions about shape.
- Weakness
- Slow at query time; needs scaled features; poor in high dimensions.
- Used for
- local price estimates, imputation
Decision tree
Split the feature space into boxes and predict the mean of each box.
- Cost
- O(n·d·log n)
- Strength
- Handles non-linearities and mixed types; readable rules.
- Weakness
- Piecewise-constant and high variance.
- Used for
- pricing rules, segmented forecasts
Gradient boosting
Add small trees one at a time, each fitting the errors left by the ensemble so far.
- Cost
- O(rounds·n·d)
- Strength
- State of the art on tabular data.
- Weakness
- Many knobs; overfits with too many rounds or a high learning rate.
- Used for
- transaction volume forecasting, demand prediction
Neural net (MLP)
Stack linear layers with non-linear activations and train by backpropagation.
- Cost
- O(epochs·n·weights)
- Strength
- A universal approximator; scales to huge data.
- Weakness
- Needs tuning and more data; hard to interpret.
- Used for
- complex non-linear signals
The maths
Linear regression minimises the mean squared error
which has the closed-form solution (ordinary least squares), or can be reached step by step with gradient descent: .
Regularisation adds a price for large weights:
Ridge shrinks all weights smoothly; lasso pushes some to exactly zero. Fit quality is usually reported as : the share of the target’s variance the model explains.
Real-world examples
- Transaction volume forecasting: predicting daily volumes per platform so capacity and liquidity can be planned.
- Credit limit estimation: a first limit proposal from income and behaviour features.
- House prices and demand: the classic tabular regression problems.
Playground
Start with a polynomial, ridge and gradient boosting on the sine wave: the polynomial and ridge share the same flexibility, but the penalty keeps ridge from wiggling, while boosting builds the curve from small steps. Then try a deep tree on few points and watch test error climb.
Click the plot to add a point.
Press Train to compare.
Press Train to compare.
Train a model to see its loss curve.
| Step - | train ━ | test ┅ |
|---|
● train ○ test
Press Train. The verdict appears here.
In Python
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import RidgeCV
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, shuffle=False) # time-ordered
baseline = RidgeCV(alphas=[0.1, 1, 10]).fit(X_tr, y_tr)
model = HistGradientBoostingRegressor(max_depth=6, learning_rate=0.05).fit(X_tr, y_tr)
for name, m in [("ridge", baseline), ("boosting", model)]:
pred = m.predict(X_te)
print(name, mean_absolute_error(y_te, pred), r2_score(y_te, pred))
Pitfalls
- Extrapolation. Trees predict a constant outside the training range; polynomials explode. Know which you are using before forecasting beyond the data.
- A high on the training set says nothing. Compare train and test.
- Reporting only RMSE. One bad day can dominate it; look at MAE and at the residuals.
- Heteroscedastic noise. If errors grow with the target, a single error bar is wrong. Try the heteroscedastic dataset above.