Skip to content

Learn / Supervised

Classification

Predicting a category, and a probability for it. Decision boundaries, thresholds and metrics that survive imbalance.

Jump to the playground

On this page
  1. Intuition
  2. What the data looks like
  3. How the data is prepared
  4. Algorithms
  5. The maths
  6. Real-world examples
  7. Playground
  8. In Python
  9. Pitfalls

Intuition

A classifier draws a boundary: on one side it says “A”, on the other “B”. Simple models draw straight boundaries; flexible ones draw curves, islands and checkerboards. Good classifiers also say how sure they are, because the decision (block the payment, call the customer) depends on the cost of being wrong.

What the data looks like

Each row is one example, the inputs describe it, and the answer is a category.

Fraud detection: one row per card payment One row = one card payment. Answer: a category: fraud or legitimate.
rowpaymentinputamount_leiinputhourinputmerchant_typeinputnew_deviceinputkm_from_homeansweris_fraud
P-8831024914groceryno2no
P-8831198003electronicsyes1430yes
P-88312759fuelno11no
P-88313420023jewelleryyes860yes

The answer is a category, so this is classification. In the playground each dot is one row: two inputs (x, y) and the class as its colour.

Example rows are invented for illustration.

How the data is prepared

  • Stratify the split so every set keeps the same class mix.
  • Scale features for kNN, SVM, logistic regression and neural nets.
  • Imbalance: keep the test set at the real class ratio; rebalance only the training set, or use class weights.
  • Leakage: the most common cause of a “95% accurate” model that fails in production. See data preparation.

Algorithms

Logistic regression

A linear score squashed through a sigmoid into a probability.

Cost
O(epochs·n·d)
Strength
Calibrated probabilities, interpretable, hard to beat as a baseline.
Weakness
Linear decision boundary.
Used for
credit scoring, churn baseline
Open in the playground

k-nearest neighbours

Vote among the k closest labelled points.

Cost
O(n) per query
Strength
Simple and flexible boundaries.
Weakness
Needs feature scaling; slow on large data.
Used for
similar-customer lookup, anomaly triage
Open in the playground

Naive Bayes

Assume features are independent given the class and apply Bayes’ rule.

Cost
O(n·d)
Strength
Tiny, fast, works with little data.
Weakness
The independence assumption is usually false.
Used for
spam filtering, text routing
Open in the playground

Decision tree

Ask a sequence of yes/no questions that make the groups as pure as possible.

Cost
O(n·d·log n)
Strength
Readable rules, no scaling needed.
Weakness
Overfits when grown deep.
Used for
eligibility rules, explainable decisions
Open in the playground

Random forest

Average many deep trees, each trained on a bootstrap sample with random feature choices.

Cost
O(trees·n·log n)
Strength
Robust, little tuning, good out of the box.
Weakness
Larger and slower than a single tree; less interpretable.
Used for
fraud detection, churn prediction
Open in the playground

Support vector machine

Find the boundary with the widest margin; kernels bend it.

Cost
O(n²) - O(n³) exact
Strength
Effective in high dimensions; strong with the right kernel.
Weakness
Scores are not probabilities; kernel SVMs scale poorly.
Used for
text classification, small tabular sets
Open in the playground

Neural net (MLP)

Learn the features and the boundary together.

Cost
O(epochs·n·weights)
Strength
Learns arbitrary boundaries.
Weakness
Data-hungry; easy to overfit.
Used for
onboarding conversion, behaviour models
Open in the playground

The maths

Logistic regression turns a linear score into a probability with the sigmoid, and is trained by minimising log-loss:

p(y=1∣x)=σ(w⊤x+b)=11+e−(w⊤x+b)L=−1n∑i[ yilog⁡pi+(1−yi)log⁡(1−pi) ]p(y = 1 \mid x) = \sigma(w^\top x + b) = \frac{1}{1 + e^{-(w^\top x + b)}} \qquad \mathcal{L} = -\frac{1}{n}\sum_{i} \big[\, y_i \log p_i + (1 - y_i)\log(1 - p_i) \,\big]

A decision tree instead picks the split that most reduces impurity, for example Gini G=1−∑kpk2G = 1 - \sum_k p_k^2. A support vector machine maximises the margin by minimising the hinge loss max⁡(0, 1−y (w⊤x+b))\max(0,\, 1 - y\,(w^\top x + b)) plus λ∥w∥2\lambda \lVert w \rVert^2.

From the confusion matrix:

precision=TPTP+FPrecall=TPTP+FNF1=2⋅precision⋅recallprecision+recall\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN} \qquad F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}

Real-world examples

  • Churn prediction: which mobile app users are about to leave, early enough to act.
  • Fraud detection: rare positives and expensive misses, so recall at a fixed false-positive budget is the metric.
  • Onboarding conversion: who will finish signing up, and where a nudge helps.
  • Credit scoring and spam filtering: the textbook cases.

Playground

A single tree, a random forest and logistic regression on moons: the tree’s boundary is a jagged staircase, the forest averages thirty of them into something smooth, and logistic regression can only draw one straight line. Switch the dataset to spirals to see which models can follow it at all.

Data
More data options
Model A
Model B
Model C
A · Decision tree

Tap or click the plot to add a class A point. Tick "Add clicks as class B" for class B.

B · Random forest

Press Train to compare.

C · Logistic regression

Press Train to compare.

log-loss per step (axis fitted to the range)

Train a model to see its loss curve.

Step -train ━test ┅

class A class B · shading = the model's prediction, stronger where it is surer · ● train ○ test

Press Train. The verdict appears here.

In Python

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import average_precision_score, classification_report
from sklearn.model_selection import train_test_split

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

clf = RandomForestClassifier(n_estimators=300, class_weight="balanced", n_jobs=-1).fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]

print("PR-AUC", average_precision_score(y_te, proba))
print(classification_report(y_te, proba > 0.3))  # pick the threshold from costs, not 0.5 by habit

Pitfalls

  • Accuracy on imbalanced data. 99% accuracy can mean “never predicts fraud”.
  • The 0.5 threshold. It is a default, not a decision. Choose it from the cost of false positives and false negatives.
  • Uncalibrated scores. A forest’s 0.8 is not an 80% chance; calibrate if you use probabilities as numbers.
  • A boundary that is too clever. If kNN with k = 1 scores 100% on training data, that is memorisation.