Skip to content

Learn / Foundations

Data preparation

Most of the work, and most of the mistakes, happen before any model is trained. How data is prepared for each kind of learning.

On this page
  1. Intuition
  2. What the data looks like in each paradigm
  3. Supervised learning
  4. Unsupervised learning
  5. Embeddings and RAG
  6. Recommender systems
  7. In Python
  8. Pitfalls

Intuition

A model can only learn what the data shows it. If the data that reaches the model differs from the data it will meet in production (different scale, different time period, a column that will not exist yet), the model learns the wrong thing and the test score hides it. Data preparation is the discipline of making training look like reality.

What the data looks like in each paradigm

Almost every dataset in machine learning is a table: one row per example, one column per fact about it. What changes between paradigms is what a row stands for and whether there is an answer column. Amber marks the answer, or the value the model produces.

ParadigmOne row isAnswer columnExample columns
Regression one day on one platform a number: tomorrow’s transaction count weekdaypaydayholidayvolume_last_weekvolume
Classification one card payment a category: fraud or legitimate amount_leihourmerchant_typenew_devicekm_from_homeis_fraud
Clustering one ATM over the last quarter none in the data; the model adds a cluster number withdrawals_daypeak_hourweekend_sharecash_outs_monthcluster
Dimensionality reduction one customer with 50 spending-category columns none in the data; the method outputs 2 new coordinates foodfueltravelonline…pc1pc2
Embeddings one piece of text none; the model outputs a list of 384 numbers (the embedding) messagedim_1dim_2dim_3…dim_384
Retrieval-augmented generation one chunk of a document none; at question time the system ranks chunks by relevance sourcesectiontextembedding
Recommender systems one user; each cell is that user’s rating of one title the empty cells: the model predicts them Nebula RisingVault JobParis LettersDeep OceansMidnight Score

Supervised learning

  • Labels. Decide exactly what you predict and when. “Churned within 30 days of the snapshot date” is a label; “churned” is not.
  • Train / validation / test. Train fits the model, validation chooses hyper-parameters, test is opened once at the end. For anything that evolves over time, split by time.
  • Leakage. Any feature that would not be known at prediction time leaks the answer. Fit scalers, encoders and imputers on the training set only; fitting them on all rows is leakage too.
  • Scaling. Distance- and gradient-based models (kNN, SVM, logistic regression, neural nets) need features on comparable scales. Trees do not.
  • Encoding. One-hot for low-cardinality categories; target or frequency encoding (computed inside cross-validation folds) for high cardinality.
  • Imbalance. With 1% fraud, accuracy is meaningless. Use precision, recall, F1 or PR-AUC, and class weights or resampling on the training set only.
  • Cross-validation. Rotate the validation fold to get a stable estimate from little data; use grouped or time-series folds when rows are related.
Churn prediction: one row per customer One row = one customer on the snapshot date. Answer: churned in the next 30 days (yes / no).
rowcustomerinputlogins_30dinputbalance_leiinputtenure_monthsinputsalary_cardanswerchurned_next_30d
C-104123107noyes
C-1042261840054yesno
C-10430953noyes
C-104414520022yesno

Supervised learning needs the answer column for every training row. The model learns to fill it in for new customers whose answer is not known yet.

Example rows are invented for illustration.

Split visualiser

train 70% validation 15% test 15% · rows are in time order, oldest first

Random split: fine when rows are independent. With time-dependent data it lets the model peek at the future.

Class imbalance: accuracy vs F1
“Always say no” accuracy
94.9%
Model accuracy
88.9%
Model precision
26.8%
Model recall
68.0%
Model F1
38.4%

At 5% positives, a model that never flags anything already scores 94.9% accuracy while catching nothing. Precision, recall and F1 look only at the class you care about.

Target leakage
Test accuracy
100.0%
Accuracy in production
51.7%

The feature is only filled in after the customer has churned, so it hands the model the answer. The test score looks perfect; in production the column is empty and the model collapses.

Unsupervised learning

  • Scaling is not optional. Clustering and PCA are driven by distance and variance, so the feature with the largest numbers wins by default.
  • Distance metric. Euclidean for dense numeric data, cosine for embeddings and text, Gower for mixed types.
  • Outliers. A handful of extreme points can pull a centroid or define a principal component on their own. Clip, transform (log) or use a robust method.
Customer segmentation: one row per customer, no answer column One row = one customer. Answer: none: there is nothing to predict, the model looks for structure.
rowcustomerinputavg_balance_leiinputpayments_monthinputcard_shareinputapp_logins_month
C-20011200380.8241
C-20026400060.13
C-20032100450.9152
C-20045800040.052

There is no label. Clustering will group similar rows; dimensionality reduction will summarise the columns. You judge whether the result means something.

Example rows are invented for illustration.

Embeddings and RAG

  • Parsing. Get clean text out of PDFs, HTML and scans; keep headings and tables intact.
  • Cleaning. Remove navigation, footers, duplicated boilerplate; normalise whitespace.
  • Chunk size. Too small and an answer is split across chunks; too large and it is buried in unrelated text. The RAG lab lets you measure this.
  • Metadata. Store source, section, date and access rights with every chunk so you can filter and cite.
  • Deduplication. Near-identical chunks crowd out other evidence in the top k.
Knowledge base: one row per chunk One row = one chunk of a document. Answer: none; at question time the system ranks chunks by relevance.
rowchunkinputsourceinputsectioninputtextoutputembedding
1faq.mdCards and limitsEvery debit card has a daily ATM limit of 5,000 lei…[0.02, -0.11, …]
2faq.mdTransfersInternational transfers settle within two business days…[-0.08, 0.04, …]
3faq.mdBranchesThe Orbit City branch is open on weekdays from 09:00…[0.05, 0.09, …]

Documents are split into chunks, each stored with its source and an embedding. A question is embedded the same way and matched against this table.

Example rows are invented for illustration.

Recommender systems

  • Interaction matrix. Rows are users, columns are items, almost every cell is empty.
  • Implicit signals. Views, clicks and purchases are abundant but noisy; an absence of interaction is not a dislike.
  • Negative sampling. Sample unseen items as negatives rather than treating all of them as negative.
  • Time-based split. Train on the past, test on the future. A random split leaks what users will like later.
Ratings: users down the side, titles across the top One row = one user; each cell is that user’s rating of one title. Answer: the empty cells: the model predicts them.
rowuserinputNebula RisinginputVault JobinputParis LettersinputDeep OceansinputMidnight Score
U-015·2·4
U-02·5·15
U-034··2·
U-04·41·?

Most cells are empty (·): people rate only a few titles. Recommending means predicting the empty cells, for example the ? for U-04.

Example rows are invented for illustration.

In Python

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import TimeSeriesSplit, cross_val_score

prep = ColumnTransformer([
    ("num", StandardScaler(), ["age", "balance", "tx_count_30d"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["segment", "channel"]),
])
# The pipeline re-fits the scaler inside every fold, so nothing leaks from validation rows.
model = Pipeline([("prep", prep), ("clf", LogisticRegression(class_weight="balanced"))])
scores = cross_val_score(model, X, y, cv=TimeSeriesSplit(5), scoring="average_precision")

Pitfalls

  • Fitting preprocessing on the full dataset before splitting.
  • Random splits on time-dependent data.
  • Evaluating on resampled (balanced) data instead of the real class mix.
  • Features that are updated after the event you are predicting.