Pipelines and ColumnTransformer

By now you have met two preprocessing steps — scaling numbers and encoding categories — and one ironclad rule: fit preprocessing on the training set only, then apply it to everything else. You have also seen how easy it is to break that rule by accident. A misplaced fit_transform, a pd.get_dummies called separately on train and test, a scaler quietly fit on all the data — each leaks the test set and inflates your score.

This page introduces the tools that make doing it right easier than doing it wrong: Pipeline, which chains preprocessing and a model into a single estimator, and ColumnTransformer, which applies different preprocessing to different columns. Together they turn a fragile, multi-step ritual into one object you fit once. This is not a convenience feature — it is how professional scikit-learn code stays correct.

The problem: preprocessing and modeling are easy to desynchronize

Without pipelines, a realistic workflow looks like a chain of manual steps you must perform in the right order, on the right data, every time:

1. Split into train and test.
2. Fit a scaler on the training features, transform train, transform test.
3. Fit an encoder on the training categories, transform train, transform test.
4. Glue the scaled numbers and encoded categories back together.
5. Fit the model on the training matrix.
6. Repeat steps 2–4 exactly on any new data before predicting.

Every one of those steps is a chance to leak the test set (fit on the wrong data), to forget a transform, or to apply transforms in a different order at predict time than at training time. And it gets worse: when you later use cross-validation (its own page), the data is split into folds many times, and the preprocessing must be re-fit fresh inside every fold. Doing that by hand is so tedious that people skip it — and silently leak.

Pipeline: one estimator, one fit, no leaks

A Pipeline chains a sequence of steps — any number of transformers followed by a final estimator (a model) — into a single object that behaves exactly like a model. You build it from a list of (name, step) pairs:

Pipeline([
    ("scaler", StandardScaler()),
    ("model",  LogisticRegression(max_iter=1000)),
])

When you call .fit(X_train, y_train) on this pipeline, it does the right thing automatically: it fits the scaler on X_train, transforms X_train, and feeds the result to the model's fit. When you call .predict(X_test), it transforms X_test with the already-fitted scaler (no re-fitting) and passes that to the model's predict. The fit-on-train / transform-both rule is enforced by construction — you cannot accidentally fit a step on the test set, because you never touch the steps individually.

Notice the asymmetry in those two diagrams: during training the scaler both fits and transforms; during prediction it only transforms, reusing what it learned. That is precisely the discipline you would have to remember by hand — and the pipeline never forgets it.

Here is the simplest possible pipeline, end to end:

The scaling happens inside the pipeline, fit on the training data only, and applied consistently to the test data — and you never wrote a single scaler.fit or scaler.transform call. The leak is impossible to commit here, and the code is shorter to boot.

make_pipeline: the same thing, names for free

If you do not care to name the steps, make_pipeline builds the identical object and names each step after its class automatically:

Why the pipeline prevents leakage in cross-validation

This is the reason pipelines matter most, so it deserves its own moment. Cross-validation (covered fully on its own page) repeatedly splits the data into a training portion and a held-out portion, scoring on each. If your scaler was fit once on the whole dataset before cross-validation, then in every fold the "held-out" portion was already seen by the scaler — a leak, repeated in every fold, quietly inflating the average.

When you cross-validate a pipeline, scikit-learn re-fits the entire pipeline — scaler, encoder, and all — separately inside each fold, using only that fold's training portion. The held-out portion of each fold is never seen by any preprocessing step until it is scored. The leak becomes structurally impossible.

Hand cross_val_score a bare model with pre-scaled data and you leak; hand it a pipeline and you do not. Same number of lines, opposite correctness. This is the single strongest argument for building pipelines as a default habit rather than an afterthought.

ColumnTransformer: different transforms for different columns

Real datasets are not all numeric or all categorical — they are mixed. You want to StandardScaler the numeric columns and OneHotEncoder the categorical ones, in the same dataset, at the same time. A single Pipeline step applies one transform to all the columns it receives, which is not what you want here.

ColumnTransformer solves this. You give it a list of (name, transformer, columns) triples, and it routes each group of columns to its own transformer, then concatenates the results side by side.

Let us build a realistic mixed dataset inline and assemble the full thing: a ColumnTransformer for preprocessing feeding a LogisticRegression, all wrapped in one Pipeline.

One pipe.fit(X_train, y_train) call: it scaled age and income, one-hot encoded city and plan, glued the columns together, and trained the classifier — each preprocessing step fit on the training data only. You handed it a raw mixed DataFrame and got a trained model, with leakage ruled out by construction.

Seeing the columns the model actually receives

It is worth peeking at what comes out of the ColumnTransformer, because the model never sees your original columns — it sees the transformed ones.

The two numeric columns stay as two columns (now scaled), while city and plan expand into one 0/1 column per category. The num__ and cat__ prefixes tell you which transformer produced each column.

The whole recipe travels as one object

Because the pipeline is an estimator, the complete preprocessing-plus-model recipe moves around as a single thing. You can pass it to a train/test workflow, to cross_val_score, or (with a small grid) to GridSearchCV for tuning — all covered on their own pages — and the preprocessing rides along, re-fit correctly in each context. You predict on brand-new raw data by calling .predict on the same object, and it applies the identical scaling and encoding it learned at training time.

The new batch included Tokyo, a city the encoder never saw at training time. Because the encoder was created with handle_unknown="ignore", the pipeline handled it gracefully instead of crashing — the exact robustness the encoding page described, now happening automatically inside the pipeline.

When NOT to reach for a pipeline (and misconceptions)

Pipelines are a strong default, but a few honest notes:

• Pure exploration. When you are poking at data interactively to see what a transform does, calling a transformer directly is fine and often clearer. Reach for a pipeline once you are training and evaluating a model you will trust.
• Steps that are not fit-then-transform. A pipeline composes scikit-learn transformers (objects with fit/transform). One-off cleaning that does not learn anything from the data — dropping a column, parsing a date string — can happen before the pipeline; it does not leak because it learns nothing. (Engineering features with a fit step, however, belongs inside.)
• Misconception: a pipeline is just tidier code. Tidiness is a bonus; the real payoff is correctness. A pipeline makes leak-free preprocessing the default behavior, especially under cross-validation. That is a safety property, not a style preference.
• Misconception: pipelines change the model. They do not. A pipeline produces the same result as doing every step correctly by hand — it just makes "correctly" automatic. If a pipeline scores differently from your manual code, your manual code almost certainly had a leak.

Real-world applications

Pipelines are the backbone of production scikit-learn:

• Tabular prediction at companies — churn, fraud, credit, demand — almost always ships as a single Pipeline of ColumnTransformer plus a model, so the same preprocessing that trained the model also runs at prediction time, with no drift between the two.
• Reproducibility and handoff. One object captures the entire recipe, so a teammate (or a future you) can load it, call .predict, and get exactly the training-time behavior — no separate, undocumented preprocessing script to forget.
• Honest model selection. Because the whole pipeline cross-validates without leaking, comparisons between models and preprocessing choices are fair. That is the foundation the tuning page builds on.

The shape to remember: raw columns → ColumnTransformer (scale numeric, encode categorical) → model — all inside one Pipeline you fit once.

Your turn

Check your understanding