Scaling & regularization

Three moves that get confused because they all involve a tuning knob and the word "scale" — but two of them reshape the data and the third constrains the model. Here they are from every angle — when, why, how, where, what — with ridge-vs-lasso geometry made visible.

Standardization & whitening are developed in the L² deep dive and the change-of-basis catalog; the Bayesian prior reading sits beside the the paradigms. Cross-referenced, not repeated.

1 · The three, at a glance

The fastest way to stop confusing them — what each one acts on:

Move	Acts on	Operation	Effect	Typical use
Normalization	the data	rescale to a fixed range [0,1]	changes scale, keeps shape	kNN, k-means, image pixels
Standardization	the data	center & scale (z-score)	mean 0, var 1; keeps shape	PCA, pre-regularization, SVM
Regularization	the model	penalize coefficients (L1/L2)	shrinks / zeros coefficients	overfitting, p ≫ n, selection

2 · Normalization — transforms the data

When: Features live on very different ranges and the method cares about magnitude (kNN, k-means, neural-net inputs, image pixels).
Why: Stop a wide-range feature from dominating distances or gradients just because of its units.
How: Rescale each feature to a fixed interval: $x^{'} = \frac{x - m i n}{m a x - m i n} \in [0, 1]$ (or to unit norm).
Where: Distance- and gradient-based models, bounded inputs, image data.
What: Same shape, squeezed into a fixed range — but outliers can compress everything else against one wall.

3 · Standardization — transforms the data

When: Features differ in scale/units and you want them comparable, or a method assumes roughly unit variance (PCA, regularized regression, SVM, gradient descent).
Why: Put every feature in the same currency — standard deviations from its own mean.
How: The z-score $z = \frac{x - μ}{σ}$ ; its multivariate cousin is whitening $z = Σ^{- 1/2} (x - μ)$ .
Where: PCA, ridge/lasso, logistic regression, z-tests — and as the step before regularizing.
What: Mean 0, variance 1; the shape (skew, kurtosis) is unchanged. It's a change of basis — see the L² deep dive.

4 · Regularization — constrains the model

When: The model overfits — many features and few rows (p ≫ n), or coefficients blowing up and flipping sign run to run.
Why: Trade a little bias for much less variance, and encode a prior that coefficients should be small or sparse.
How: Add a penalty to the loss: ridge $λ \sum β_{j}^{2}$ , lasso $λ \sum ∣ β_{j} ∣$ , or elastic net (a mix).
Where: Regression, GLMs, SVMs, and neural nets (L2 = weight decay; dropout is a stochastic cousin).
What: Shrinks coefficients (ridge) or sets some exactly to zero (lasso → feature selection); λ is tuned by cross-validation.

The whole story is geometric. Toggle the penalty and slide the strength: the L1 diamond's corners pull the solution onto an axis (a coefficient set to exactly zero), while the round L2 disc only shrinks. The shrinkage-paths view shows the same thing over a sweep of λ.

penaltystrength

Loss ellipses (grey) centred at the OLS estimate (grey dot); the gold region is the L1 diamond budget. The solution (gold dot) is where the smallest ellipse meets the budget. Shrink the strength and the L1 corner pulls the solution onto an axis — a coefficient set exactly to zero. That corner is why lasso selects features; the round L2 disc has no corners, so ridge only shrinks.

5 · The Bayesian view — a penalty is a prior

Regularization isn't an ad-hoc fudge; it's a belief. Maximizing the posterior (MAP) with a prior on the coefficients is exactly penalized least squares, because the penalty is $- lo g (prior)$ :

- l o g likelihood ∥ y - X β ∥^{2} + Gaussian prior λ ∥ β ∥_{2}^{2} (ridge), + Laplace prior λ ∥ β ∥_{1} (lasso) .

A Gaussian prior says coefficients are probably small (ridge); a Laplace prior, with its sharp peak at zero, says many are probably exactly zero (lasso). The strength λ is the prior's confidence. This is the frequentist/Bayesian bridge from the the paradigms, seen from the estimator side — and it's why dropout (a Bernoulli-noise regularizer) and weight decay live in the same family.

Study these — derivation cards

Reconstruct before you flip. The matching spaced-repetition deck lives in the Obsidian vault.

Prompt 1 · tap to reveal

Normalization vs standardization vs regularization — what does each act on?

1 / 8