Turning the world into numbers

Features, encoding, and why inputs get scaled

A model does arithmetic and nothing else. It multiplies numbers by its knobs, adds them up, and reads out a result. So before any learning can happen, whatever you want to predict from — a house, an email, a customer — has to be turned into a list of numbers. Getting that translation right matters more than beginners expect: a model can only ever be as good as the numbers you feed it.

Features: the numeric inputs

Each number describing an example is a feature feature An individual input the model reads — a house's size, an email's word counts, a pixel. Each feature gets its own weight saying how much it matters. See in glossary → . A house might be described by the features size in square feet, age in years, number of bedrooms. Stack them into a list — [1800, 12, 3] — and that list is what the model actually sees. It never meets the house; it meets the vector.

Genuinely numeric features slot straight in: size, age, temperature, count. They already are numbers, and their magnitudes carry meaning a model can use.

Categories don’t have a number

Plenty of features aren’t numeric, though. A house’s neighborhood, a shirt’s color, a user’s country — these are categorical features, and they resist a naive translation. You might be tempted to number the neighborhoods 1, 2, 3, but that quietly lies to the model: it implies neighborhood 3 is “three times” neighborhood 1, or that 2 sits between them. There’s no such order.

The standard fix is one-hot encoding one-hot vector A vector that is 1 at a single index and 0 everywhere else. Useful for representing categories without inventing a fake numeric order. See in glossary → : give each category its own column, set to 1 for the matching category and 0 everywhere else. Three neighborhoods become three columns; “Riverside” is [1, 0, 0], “Hillcrest” is [0, 1, 0]. Now no false ordering sneaks in — each category is just its own on/off switch. Plain yes/no features (has a garage?) are the easy case of the same idea: a single column of 0 or 1.

Why raw features need scaling

Now the subtle one. Suppose two of your features are size (roughly 500–4,000) and bedrooms (roughly 1–6). Both are perfectly valid numbers, but they live on wildly different scales — one is in the thousands, the other in single digits. That mismatch quietly sabotages training.

To learn, training has to find the bottom of a bowl-shaped error surface, and the shape of that bowl depends on the features. Feature scales stretch it: a knob attached to a huge-valued feature has an outsized effect on the error, making its direction steep; a knob on a small-valued feature makes a shallow one. Mix the two and the bowl becomes a long, thin canyon — steep across, shallow along — so any single step size is wrong for both directions at once, and progress crawls, zig-zagging down the canyon. (We’ll make this picture precise in a few chapters; here, just watch the shape.)

The fix is normalization: rescale every feature to a comparable range — typically by subtracting its mean and dividing by its spread, so each feature ends up centered at 0 with a similar size. This doesn’t change what the features mean; it just reshapes that surface from a stretched canyon into a round bowl, so training can head straight for the bottom instead of zig-zagging. Toggle it and watch:

Once the inputs are clean numbers — categoricals encoded, scales tamed — the model can finally do its one trick: combine those numbers into a prediction. The next section builds the simplest machine that does exactly that, a weighted sum, and shows it’s already enough to learn something real.