Section 04

The simplest model

Weights, bias, and why parameters are just knobs

The one-dial machine could only stretch its input. Real problems need a model that can combine several inputs and shift its baseline. The simplest such model — the workhorse that neural networks are ultimately built from — is the linear model. Understanding it fully means you already understand most of what a giant network is doing, just repeated.

A line, then a plane

Suppose we’re predicting a house’s price. With one input, size, the model is a straight line:

price=w⋅size+b\text{price} = w \cdot \text{size} + b

The weight weight One adjustable number that scales an input inside a model — how strongly that input pushes the prediction up or down. The weights (plus biases) are the parameters training adjusts. See in glossary → ww is the slope — how many dollars each extra square foot adds. The bias bias The constant term added to a weighted sum — the model's baseline output when the inputs contribute nothing. It lets a line shift up and down rather than being pinned to the origin. (Unrelated to "bias" in the fairness sense.) See in glossary → bb is the intercept — the baseline price when everything else is zero. Two dials instead of one, and now the line can both tilt and slide up and down.

Add a second input — the house’s age — and the model becomes:

price=w1⋅size+w2⋅age+b\text{price} = w_1 \cdot \text{size} + w_2 \cdot \text{age} + b

Each input gets its own weight, saying how much that input pushes the prediction up or down; the bias sets the floor. This is regression regression A prediction task whose answer is a number (a price, a temperature), as opposed to a category. Usually scored with squared error. See in glossary → : predicting a number from a weighted combination of features feature An individual input the model reads — a house's size, an email's word counts, a pixel. Each feature gets its own weight saying how much it matters. See in glossary → (the inputs). The weights and bias together are the model’s parameters parameters The numbers (weights) inside a model that get adjusted during training. A “7B model” has 7 billion of them. See in glossary → — the formal name for those dials.

Fit it by hand

Two inputs make the model impossible to draw as a single tilted line, so the widget below shows the fit a different way. The main plot is predicted price vs. actual price: every house is a dot, and a dot lands on the diagonal when the prediction is exactly right. A model that’s fit well pulls all the dots onto that line.

Start with only the bias b switched on — a flat guess that ignores the house entirely — then switch on w₁·size, then w₂·age, tuning each knob to pull the dots onto the diagonal and shrink the loss. When you’ve had enough, “Auto-fit” solves for the best settings so you can see how close you got.

Fit house prices by hand
14 homes, each with a size and an age, and a sale price. Switch terms on and drag their knobs to make the model's predictions match reality. (size in thousands of sq ft, age in decades, price in $1000s.)
predicted vs actual price — dots on the line = perfect
actual →↑ predicted
price vs size — model curve at mid-range age
size →↑ price
180
0
0
0.0
0.0
0.0
price ≈ 180
loss L = (1/n) Σ (ŷ − y)² = 35102.9R² = —· poor fit
Auto-fit solves for the best coefficients of whatever terms are on — try beating it by hand.
Start with just b — a flat guess, blind to the house. Add w₁·size and the line tilts; add w₂·age and the leftover spread from age collapses. The coefficients are the knobs; the loss is the score. Learning, in one sentence, is just turning the knobs to shrink the score — and the next sections are about doing it automatically.

A few things worth noticing as you play:

  • With only b, the model is blind — it predicts the same price for a mansion and a shack, and the loss is huge.
  • Adding size tilts the fit and captures most of the signal. Adding age mops up the spread that size alone couldn’t explain.
  • You can keep adding fancier terms — size², size·age — and the loss keeps dropping. Hold that thought: fitting the training dots ever more tightly is not automatically a good thing, and section 18 is about exactly when it goes wrong.

The equation and the score, side by side

The widget shows two things live: the current equation (only the terms you switched on) and the loss — here the mean squared error mean squared error The standard regression loss: average the squared gap between each prediction and its true value. Squaring makes all errors positive and punishes big misses hardest. L = (1/n) Σ (ŷ − y)². See in glossary → , written L=1n∑(y^−y)2L = \frac{1}{n}\sum(\hat{y}-y)^2. That formula just says: for each house, take the gap between the prediction y^\hat{y} and the truth yy, square it so under- and over-shooting both count as error, and average over all nn houses. One number for total wrongness.

The takeaway

A linear model is just a weighted sum of inputs plus a bias, and “fitting” it means turning those knobs until the loss is small. That’s the atom. A neural network neural network A function built by stacking many simple operations — mostly matrix multiplies with nonlinearities between them — whose behavior is shaped by tuning billions of internal numbers (its parameters) from data. See in glossary → , when we get to it, is thousands of these linear models stacked with a twist between them — but the knob-turning, and the loss it’s aimed at, are identical to what you just did by hand. Before we automate the knob-turning, we need to pin down the score itself, because “how wrong are we?” is answered differently for numbers than for categories.