Foundations · /linear

Linear regression

Fit ŷ = wx + b by gradient descent. The red segments are the residuals the loss sums over; the Internals tab shows descent rolling down the actual loss surface toward the normal-equation optimum (dashed line).

curve 120 samples · seed 42

datafilled = train, hollow = testhover to probe · click to pin

step 0MSE —train RMSE —test RMSE —

Data

datasetpoints150noise0.15seedsplitmouse

Model

\overset{y}{^} = w x + b

Mean squared error

L (w, b) = \frac{1}{n} i = 1 \sum n (\overset{y}{^}_{i} - y_{i})^{2} + λ w^{2} = L

Each squared term is the length of a residual segment on the plot, squared.

Gradient descent update

w \leftarrow w - η \frac{\partial L}{\partial w} b \leftarrow b - η \frac{\partial L}{\partial b}

Where it converges (normal equation)

w^{*} = \frac{\sum _{i} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i} ( x _{i} - x ˉ ) ^{2} + nλ} = w^{*} b^{*} = \overset{y}{ˉ} - w^{*} \overset{x}{ˉ} = b^{*}

Gradient descent walks toward this point; it is marked on the loss surface.

What's simplified here

Nothing material — this is textbook least squares. The normal equation shown uses the ridge form with an unpenalized intercept, matching the gradient exactly.
Mini-batch mode reshuffles once per epoch with the page seed, so runs stay reproducible.