Linear Regression

Linear regression is often introduced as the simplest supervised learning model, but it is more than a toy. It gives a clean view of the whole machine learning loop: choose a model, define a loss, fit parameters, and inspect the residuals.

The model assumes the target is approximately a weighted sum of the inputs:

$$ \hat{y} = w_0 + w_1x_1 + w_2x_2 + \cdots + w_dx_d $$

The weights say how much each feature contributes, and the intercept w_0 handles the baseline prediction when all features are zero.

The loss

A common objective is mean squared error:

$$ \operatorname{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 $$

Squaring the residuals does two useful things. It makes positive and negative errors count the same, and it punishes large mistakes more than small mistakes.

Closed form and gradient descent

For ordinary least squares, there is a direct solution:

$$ w = (X^\top X)^{-1}X^\top y $$

That formula is elegant, but it is not always the best way to train. In larger systems we usually think in terms of gradient descent:

import numpy as np

def fit_linear_regression(X, y, steps=1000, lr=0.05):
    X = np.c_[np.ones(len(X)), X]
    w = np.zeros(X.shape[1])

    for _ in range(steps):
        y_hat = X @ w
        grad = (2 / len(X)) * X.T @ (y_hat - y)
        w -= lr * grad

    return w

This loop is the same pattern used in much larger models: predict, measure error, compute a gradient, update parameters.

What to check

Interpretation

Linear regression is useful because it is readable. A weight is not a complete causal story, but it is often a good starting hypothesis:

If all other features are held fixed, changing this feature by one unit changes the prediction by roughly this weight.

That sentence is doing a lot of work. It depends on the data, the feature design, and whether the model assumptions are reasonable.

A small mental model

Linear regression draws the best flat surface through the data. In one dimension that surface is a line. In many dimensions it is a hyperplane. Training is just the process of moving that surface until the average squared vertical distance to the data is small.