netcl wiki
concepts

MSE Loss: Mean Squared Error Regression

MSE Loss: Mean Squared Error Regression

Mean Squared Error (mse_loss) is the foundational loss function for continuous regression tasks. It evaluates the mean squared Euclidean discrepancy between predicted values and continuous target values.

[!TIP] Theory & Curriculum Link: For the first-principles derivation of quadratic loss functions, error surfaces, and gradient descent, see Curriculum: What is Machine Learning?.

1. Quick Example: Linear Regression

import numpy as np
import netcl.autograd as ag
import netcl.nn as nn
from netcl.core.device import manager
from netcl.core.tensor import Tensor

q = manager.default("auto").queue

# 1. Define regression architecture
model = nn.Linear(q, in_features=8, out_features=1)

# 2. Input features and continuous ground-truth targets
x = Tensor.from_host(q, np.random.randn(16, 8).astype(np.float32))
y_true = Tensor.from_host(q, np.random.randn(16, 1).astype(np.float32))

# 3. Differentiable forward pass and loss evaluation
with ag.Tape() as tape:
    pred = model(ag.tensor(x))
    diff = pred - ag.tensor(y_true)
    loss = ag.mean(diff * diff)

tape.backward(loss)
print(f"MSE Loss: {loss.value.to_host()[0]:.4f}")

2. Mathematical Formulation

For a mini-batch of $N$ samples with $D$ output coordinates:

$$\hat{\mathbf{y}} \in \mathbb{R}^{N \times D}, \quad \mathbf{y} \in \mathbb{R}^{N \times D}$$

Mean Reduction (Default)

$$\mathcal{L}_{\text{MSE}} = \frac{1}{N \cdot D} \sum_{n=1}^N \sum_{d=1}^D (\hat{y}_{n, d} - y_{n, d})^2$$

Analytical Gradient

The derivative with respect to the model prediction $\hat{\mathbf{y}}$ is linear:

$$\frac{\partial \mathcal{L}_{\text{MSE}}}{\partial \hat{y}_{n, d}} = \frac{2}{N \cdot D} (\hat{y}_{n, d} - y_{n, d})$$

3. Practical Usage: Training Loop

import numpy as np
import netcl.autograd as ag
import netcl.nn as nn
import netcl.optim as opt
from netcl.core.device import manager
from netcl.core.tensor import Tensor

q = manager.default("auto").queue

model = nn.Sequential(
    nn.Linear(q, 4, 16),
    nn.ReLU(),
    nn.Linear(q, 16, 1),
)
optimizer = opt.AdamW(model.parameters(), lr=0.01)

x_batch = Tensor.from_host(q, np.random.randn(32, 4).astype(np.float32))
y_batch = Tensor.from_host(q, np.random.randn(32, 1).astype(np.float32))

for step in range(5):
    with ag.Tape() as tape:
        pred = model(ag.tensor(x_batch))
        diff = pred - ag.tensor(y_batch)
        loss = ag.mean(diff * diff)

    tape.backward(loss)
    optimizer.step()
    optimizer.zero_grad()

    val = float(loss.value.to_host()[0])
    print(f"Step {step+1:02d}: MSE Loss = {val:.4f}")

4. Robust Regression: Smooth L1 / Huber Loss

When dataset targets contain severe outliers, squaring the error produces massive gradients that destabilize training. Smooth L1 Loss transitions smoothly from quadratic error for small residuals to linear error for large residuals:

$$\mathcal{L}_{\text{SmoothL1}}(d) = \begin{cases} \frac{0.5 \cdot d^2}{\beta} & \text{if } |d| < \beta \\ |d| - 0.5 \cdot \beta & \text{otherwise} \end{cases}$$

Where $d = \hat{y} - y$. For bounding box regression in object detection, NetCL provides netcl.nn.loss.weighted_box_smooth_l1_loss.