netcl wiki
concepts

Cross-Entropy Loss: Multi-Class Classification

Cross-Entropy Loss: Multi-Class Classification

Cross-Entropy Loss is the standard loss function for multi-class classification tasks. It fuses the Softmax probability distribution with Negative Log-Likelihood (NLL). In netcl, the calculation is numerically stabilized via the Log-Sum-Exp trick to prevent floating-point overflow and underflow on GPU hardware.

[!TIP] Theory & Curriculum Link: For the first-principles derivation of loss functions, optimization, and analytical gradients, see Curriculum: What is Machine Learning? and Curriculum: Calculus & Gradients.

1. Quick Example: 10-Class Classification

import numpy as np
import netcl.autograd as ag
import netcl.nn as nn
from netcl.core.device import manager
from netcl.core.tensor import Tensor

q = manager.default("auto").queue

# 1. Multi-class classification network
model = nn.Sequential(
    nn.Linear(q, 64, 32),
    nn.ReLU(),
    nn.Linear(q, 32, 10),
)

# 2. Batch: 4 samples, 10 classes, target class indices (0 to 9)
x = Tensor.from_host(q, np.random.randn(4, 64).astype(np.float32))
y_indices = Tensor.from_host(q, np.array([2, 0, 9, 1], dtype=np.int32))

# 3. Differentiable forward pass and loss evaluation
with ag.Tape() as tape:
    logits = model(ag.tensor(x))
    loss = ag.cross_entropy(logits, ag.tensor(y_indices))

tape.backward(loss)
print(f"Cross-Entropy Loss scalar: {loss.value.to_host()[0]:.4f}")

2. Mathematical Formulation

For a batch of $N$ samples and $C$ classes, let $z_{b, c}$ denote the unnormalized logit score produced by the model for sample $b$ and class $c$. The true target label is $y_b \in \{0, \dots, C-1\}$.

Softmax Probability Distribution

The predicted probability for class $c$ is given by the Softmax function:

$$P(y = c \mid \mathbf{z}_b) = \frac{\exp(z_{b, c})}{\sum_{c'=1}^C \exp(z_{b, c'})}$$

Negative Log-Likelihood and the Log-Sum-Exp Trick

A naive implementation of $-\log P(y = y_b \mid \mathbf{z}_b)$ computes $\exp(z)$, which overflows float32 when logits exceed $+88$.

To guarantee numerical stability, netcl shifts logits by the row maximum:

$$m_b = \max_{c=1, \dots, C} z_{b, c}$$

Using the identity:

$$\log \sum_{c'=1}^C \exp(z_{b, c'}) = m_b + \log \sum_{c'=1}^C \exp(z_{b, c'} - m_b)$$

The mean Cross-Entropy loss across the batch is:

$$\mathcal{L} = \frac{1}{N} \sum_{b=1}^N \left[ m_b + \log \left( \sum_{c=1}^C \exp(z_{b, c} - m_b) \right) - z_{b, y_b} \right]$$

Analytical Gradient

The derivative of Cross-Entropy with respect to logits has an exceptionally clean analytical form:

$$\frac{\partial \mathcal{L}}{\partial z_{b, c}} = \frac{1}{N} \left( P(y = c \mid \mathbf{z}_b) - \mathbb{I}_{[c = y_b]} \right)$$

Where $\mathbb{I}_{[c = y_b]} = 1$ if $c$ is the correct class and $0$ otherwise. The backward pass simply subtracts $1.0$ from the predicted probability of the true class!

3. Practical Usage: Training Loop & Padding Masking

import numpy as np
import netcl.autograd as ag
import netcl.nn as nn
import netcl.optim as opt
from netcl.core.device import manager
from netcl.core.tensor import Tensor

q = manager.default("auto").queue

model = nn.Linear(q, 16, 4)
optimizer = opt.AdamW(model.parameters(), lr=0.01)

x_batch = Tensor.from_host(q, np.random.randn(8, 16).astype(np.float32))
y_batch = Tensor.from_host(q, np.random.randint(0, 4, size=(8,)).astype(np.int32))

for step in range(5):
    with ag.Tape() as tape:
        logits = model(ag.tensor(x_batch))
        loss = ag.cross_entropy(logits, ag.tensor(y_batch))

    tape.backward(loss)
    optimizer.step()
    optimizer.zero_grad()

    loss_scalar = float(loss.value.to_host()[0])
    print(f"Step {step+1:02d}: Loss = {loss_scalar:.4f}")

Masking Padding in Language Models (ignore_index)

When training autoregressive language models, padding tokens are ignored using ignore_index = -100:

from netcl.autograd.shape_ops import cross_entropy_indices

# logits: (Batch * Seq_Len, Vocab_Size)
# targets: (Batch * Seq_Len,) with -100 at padding positions
with ag.Tape() as tape:
    loss = cross_entropy_indices(logits_node, targets_array, ignore_index=-100)

tape.backward(loss)

4. Critical Engineering Rules

  1. Do Not Apply Softmax Inside Your Model: The final layer of your neural network must output raw, unnormalized logits. Never add an explicit nn.Softmax() layer at the end of your network; ag.cross_entropy handles Softmax internally with the Log-Sum-Exp trick.
  2. Data Types: Logits must be float32 (or float16 under AMP). Target indices must be integers (int32).