Metric Learning Losses

Metric learning trains neural embeddings such that semantically similar samples map to nearby coordinates in vector space, while dissimilar samples maintain a strict minimum margin. NetCL provides optimized loss primitives for deep representation learning, face verification, and self-supervised learning.

[!TIP] Theory & Curriculum Link: For the first-principles derivation of representation alignment, uniformity, and InfoNCE loss, see Curriculum: Self-Supervised Representation Learning.

1. Quick Example: Training with Batch Hard Triplet Loss

import numpy as np
import netcl.autograd as ag
import netcl.nn as nn
import netcl.optim as opt
from netcl.core.device import manager
from netcl.core.tensor import Tensor

q = manager.default("auto").queue

# 1. Feature encoder for embeddings (e.g. 64-dimensional output)
encoder = nn.Sequential(
    nn.Linear(q, 128, 64),
    nn.ReLU(),
    nn.Linear(q, 64, 64),
)
optimizer = opt.AdamW(encoder.parameters(), lr=1e-3)

# 2. Batch with class labels (8 classes, 4 samples each = 32 samples)
x = Tensor.from_host(q, np.random.randn(32, 128).astype(np.float32))
labels = Tensor.from_host(q, np.repeat(np.arange(8), 4).astype(np.int32))

# 3. Training step with online hard negative mining
with ag.Tape() as tape:
    embeddings = encoder(ag.tensor(x))
    loss = ag.batch_hard_triplet_loss(embeddings, labels, margin=0.5)

tape.backward(loss)
optimizer.step()
optimizer.zero_grad()

print(f"Triplet loss after step: {loss.value.to_host()[0]:.4f}")

2. Loss Function Selection Guide

3. Triplet Margin Loss

Given an anchor $\mathbf{a}_i \in \mathbb{R}^D$, a positive sample $\mathbf{p}_i \in \mathbb{R}^D$ from the same class, and a negative sample $\mathbf{n}_i \in \mathbb{R}^D$ from a different class:

$$\mathcal{L}_{\text{triplet}} = \frac{1}{N} \sum_{i=1}^N \max\left(0, d(\mathbf{a}_i, \mathbf{p}_i) - d(\mathbf{a}_i, \mathbf{n}_i) + \alpha\right)$$

Where $\alpha$ is the margin parameter and $d(\mathbf{u}, \mathbf{v})$ is the chosen distance metric.

Distance Metrics

• Squared Euclidean Distance (distance="squared", Default):

$$d(\mathbf{u}, \mathbf{v}) = \|\mathbf{u} - \mathbf{v}\|_2^2 = \sum_{j=1}^D (u_j - v_j)^2$$

Avoids square root singularities at zero and yields stable analytical gradients.

• Euclidean Norm (distance="euclidean"):

$$d(\mathbf{u}, \mathbf{v}) = \sqrt{\sum_{j=1}^D (u_j - v_j)^2 + \epsilon} \quad (\epsilon = 10^{-8})$$

Analytical Gradients (distance="squared")

Let $\mathbb{I}_{[i]} = 1$ if triplet $i$ has positive loss ($\mathcal{L}_i > 0$), and $0$ otherwise:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{a}_i} = \frac{\mathbb{I}_{[i]}}{N} \cdot 2 (\mathbf{n}_i - \mathbf{p}_i), \quad \frac{\partial \mathcal{L}}{\partial \mathbf{p}_i} = \frac{\mathbb{I}_{[i]}}{N} \cdot 2 (\mathbf{p}_i - \mathbf{a}_i), \quad \frac{\partial \mathcal{L}}{\partial \mathbf{n}_i} = \frac{\mathbb{I}_{[i]}}{N} \cdot 2 (\mathbf{a}_i - \mathbf{n}_i)$$

4. InfoNCE / NT-Xent Loss

InfoNCE interprets metric learning as a multi-class classification problem over an augmented mini-batch. For query embeddings $\mathbf{q} \in \mathbb{R}^{N \times D}$ and key embeddings $\mathbf{k} \in \mathbb{R}^{N \times D}$:

$$\mathcal{L}_{\text{InfoNCE}} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp\left(\frac{\text{sim}(\mathbf{q}_i, \mathbf{k}_i)}{\tau}\right)}{\sum_{j=1}^N \exp\left(\frac{\text{sim}(\mathbf{q}_i, \mathbf{k}_j)}{\tau}\right)}$$

Where $\text{sim}(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u}^T \mathbf{v}}{\|\mathbf{u}\|_2 \|\mathbf{v}\|_2}$ is cosine similarity and $\tau$ is the temperature scaling factor.

with ag.Tape() as tape:
    q = projection(ag.tensor(view1))
    k = projection(ag.tensor(view2))
    loss = ag.info_nce_loss(q, k, temperature=0.1, normalize=True)

tape.backward(loss)

5. Contrastive Loss (Siamese Networks)

For sample pairs $(\mathbf{x}_{1, i}, \mathbf{x}_{2, i})$ with binary similarity label $y_i \in \{0, 1\}$ ($1 = \text{similar}, 0 = \text{dissimilar}$):

$$\mathcal{L}_{\text{contrastive}} = \frac{1}{2N} \sum_{i=1}^N \left[ y_i d(\mathbf{x}_{1, i}, \mathbf{x}_{2, i})^2 + (1 - y_i) \max\left(0, m - d(\mathbf{x}_{1, i}, \mathbf{x}_{2, i})\right)^2 \right]$$

• Matching pairs ($y_i = 1$) are pulled together.
• Non-matching pairs ($y_i = 0$) are repelled until their distance exceeds margin $m$. Once the distance exceeds $m$, the pair generates zero loss and zero gradient.

6. Batch Hard Triplet Loss

Rather than compiling triplets offline, batch_hard_triplet_loss mines the most informative pairs directly inside the active GPU batch:

For every anchor $i$: 1. Hardest Positive: The sample from the same class with maximum distance from anchor $i$:

$$\mathbf{p}_i^* = \arg\max_{j \in \mathcal{P}_i} d(\mathbf{x}_i, \mathbf{x}_j)$$

2. Hardest Negative: The sample from any other class with minimum distance from anchor $i$:

$$\mathbf{n}_i^* = \arg\min_{k \in \mathcal{N}_i} d(\mathbf{x}_i, \mathbf{x}_k)$$

3. Loss Evaluation:

$$\mathcal{L}_{\text{BHTL}} = \frac{1}{N} \sum_{i=1}^N \max\left(0, d(\mathbf{x}_i, \mathbf{p}_i^*) - d(\mathbf{x}_i, \mathbf{n}_i^*) + \alpha\right)$$

7. L2 Normalization (l2_normalize)

Many metric learning formulations require embeddings to lie on the unit hypersphere:

$$\hat{\mathbf{x}}_i = \frac{\mathbf{x}_i}{\|\mathbf{x}_i\|_2 + \epsilon}$$

The analytical gradient of spherical projection accounts for the tangential subspace constraint:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{x}_i} = \frac{1}{\|\mathbf{x}_i\|_2} \left( \frac{\partial \mathcal{L}}{\partial \hat{\mathbf{x}}_i} - \left( \frac{\partial \mathcal{L}}{\partial \hat{\mathbf{x}}_i} \cdot \hat{\mathbf{x}}_i \right) \hat{\mathbf{x}}_i \right)$$

NetCL's ag.l2_normalize executes this projection in an optimized OpenCL kernel for high-throughput batch evaluation.

Related Documentation

• SSL API Reference: Model implementations including SimCLR and MoCo.
• Curriculum: Self-Supervised Learning: First-principles guide to contrastive representations.
• Concepts: Cross-Entropy Loss: Discrete classification losses.