netcl wiki
concepts

Self-Supervised Learning & Deep Clustering

Self-Supervised Learning & Deep Clustering

Located at: netcl.ssl (Methods: methods.py, Deep Clustering: deep_cluster.py, Augmentations: views.py, Evaluation: evaluate.py)

Self-Supervised Learning (SSL) trains high-quality visual and semantic representations on unlabelled datasets. The netcl.ssl module provides optimized architectures for contrastive, non-contrastive, and cluster-based SSL methods.

[!TIP] Theory & Curriculum Link: For the first-principles derivation of representation collapse, InfoNCE alignment, and BYOL stop-gradient dynamics, see Curriculum: Self-Supervised Representation Learning.

Method Selection Guide

Method Requires Negative Pairs? Requires Momentum Target Encoder? Optimal Use Case
SimCLR Yes (in-batch) No High-memory setups supporting large batches ($B \ge 128$). Clean and robust.
MoCo Yes (memory queue) Yes (EMA) Small-batch setups ($B \le 64$) where a FIFO queue caches negative representations.
BYOL No Yes (EMA) When defining or mining negative pairs is undesirable or domain-inappropriate.
SimSiam No No (Stop-gradient) Simplest bootstrap architecture; minimal parameter footprint and zero EMA overhead.
BarlowTwins No No Redundancy reduction: decorrelates feature dimensions across cross-correlation matrix.
VICReg No No Explicitly prevents collapse via variance, invariance, and covariance regularization bounds.
DeepCluster No No Pairs iterative GPU KMeans clustering with supervised classification training.
SCAN No No Graph-based clustering: forces nearest neighbors in feature space to share identical cluster assignments.

1. Quick Example: SimCLR Pretraining Loop

import numpy as np
import netcl
import netcl.autograd as ag
from netcl import ssl
from netcl.core.device import manager
from netcl.nn import Sequential, Linear, ReLU
import netcl.optim as optim

q = manager.default("auto").queue

# 1. Define neural backbone encoder
encoder = Sequential(
    Linear(q, 128, 256), ReLU(),
    Linear(q, 256, 128),
)

# 2. Instantiate SimCLR wrapper (automatically attaches MLP projection head)
method = ssl.SimCLR(
    encoder,
    feature_dim=128,          # Encoder output dimensionality
    projection_dim=64,        # Projection head bottleneck dimension
    temperature=0.1,          # InfoNCE temperature
    queue=q,
)
opt = optim.AdamW(method.parameters(), lr=1e-3, weight_decay=1e-4)

# 3. Multi-view GPU augmentation generator
views = ssl.MultiView.two_crop(out_size=(32, 32), seed=0)

# 4. Self-Supervised Training Step
# Simulate batch of 32 images
raw_batch = Tensor.from_host(q, np.random.randn(32, 128).astype(np.float32))
v1, v2 = views(raw_batch)

with ag.Tape() as tape:
    # Forward both augmented views and compute InfoNCE loss
    loss = method.training_step([v1, v2])

tape.backward(loss)
opt.step()
opt.zero_grad()

# Update internal step counters and EMA momentum buffers
method.on_step_end()
print(f"SimCLR Step Complete. Loss: {loss.value.to_host()[0]:.4f}")

The InfoNCE Mathematical Objective

SimCLR minimizes the normalized temperature-scaled cross-entropy loss for a positive pair $(z_i, z_j)$:

$$\mathcal{L}_{\text{NT-Xent}}(i, j) = -\log \frac{\exp\left(\frac{\text{sim}(z_i, z_j)}{\tau}\right)}{\sum_{k=1}^{2B} \mathbb{I}_{[k \ne i]} \exp\left(\frac{\text{sim}(z_i, z_k)}{\tau}\right)}$$

Where $\text{sim}(u, v) = \frac{u^T v}{\|u\|_2 \|v\|_2}$ is cosine similarity and $\tau$ is the temperature scaling factor.

2. Non-Contrastive Learning: BYOL

If you are training with smaller batch sizes and want to eliminate negative pair repulsion, use BYOL:

byol = ssl.BYOL(
    encoder,
    feature_dim=128,
    projection_dim=64,
    pred_dim=64,
    momentum=0.996,           # Target network EMA decay rate
    queue=q,
)

BYOL minimizes the mean squared error between the normalized online prediction and the target network output:

$$\mathcal{L}_{\text{BYOL}} = 2 - 2 \cdot \frac{\langle q_\theta(z_\theta), z'_\xi \rangle}{\|q_\theta(z_\theta)\|_2 \cdot \|z'_\xi\|_2}$$

Target parameters $\xi$ update slowly via Exponential Moving Average:

$$\xi \leftarrow \mu \xi + (1 - \mu) \theta \quad (\mu \approx 0.996)$$

3. Deep Clustering Integration

When your goal is to discover natural categories directly from unlabelled data, DeepCluster links feature representations directly to GPU clustering in netcl.cluster:

dc = ssl.DeepCluster(
    encoder,
    feature_dim=128,
    n_clusters=10,
    recluster_every=1,        # Recompute KMeans centroids every N epochs
    balanced=True,            # Uses BalancedKMeans to prevent cluster collapse
    queue=q,
)

4. Evaluation Probes: Validating Feature Quality

Because self-supervised losses can decrease even if representations fail to capture semantics, netcl.ssl.evaluate provides diagnostic evaluation tools:

Online kNN Monitor (Zero Additional Training)

Evaluates validation accuracy every few epochs using GPU k-nearest-neighbors:

from netcl.ssl import embed_dataset, knn_monitor

X_train, y_train = embed_dataset(encoder, train_loader)
X_test,  y_test  = embed_dataset(encoder, test_loader)

acc = knn_monitor(X_train, y_train, X_test, y_test, k=20, temperature=0.1)
print(f"k-NN Test Accuracy: {acc * 100:.2f}%")

Linear Probing

The standard benchmark: freeze encoder weights and train a single linear classifier on top:

from netcl.ssl import linear_probe

probe_acc = linear_probe(X_train, y_train, X_test, y_test, epochs=15, lr=0.01)
print(f"Linear Probe Accuracy: {probe_acc * 100:.2f}%")