netcl wiki
api

netcl.trainer — Training Loop Wrapper

netcl.trainer — Training Loop Wrapper

netcl.trainer.Trainer is a small, optional convenience wrapper around the manual Tape training loop shown in the MNIST tutorial. It exists for the common classification-training case (forward -> loss -> backward -> step -> loop) so you do not have to hand-write the tape/backward/step boilerplate for every script; anything more custom (multi-input models, non-classification metrics, custom loss shapes) is usually still clearer as an explicit with ag.Tape() as tape: loop.

from netcl.trainer import Trainer

Trainer(model, optimizer, device_queue=None, mixed_precision=False, grad_clip=None, loss_fn=None, metrics_stride=10)

Parameter Type Default Meaning
model Callable required Anything callable as model(x_node) -> logits (a Node or a Tensor).
optimizer optimizer instance required Any netcl.optim optimizer.
device_queue queue | None None Defaults to manager.default()'s queue.
mixed_precision bool False Wraps the forward in autocast and scales the loss with AMPGradScaler. Forced off on the CPU backend.
grad_clip float | None None Global-norm gradient clipping value, applied every step.
loss_fn Callable | None None Default loss if fit()/evaluate() are not given one explicitly; falls back to functional.cross_entropy.
metrics_stride int 10 Only compute/report train accuracy every metrics_stride-th step (an accuracy computation forces a to_host() sync, so this trades metric resolution for throughput).

fit(train_loader, val_loader=None, epochs=1, loss_fn=None, log_every=1)

Runs the training loop for epochs epochs, printing per-epoch loss/accuracy/throughput, and calling evaluate(val_loader, ...) every log_every epochs if val_loader is given.

from netcl.trainer import Trainer
from netcl.optim import AdamW
from netcl.data.dataloader import DataLoader

opt = AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
trainer = Trainer(model, opt, grad_clip=1.0, metrics_stride=10)

train_loader = DataLoader(train_ds, batch_size=128, num_workers=4, use_shared_mem=True)
val_loader = DataLoader(val_ds, batch_size=256, num_workers=2, shuffle=False)

trainer.fit(train_loader, val_loader, epochs=10)

train_loader can yield either plain NumPy (xb, yb) tuples or, if you called train_loader.bind_queue(queue) beforehand, GpuBatch tuples whose xb is already a device Tensor. Trainer accepts both transparently — only extra dataset fields beyond (xb, yb) are ignored by fit().

Pipeline (why fit() looks the way it does)

Each step in fit() follows the same async pattern the WIDER-FACE reference training script uses, not a naive forward(); backward(); step():

  1. Forward is enqueued (_forward, without waiting for it to finish).
  2. Drain (interruptible_finish, OpenCL backend only): waits for the GPU to catch up to the enqueued forward. This is the point at which the previous step's backward + optimizer kernels are guaranteed complete, so their pool buffers can be safely released — and it is the natural point to read loss.value.to_host() for the metrics/progress bar, because the forward is already done by then.
  3. Metrics (every metrics_stride-th step): one to_host() for the loss, one for the logits.
  4. Backward + step are enqueued (not waited on) — tape.backward(loss) followed by optimizer.step(). If the optimizer's step accepts a max_norm keyword (checked once at Trainer.__init__ via inspect.signature), grad_clip is applied through that fused kernel path instead of a separate clip_grad_norm call — see AdamW: Fused Clip + Step.
  5. The loop then moves on to the next step's data loading and forward-enqueue while the GPU is still executing this step's backward + optimizer — the host never waits for GPU work it does not immediately need.

A gc.collect() runs once per step: the autograd graph is cyclic (Node objects hold closures that reference their parents), so relying on the default generational GC to collect it eventually causes buffers to pile up in bursts instead of returning to the BufferPool steadily.

evaluate(data_loader, loss_fn)

Runs one pass over data_loader in eval mode (model.eval()/training=False, restored afterwards) and prints average loss and accuracy. Does not run backward or touch the optimizer.

Errors and Edge Cases

  • fit() raises RuntimeError("non-finite loss at step {i}") on a NaN/Inf loss at a metrics-stride step — training does not silently continue with a broken loss.
  • Labels can be integer class indices (shape == (N,)) or already one-hot (shape == (N, num_classes)); Trainer converts as needed and validates the label range against the model's output width, raising ValueError on an out-of-range label.
  • mixed_precision=True on the CPU backend is silently downgraded to False (with a printed notice) rather than raising, since AMP has no meaning without device kernels.

See also

  • MNIST with MLP — the explicit-Tape alternative to Trainer, useful when you need more control than fit() gives you.
  • optim — optimizers, including the fused-clip step(max_norm=...) path Trainer prefers automatically.
  • data — DataLoader and bind_queue(), the GPU-prefetch path Trainer.fit consumes transparently.
  • AMP — the mixed-precision machinery mixed_precision=True wires up.
  • BufferPool — why fit() calls gc.collect() once per step.
  • Tape — the autograd recorder fit() drives under the hood.