netcl wiki
concepts

Tape

Tape

Status: Public API in netcl.autograd.Tape

The Tape is netcl's autograd graph. It records a DAG of Node objects — one per intermediate tensor — as ops are executed. Walking the tape in reverse is how netcl computes gradients.

Overview

Each Node on the tape carries:

field description
value the Tensor produced by the op
grad_fn callable: (upstream_grad) → [grad_per_input, ...]
parents list of input Nodes this op consumed
requires_grad True if any input had it True
grad populated during backward(); None until then
op_name debug identifier

When tape.backward(loss) is called, the engine builds a topological order from loss and walks it in reverse. For each node, it calls grad_fn(upstream) and accumulates the result into each parent's grad field.

Usage

import netcl.autograd as ag

tape = ag.Tape()
with tape:                          # sets tape as current; clears it on exit
    pred  = model(x_node)
    loss  = ag.mse_loss(pred, y_node)

tape.backward(loss)                 # reverse pass
optimizer.step()
optimizer.zero_grad()
# tape goes out of scope → __del__ frees GPU buffers automatically

Or more concisely — with ag.Tape() as tape: in one line:

with ag.Tape() as tape:
    loss = criterion(model(x), y)
tape.backward(loss)

Explicit tape parameter

Some ops accept an optional tape= argument for cases where no global tape is active (e.g. inside a no_grad block or a nested call):

tape = ag.Tape()
out  = ag.relu(x_node, tape=tape)
loss = ag.mse_loss(out, target, tape=tape)
tape.backward(loss)

Gradient accumulation across multiple forward passes

optimizer.zero_grad()
for branch in [anchor, positive, negative]:
    tape = ag.Tape()
    with tape:
        emb  = encoder(branch)
        loss = partial_triplet_loss(emb)
    tape.backward(loss)          # grads accumulate in param .grad fields
optimizer.step()

Automatic memory cleanup

After tape.backward() the computation graph is no longer needed. netcl frees it automatically when the Tape object goes out of scope:

backward()         → GPU kernels enqueued
optimizer.step()   → runs
tape = ag.Tape()   → old tape reassigned → Tape.__del__ fires
  ├ node.grad_fn = None  (releases closure-captured activation tensors)
  ├ node.parents = []    (breaks reference chain → intermediate buffers freed)
  └ tape.nodes.clear()   (releases tape's own list references)
next forward()     → starts; GPU buffers already back in pool

The cleanup happens in the idle window between optimizer end and the next forward — no CPU stall during backward, no manual del required.

Thread safety

The tape is thread-local. Each thread has its own current tape, accessed via get_current_tape() / set_current_tape(). Multi-threaded data parallel training uses separate tapes per replica and accumulates gradients explicitly.

Performance notes

  • Tape.__enter__ calls queue.flush() before returning to submit any pending optimizer kernels from the previous step. This prevents the GPU command-queue from filling up between iterations.
  • A no_grad() context suppresses recording entirely — no Node is allocated, no grad_fn closure is built. Use it for evaluation and inference.
  • detect_anomaly(True) inserts NaN/inf checks at every node. It is expensive; turn it on only when debugging.

Code example: full training step

import netcl as nc
import netcl.autograd as ag

model = build_model(queue)
optimizer = nc.optim.AdamW(model.parameters(), lr=1e-3)

for x_np, y_np in dataloader:
    x = ag.tensor(nc.Tensor.from_host(queue, x_np), requires_grad=False)
    y = ag.tensor(nc.Tensor.from_host(queue, y_np), requires_grad=False)

    with ag.Tape() as tape:
        pred = model(x)
        loss = ag.cross_entropy(pred, y, tape=tape)

    tape.backward(loss)
    optimizer.step()
    optimizer.zero_grad()
    # tape goes out of scope → automatic GPU buffer cleanup

See also