netcl wiki
architecture

Architecture: Distributed & Multi-Device Execution

Architecture: Distributed & Multi-Device Execution

netcl's distributed engine is engineered specifically for single-workstation multi-GPU execution. It allows machine learning practitioners to train models in parallel across multiple OpenCL accelerators on a single machine with zero external cluster dependencies, zero proprietary network fabric requirements, and minimal setup overhead.

The Workstation Multi-GPU Architecture

In enterprise datacenters, multi-GPU clusters rely on proprietary NVLink bridges and NCCL. On consumer and workstation hardware, developers frequently have mismatched GPUs (for example, an AMD Radeon paired with an Intel Arc card, or two PCIe cards without hardware interconnects).

Because OpenCL contexts cannot directly share pointers across different devices or platforms, netcl utilizes a robust host-staged reduction topology:

flowchart TD subgraph Host["Host Process (CPU RAM)"] Ring["Shared-Memory Ring Buffer / Multiprocess Queues"] Reducer["Synchronous Host Reducer (all_reduce: sum / mean)"] Ring --- Reducer end subgraph Dev0["Replica 0 (Worker Process)"] GPU0["Device: AMD Radeon RX 7900
Local Batch Slice: B / 2
OpenCL Context & Command Queue
Model Forward & Tape Backward"] end subgraph Dev1["Replica 1 (Worker Process)"] GPU1["Device: Intel Arc A770
Local Batch Slice: B / 2
OpenCL Context & Command Queue
Model Forward & Tape Backward"] end GPU0 -->|"D2H Gradients"| Ring Ring -->|"H2D Synchronized"| GPU0 GPU1 -->|"D2H Gradients"| Ring Ring -->|"H2D Synchronized"| GPU1

Each physical device runs in an isolated worker subprocess. Worker processes maintain their own OpenCL command queues, forward models independently on their batch slice, and execute backward passes through their local autograd tape. Gradients are then synchronized across workers via fast shared memory.

Key Benefits of Host-Staged Architecture

  1. Vendor Independence: Combine cards from different manufacturers (AMD, Intel, NVIDIA, Apple Silicon) in the same training job without driver conflicts.
  2. Zero Configuration: No MPI daemons, no network subnetting, and no cluster orchestration required. It runs directly within a Python virtual environment.
  3. Driver Isolation: If one GPU encounters a driver reset or memory timeout, it is isolated to its worker subprocess without corrupting the host OS.
  4. Predictable Performance: Host-to-Device (H2D) and Device-to-Host (D2H) memory transfers occur over PCIe. For standard convolutional networks, transformers, and MLPs, the PCIe transfer time is a fraction of the compute time.

Core Collective Primitives

Collective communication operations live in netcl.distributed.collectives:

Primitive Mechanism Primary Use Case
all_reduce(tensors, op="mean") Pulls replica gradients via D2H, computes elementwise reduction in host memory, and transfers result back via H2D. Synchronizing parameter gradients before optimizer step.
broadcast(tensors, root=0) Reads tensor from root device to host memory and writes it to all other replica devices. Distributing initial weights or re-synchronizing parameters.
scatter(tensor, chunks) Splits a master batch along the sample axis into $N$ device slices. Sharding input batches across workers.
gather(tensors) Concatenates per-device tensors along axis 0 into a single host tensor. Aggregating evaluation predictions or validation metrics.

When devices share an identical OpenCL context (such as identical dual GPUs from the same vendor), an optimized pairwise device reduction kernel (all_reduce_p2p) is available to bypass host memory.

The Training Step Lifecycle (data_parallel_step)

The high-level orchestrator data_parallel_step executes a synchronized iteration:

from netcl.distributed import prepare_replicas, data_parallel_step

# 1. Initialize replicas across target devices
replicas = prepare_replicas(model, opt, n_replicas=2, devices=["gpu:0", "gpu:1"])

# 2. Iterate through data batches
for x_batch, y_batch in loader:
    loss = data_parallel_step(
        replicas,
        (x_batch, y_batch),
        loss_fn=lambda logits, y: ag.cross_entropy(logits, y),
        tape_factory=ag.Tape,
    )

Under the hood, data_parallel_step coordinates: 1. Batch Sharding: Input features and labels are split evenly along batch dimension $B$. 2. Parallel Forward & Backward: Each replica executes model.forward() and tape.backward(loss) on its local command queue. 3. Gradient Synchronization: sync_grads computes the mean gradient across all replicas using all_reduce. 4. Optimizer Update: Each replica updates its local weights using its synchronized gradients.

Performance Considerations

  • Compute-to-Transfer Ratio: For dense models (e.g. ResNet, Vision Transformers, and modern MLPs), computational FLOPs vastly exceed the byte transfer size, yielding near-linear scaling across 2 to 4 GPUs.
  • Pinned Host Memory: By utilizing pinned host staging buffers (PinnedBufferPool), H2D and D2H PCIe copies run asynchronously without blocking Python execution.