netcl.runtime — Cache, Graph, Capture, Scheduler

The netcl.runtime package groups the low-level support layer that the rest of netcl builds on. It owns the on-disk OpenCL program cache, a static-graph pipeline builder with buffer reuse, a CUDA-Graph-style kernel capture/replay mechanism, and a lightweight scheduler for sequencing async ops.

Long-form imports. netcl/runtime/__init__.py is empty. Every symbol must be imported from its submodule:

python from netcl.runtime.cache import ProgramCache, cacher from netcl.runtime.graph import Graph, GraphExecutor, TensorRef from netcl.runtime.capture import KernelCapture, capture_disabled, get_capture from netcl.runtime.scheduler import Scheduler, ExecutionPlan

Module Map

Kernel Cache (runtime/cache.py)

ProgramCache compiles an OpenCL program from source, caches the binary to disk keyed on sha256(source + device identity), and returns the cached binary on subsequent calls.

from netcl.runtime.cache import cacher, ProgramCache

# Using the global singleton:
prg = cacher.get_program(ctx, src_code, device)

# Or create your own instance with a custom cache directory:
my_cache = ProgramCache(cache_dir="/tmp/my_kernels")
prg = my_cache.get_program(ctx, src_code, device)

ProgramCache(cache_dir=None)

ProgramCache.get_program(ctx, src_code, device)

Returns a compiled cl.Program. If a matching binary exists on disk it is loaded directly (skipping compilation). If the cached binary is corrupt or stale the method falls back to source compilation and overwrites the cache entry.

The global cacher singleton is created at import time with cache_dir=~/.cache/netcl. All netcl ops use it internally — you only need to call it directly when writing custom OpenCL kernels.

Graph Pipeline (runtime/graph.py)

Graph lets you describe a sequence of tensor operations as named nodes without executing them. GraphExecutor then runs the graph in topological order, allocating intermediate buffers from a BufferPool and releasing them as soon as their last consumer has finished.

from netcl.core.memory import BufferPool
from netcl.runtime.graph import Graph, GraphExecutor

pool = BufferPool(queue)
graph = Graph()

def add_fn(inputs, out):
    # inputs: List[Tensor], out: Tensor — fill out in place
    ...

ref_a = graph.add_op("add", add_fn, [tensor_x, tensor_y], shape=(4,), dtype="float32")

# ref_a is a TensorRef — a lightweight placeholder for the output of "add".
# Pass it as input to subsequent ops:
def relu_fn(inputs, out):
    ...

ref_b = graph.add_op("relu", relu_fn, [ref_a], shape=(4,), dtype="float32")

executor = GraphExecutor(pool)
result = executor.run(graph)   # returns the final Tensor

Graph

add_op parameters:

GraphExecutor(pool)

run performs a topological sort of the ops (raises ValueError on cycles), calls each op's fn with resolved inputs and a freshly-allocated output buffer from the pool, and releases intermediate buffers back to the pool when their refcount drops to zero. Returns the last op's output as a live Tensor.

Kernel Capture (runtime/capture.py)

KernelCapture is a lightweight CUDA-Graph-style mechanism for OpenCL. During a recording phase every cl.Kernel.__call__ is intercepted and stored. A later replay enqueues the same sequence with near-zero Python overhead.

from netcl.runtime.capture import KernelCapture, capture_disabled

cap = KernelCapture()
cap.begin()              # start intercepting cl.Kernel calls
run_forward_pass()       # launches are recorded AND executed normally
n = cap.end()            # stop recording; returns number of captured launches
print(f"captured {n} launches")

# Subsequent iterations (no Python overhead per launch):
cap.replay()

KernelCapture methods

KernelCapture properties

capture_disabled

A context manager that temporarily suppresses recording inside a begin()/end() block. Use it around sub-calls that must run normally (e.g. buffer allocations) without being captured:

cap.begin()
with capture_disabled():
    buf = allocate_scratch_buffer(queue)   # not captured
run_kernels(buf)                           # captured
cap.end()

get_capture()

Returns the process-wide KernelCapture singleton, creating it on first call. Useful when multiple call sites need to share one capture object without passing it explicitly.

Buffer rebinding

replay() re-enqueues the exact arguments from the recording phase. When input or output buffers change between iterations, use replay_with_new_buffers(buffer_map). The map keys are cl.Buffer.int_ptr values (GPU memory addresses), not Python object ids — this avoids false matches when a buffer pool reuses the same Python wrapper for a different underlying allocation.

Scheduler (runtime/scheduler.py)

Scheduler is a minimal sequencer that collects callables or cl.Event-like objects into ExecutionPlans and runs them in submission order.

from netcl.runtime.scheduler import Scheduler, ExecutionPlan

sched = Scheduler(async_mode=False)

plan_a = ExecutionPlan(description="h2d copy")
plan_b = ExecutionPlan(description="compute")

sched.submit(lambda: copy_to_device(x), plan=plan_a)
sched.submit(lambda: run_kernel(x),     plan=plan_b)

events = sched.run(wait=True)   # execute all plans, wait for every returned event

ExecutionPlan(ops=[], description="")

A plain dataclass: a list of callables/events plus an optional label. Created automatically by Scheduler.submit when no plan is provided.

Scheduler(async_mode=False)

After run, the internal plan list is cleared so the scheduler can be reused. Callables that return a non-None value are treated as events and added to the wait list; callables that return None are fire-and-forget.

See also

• profiling API — EventTimer, KernelProfiler, AutoTuner, Timer, ProfileContext
• Tensor — the on-device value type that Graph and KernelCapture operate on
• nn API — how Conv2d, BatchNorm2d, and the other layers use cacher internally