concepts

Runtime Autotune

Status: Public API in netcl.core.autotune (not re-exported from netcl.core)

netcl.core.autotune is a generic runtime tuner for kernel-parameter candidates — tile sizes, register-blocking factors, or even entirely different kernel source variants for the same op. Unlike hand-picked constants baked into a kernel builder, a hot kernel declares a small set of correctness-invariant candidates (same math, different compile-time constants), and autotune() measures each one on the caller's real buffers the first time that (device, kernel, shape) combination is seen. The winner is cached in memory for the process and in a JSON file on disk, so every subsequent run — including on a completely different machine — reuses the measured result instead of re-guessing.

This is the mechanism that lets the same conv2d/Winograd kernel code hit close to peak on very different GPUs (a Quadro P6000 and an AMD gfx902 APU, for example) without the ops code branching on vendor strings: the ops code only ever asks "which of these candidates is fastest here", and the answer is discovered, not hardcoded.

Overview

from netcl.core.autotune import autotune

def bench(tile_size):
    # Build/dispatch the kernel with this candidate and time it.
    ...
    return elapsed_seconds

best_tile = autotune(
    queue,
    kernel="wino_gw_gemm",       # logical name, part of the cache key
    shape_key=(F, C, T, N),      # shape class, part of the cache key
    candidates=[8, 16, 32, 64],
    bench=bench,
    default=16,                  # used if disabled or every candidate fails
)

autotune() returns a single winning candidate — a tile size, a tuple like ("v5", 32) selecting both a kernel variant and a tile size, or anything else hashable-after-JSON-round-trip. The caller is responsible for using the returned value to build/dispatch the real kernel.

Where It Lives

File path: core/autotune.py.
Module path: netcl.core.autotune.
No package-level re-export — import the module directly: from netcl.core.autotune import autotune, clear_cache, is_enabled.
Callers: ops/conv2d_planner.py (Winograd grad_w/grad_in GEMM tile and variant selection, implicit_grad_w OW-tile selection) and ops/conv2d_optimized.py (forward Winograd batched-GEMM tile/variant).

How It Works

The cache key is f"{device_name}|{driver_version}::{kernel}::{shape_key}" — a new GPU, a driver upgrade, or a different problem shape all get a fresh measurement. shape_key is whatever the caller passes (typically a tuple of the dimensions that affect kernel performance).
On a cache miss, every candidate is benched: one untimed warmup call (pays for JIT/kernel-build) followed by reps (default 5) timed calls, keeping the minimum. A candidate whose bench() raises is disqualified (it does not disqualify the whole call — the remaining candidates are still tried).
The fastest surviving candidate is written to the in-memory cache and to the on-disk JSON file, then returned. If every candidate fails, default is returned instead.
On the next call with the same key, the cached value is returned directly — but only if it is still a member of the current candidates list; if the caller changed the candidate set and the old winner fell out of it, the shape is re-tuned rather than trusting a stale value.
Because JSON does not round-trip tuples, autotune() normalizes both the cached value and each candidate through a json.dumps/json.loads pass before comparing — so a candidate ("v5", 32) correctly matches a cached ["v5", 32] read back from disk.

The cache file lives at ~/.cache/netcl/autotune.json on Linux/macOS (respecting XDG_CACHE_HOME) or %LOCALAPPDATA%\netcl\autotune.json on Windows. It is a flat {cache_key: winning_candidate} JSON object, safe to delete (the next run just re-tunes) or commit to a shared location if you want to ship pre-tuned results for known hardware.

Code Example

Registering a kernel-variant-and-tile search, mirroring the real conv2d_optimized.py usage:

from netcl.core.autotune import autotune

def bench(cand):
    variant, tile = cand
    k = build_kernel(ctx, dtype_c, tc=tile, variant=variant)
    queue.finish()
    t0 = time.perf_counter()
    k(queue, gsize, lsize, *args)
    queue.finish()
    return time.perf_counter() - t0

candidates = [("v4", 8), ("v4", 16), ("v4", 32),
              ("v5", 16), ("v5", 32), ("v5", 64)]
variant, tile = autotune(queue, "wino_batched_gemm", (F, C, N, T),
                         candidates, bench, default=("v4", 16))
kernel = build_kernel(ctx, dtype_c, tc=tile, variant=variant)

Disabling autotuning (e.g. for reproducible benchmarking, or on a shared CI runner where per-run tuning noise is undesirable) falls back to default everywhere:

NETCL_AUTOTUNE=0 python train.py

Pointing at a specific cache file (useful for CI or for shipping a pre-tuned cache alongside a deployment):

NETCL_AUTOTUNE_CACHE=/opt/netcl/gpu-a100.json python train.py

Performance & Trade-offs

First-run cost is real but one-time. Each new (device, kernel, shape) combination pays (len(candidates) * (1 + reps)) kernel launches once; on a training script this shows up as a few seconds added to the very first step, invisible afterwards.
The cache is per-shape, not per-model. A model whose spatial size changes every batch (e.g. multi-scale training) re-tunes on every new shape it encounters — for that use case, keep the shape space small (bucket into a handful of sizes) rather than letting every exact resolution create its own cache entry.
Correctness is the caller's responsibility. autotune() has no concept of numerical tolerance; it assumes every candidate computes the identical result and only differs in speed. Candidates must be validated for correctness separately (typically with a dedicated test comparing each variant against a reference implementation) before being added to a candidate list.
This is not the same tool as Autotuner. netcl.profiling.autotuner.Autotuner (the WorkGroupTuner-based one) picks the OpenCL local work-group size for a kernel whose source is fixed. netcl.core.autotune picks between different kernel source variants and tile constants — a superset of what a local-size search can express, because two candidates can be entirely different kernel bodies. The two compose: a kernel can first be selected by netcl.core.autotune, then have its work-group size tuned by Autotuner.