netcl.text: Tokenization & Chat Formatting

The netcl.text module provides byte-level Byte-Pair Encoding (BPE) tokenization and structured multi-turn conversation formatting. It prepares raw text sequences for transformer models in netcl.nn.transformer and netcl.nn.modern.

[!TIP] Theory & Curriculum Link: For the first-principles derivation of tokenization, embedding matrices, and vocabulary merging, see Curriculum: Tokens & Attention.

1. Quick Example: Training a Tokenizer & Loading onto GPU

import numpy as np
from netcl.text import BPETokenizer, ChatFormat, ChatTurn
from netcl.core.device import manager
from netcl.core.tensor import Tensor

q = manager.default("auto").queue

# 1. Train tokenizer with special chat tokens
tok = BPETokenizer(special_tokens=ChatFormat.SPECIAL_TOKENS)
tok.train("OpenCL and netcl accelerate transformers directly on the GPU.", vocab_size=300)

# 2. Encode text into token IDs and transfer to GPU device memory
token_ids = tok.encode("Transformers on OpenCL.")
x = Tensor.from_host(q, np.array([token_ids], dtype=np.int32))
print(f"Device Tensor Shape: {x.shape}")

2. Mathematical Formulation: Byte-Pair Encoding (BPE)

The BPETokenizer operates strictly at the byte level. The base vocabulary consists of the 256 possible byte values ($0$ to $255$). Consequently, any arbitrary UTF-8 string can be losslessly encoded without out-of-vocabulary (<unk>) errors.

The Merge Algorithm

1. Base Initialization: Initial vocabulary consists of base byte characters:

$$V_0 = \{0, 1, \dots, 255\}$$

2. Regex Pre-Splitting: Text is pre-segmented via SPLIT_PATTERN (GPT-2 regex) to keep punctuation, numbers, and whitespace separated across boundaries.

3. Greedy Frequency Merge: In each step $t$, the most frequent adjacent token pair $(u, v)$ across the corpus is identified:

$$(u^*, v^*) = \arg\max_{(u, v) \in V_t \times V_t} \text{count}(u, v)$$

4. Vocabulary Extension: The pair is merged into a new vocabulary token $w_{\text{new}} = u \circ v$:

$$V_{t+1} = V_t \cup \{w_{\text{new}}\}$$

This process repeats iteratively until $|V_t| = \text{vocab\_size}$ or frequency falls below min_frequency.

The final vocabulary size is:

$$|V| = 256 + |\mathcal{M}| + |\mathcal{S}|$$

Where $|\mathcal{M}|$ is the number of learned merges and $|\mathcal{S}|$ is the number of special tokens.

3. Practical Usage: Training and File I/O

from netcl.text import BPETokenizer, ChatFormat

tok = BPETokenizer(special_tokens=ChatFormat.SPECIAL_TOKENS)
corpus = """
OpenCL is an open standard for parallel programming across GPUs.
netcl utilizes OpenCL for tensors, dynamic autograd, and neural networks.
Transformer decoders utilize RMSNorm, Rotary Embeddings, and SwiGLU.
"""

tok.train(corpus, vocab_size=350, min_frequency=2, verbose=False)
print(f"Trained vocabulary size: {tok.vocab_size}")

# Encode and decode roundtrip
text = "netcl accelerates transformers."
ids = tok.encode(text)
assert tok.decode(ids) == text

# Save and restore from disk
tok.save("my_tokenizer.json")
tok_loaded = BPETokenizer.load("my_tokenizer.json")
assert tok_loaded.encode(text) == ids

BPETokenizer API Reference

4. Chat Formatting & Supervised Fine-Tuning Masking (ChatFormat)

When fine-tuning language models on conversational datasets, loss must be computed strictly on assistant responses. If loss is computed on system prompts or user turns, the model wastes capacity learning to predict the user's questions.

Mathematical Formulation of SFT Masking

For token sequence $x_1, x_2, \dots, x_T$, the autoregressive masked cross-entropy loss is:

$$\mathcal{L}_{\text{SFT}} = -\frac{1}{\sum_{t=1}^{T-1} m_t} \sum_{t=1}^{T-1} m_t \log P(x_{t+1} \mid x_1, \dots, x_t)$$

Where binary supervision mask $m_t \in \{0, 1\}$ is defined as:

$$m_t = \begin{cases} 1 & \text{if } x_{t+1} \text{ is part of an assistant turn or the } \text{<|end|>} \text{ token} \\ 0 & \text{if } x_{t+1} \text{ is part of system prompt, user turn, or role delimiter} \end{cases}$$

Formatting Dialogues with ChatTurn

from netcl.text import BPETokenizer, ChatFormat, ChatTurn

tok = BPETokenizer(special_tokens=ChatFormat.SPECIAL_TOKENS)
tok.train("Hello world. How can I help you? OpenCL is fast.", vocab_size=300)
fmt = ChatFormat(tok)

dialog = [
    ChatTurn(role="system", content="You are a helpful assistant."),
    ChatTurn(role="user", content="What is netcl?"),
    ChatTurn(role="assistant", content="netcl is an OpenCL deep learning framework."),
]

# Render formatted string with special role markers
print(fmt.render(dialog))

# Encode tokens and binary loss mask
token_ids, mask = fmt.encode_conversation(dialog)
print(f"Total tokens: {len(token_ids)}, Supervised tokens (mask=1): {int(mask.sum())}")

Packing Datasets with pack()

Padding short sequences wastes compute. ChatFormat.pack() concatenates conversations into fixed-length blocks (block_size) and fills non-supervised positions with ignore_index = -100:

dataset = [dialog, dialog]
x_batch, y_batch = fmt.pack(dataset, block_size=64, ignore_index=-100)

print(f"Input batch shape: {x_batch.shape}")   # (N, 64)
print(f"Target batch shape: {y_batch.shape}")  # (N, 64)

In this packed batch: - $x$ contains input tokens. - $y$ contains target tokens shifted by 1 position. - Unsupervised positions (system prompts, user questions, padding) have value -100, which netcl.nn.functional.cross_entropy automatically ignores during backpropagation.

Related Documentation

• Concepts: Transformer: Decoder blocks and rotary attention.
• Curriculum: Tokens & Attention: Theoretical derivation of BPE and embeddings.
• Concepts: Cross-Entropy: Cross-entropy loss with ignore_index masking.