4. Constructing inputs, targets, and batches

Now let’s turn one long token stream into examples the model can learn from. A next-token training example is simply one sequence viewed twice with a one-position offset.

For a short context:

input:  First Citizen:\nB
target: irst Citizen:\nBe

At input position zero, F predicts i. At position one, the context Fi predicts r. A context length of 16 therefore supplies 16 related prediction targets, not one.

Sampling a batch

The batch builder chooses random starting offsets from the training token stream. For each offset s and context length T:

inputs = tokens[s:s + T]
targets = tokens[s + 1:s + T + 1]

Stacking several windows produces tensors shaped:

inputs:  batch × time
targets: batch × time

The executed example creates tensors shaped (4, 16) and prints the decoded first pair so shape assertions remain connected to readable text.

.venv/bin/python lessons/03-training-examples/example.py

Teacher forcing without future leakage

During training, the correct earlier characters are present in the input even if the model would have generated something else. This is commonly called teacher forcing.

The target tensor is also in memory, but it must not become visible to the prediction that is being scored. Later, causal masking will prevent an attention query at position t from looking at positions after t.

Keep data alignment tests separate from attention tests. If the targets are shifted incorrectly, a perfect mask and correct Transformer will still learn the wrong task.

Follow the code

batching.py samples offsets and constructs the shifted tensors. The lesson example decodes the first pair again so we can inspect meaning as well as tensor shape.

Change the context length from 16 to 4, then to 32. Print the first decoded pair each time. You should see the same one-character shift, but a different amount of history available to each prediction.

Next: train a bigram baseline