1. What a language model predicts

Let’s start without a neural network or any Transformer terminology. A language model repeatedly answers one question:

Given the tokens so far, how likely is each possible next token?

It does not write a paragraph in one operation. It produces one score for every token in its vocabulary, converts those scores into probabilities, chooses one token, appends it to the context, and starts again.

existing text -> scores -> probabilities -> sample one token
      ^                                      |
      +------------ append it ---------------+

From logits to a token

The raw scores are called logits. If the vocabulary contains four characters, a model might return:

logits = torch.tensor([1.2, -0.4, 2.1, 0.7])
probabilities = torch.softmax(logits, dim=-1)

Softmax makes every value non-negative and makes the row sum to one. The highest-probability character is a reasonable choice, but always taking it can become repetitive. Sampling preserves alternatives in proportion to their probabilities.

The first course example avoids a neural network entirely. It uses a known transition table so you can inspect the exact probabilities and verify seeded sampling before learned parameters complicate the picture.

.venv/bin/python lessons/01-next-token-prediction/example.py

The executable path is intentionally small. next_token.py converts logits to probabilities and performs seeded sampling; the lesson example supplies a transition table we can understand without training.

Autoregression

Suppose the current text is:

The cat

The model samples one continuation character, perhaps a space, then evaluates The cat to choose another. A 300-character result requires 300 forward predictions.

That loop is autoregressive generation. It explains three properties that remain true for the finished model:

• generation is sequential even though training processes many positions in parallel;
• every sampled choice becomes input to later choices;
• an early unlikely choice can send the continuation in a different direction.

Parameters, training, and inference

Parameters are the adjustable numbers inside the model. At random initialisation they encode no corpus knowledge. During training, cross-entropy measures how poorly the logits predicted known next tokens, and gradient descent changes the parameters to reduce that error. During inference, the parameters stop changing and the model uses what it learned to continue new prompts.

Keep that separation clear. Training supplies the right answer and updates weights. Generation supplies only a prompt and samples from the model’s predictions.

Try it yourself

Run the example once unchanged. Then alter one row of logits and predict how the probabilities will move before running it again. If your prediction is wrong, inspect the softmax output. That small feedback loop is the foundation for every later experiment in the course.

Next: prepare the training corpus