15. Training from random weights
Optimise the complete model on a laptop, evaluate held-out loss, and save enough state to reproduce generation.
Training starts with small seeded random values in every learned matrix. The untrained model has the right architecture but no knowledge copied from another model and no statistical understanding of Shakespeare.
Each update performs the same sequence:
- sample 32 windows of 128 training characters;
- run the model to obtain logits at every position;
- calculate cross-entropy against the shifted targets;
- backpropagate gradients through all four blocks;
- clip the total gradient norm;
- update parameters with AdamW;
- periodically estimate loss on fixed training and validation batches.
The laptop configuration
.venv/bin/python lessons/11-language-model-training/train.py
The retained run used:
| Setting | Value |
|---|---|
| Parameters | 799,360 |
| Updates | 1,500 |
| Batch size | 32 |
| Context length | 128 |
| Sampled training tokens | 6,144,000 |
| Peak learning rate | 0.0003 |
| Warm-up | 100 updates |
| Schedule | Cosine decay to 10% |
| Gradient clip norm | 1.0 |
| Device | Apple M1 via PyTorch MPS |
The measured loop took 84.46 seconds on the M1 and included scheduled loss evaluations. Sample generation, checkpoint writing, and process startup were outside that timer.
I find that result genuinely exciting. There is no discrete NVIDIA GPU, hosted training service, or pretrained checkpoint involved—just an Apple laptop, roughly 1.1 MB of Shakespeare text, and code small enough to take apart. The output is not useful prose, but recognisable structure emerges from a surprisingly modest experiment.
Loss and validation
Validation loss fell from 4.6709 before the first update to 2.1298 after update 1,500, a 54.4% reduction. The final model also beat the bigram baseline’s 2.4902 validation loss.
The train and validation values remained close at the end (2.1005 and 2.1298). That does not prove broad generalisation; both splits come from the same Shakespeare collection. It does show that the improvement was not confined to the sampled training batches.
What the checkpoint contains
The 3,299,509-byte checkpoint stores:
- model type and format version;
- complete architecture configuration;
- learned parameter tensors;
- ordered character vocabulary;
- training configuration.
It deliberately does not commit the downloaded corpus or checkpoint to git. The repository retains the source checksum, checkpoint checksum, metrics, configuration, environment, and samples needed to audit the published result.