18. Testing the architecture with an ablation
Remove positional encoding while holding the remaining configuration fixed, then compare complete loss curves.
It is tempting to explain every architectural component as if its value were self-evident. I prefer to change it and measure what happens. An ablation removes one part while holding the rest of the run fixed.
The course’s first ablation disables sinusoidal position vectors. It keeps the same:
- corpus and 90/10 split;
- character vocabulary;
- random seed;
- 799,360 trainable parameters;
- four-block architecture;
- optimiser and learning-rate schedule;
- batch size and context length;
- 1,500 updates;
- validation batches;
- Apple M1 MPS environment;
- generation configuration.
The comparison script reads both saved configurations and refuses to compare them if another recorded value differs.
same corpus + same seed + same training settings
-> reference: positions enabled
-> ablation: positions disabled
-> compare complete validation curves.venv/bin/python lessons/11-language-model-training/train.py \
--device mps \
--without-position-encoding \
--output-directory output/transformer-no-position
.venv/bin/python lessons/13-ablations/compare.pyResult
| Model | Final validation loss |
|---|---|
| Sinusoidal positions | 2.12984738 |
| No explicit positions | 2.29148309 |
Removing positional encoding produced 7.59% higher final loss in this paired run.
The ablated model was actually ahead at update 100 before the reference moved ahead later. Retaining the full curves prevents the article from selecting only the checkpoint that supports a preferred story.
The no-position model did not collapse. Causal masking creates nested prefixes: later representations can aggregate contextual states built from more preceding characters. That offers some structural information even without explicit absolute coordinates.
Treat the 7.59% result as evidence about one controlled seed, not a universal estimate of positional encoding’s value. Repeating both conditions across several seeds would support a stronger conclusion.
Follow the experiment
The architecture switch is part of TinyTransformerConfig. The comparison script removes that one controlled field and rejects other mismatches before comparing results.
Fork the repository and choose a second ablation. Dropout, block count, and model width are all interesting, but each changes a different part of the contract. State exactly what you held fixed and resist turning one run into a universal claim.