19. Trying other corpora and scaling the model
Change the learned voice with new text, design fair corpus comparisons, and understand what grows on the path to a real LLM.
Once your fork can reproduce the Shakespeare run, changing the text is a wonderfully direct next experiment. Tiny Shakespeare leaves a visible fingerprint on the output: uppercase speakers, colons, short dramatic lines, and invented names. Train the same architecture on a different corpus and you can see that its style comes from data rather than a collection of rules hidden in the code.
Good next corpora include:
- Alice’s Adventures in Wonderland for dialogue, fantasy, and playful language;
- Frankenstein for longer formal and Gothic prose;
- a matched collection of public-domain fantasy books;
- TinyStories, a synthetic corpus designed to investigate coherent language in small models;
- a bounded, attribution-compliant Wikipedia extract for more factual prose.
Modern copyrighted novels should not be bundled or automatically downloaded into a public tutorial. A generic local-file option can let a reader experiment with text they are permitted to use without committing or redistributing that corpus.
Design a useful corpus comparison
To compare Shakespeare with another dataset rather than merely run two unrelated trainings:
- record source, licence, checksum, cleaning, and split;
- match the amount of training text or state the difference clearly;
- use a shared character or byte vocabulary so parameter count stays fixed;
- hold architecture, seed policy, optimiser, context, and token budget constant;
- generate from shared prompts and sampling settings;
- report within-corpus loss improvement rather than declaring the easiest corpus “best”;
- measure exact generated matches against training text to detect regurgitation;
- retain all samples selected by the predefined protocol.
An Alice-trained model should look different from a Shakespeare-trained model even when both remain mostly nonsense. That visible dependency on the corpus is exciting precisely because the implementation has no special-case knowledge of plays, fantasy, speakers, or prose.
What changes on the path to a real LLM
The conceptual objective remains next-token prediction, but engineering scale changes almost everything around it:
- character tokens give way to byte or subword vocabularies;
- thousands of documents require deduplication, filtering, licensing, and provenance;
- pre-norm blocks, modern activations, rotary positions, and other architectural variants become common;
- mixed precision, gradient accumulation, distributed training, and fault-tolerant checkpoints become necessary;
- evaluation expands beyond one validation loss;
- instruction tuning and preference optimisation are added after base-model pretraining;
- serving must manage batching, caches, latency, memory, and safety.
Do not let those systems obscure the foundation. The 799,360-parameter model and a frontier model still turn token IDs into representations, use attention to combine context, produce next-token logits, and learn by reducing prediction error. Building the tiny version gives you a concrete mental model for questioning the larger one.
Continue with your own fork
Fork the repository and begin by replacing the corpus preparation path, not the model. Preserve the source, licence, checksum, split, configuration, and unedited output for your new run.
The interesting question is not simply whether the generated text looks better. Ask what changed in its local spelling, formatting, validation loss, memorisation, and training time. By this point you have enough of the system in your head to design that experiment and explain its limitations.