Machine learning
Language models from scratch
Building GPT-style transformers from their individual components, then adapting them for instructions and spam classification.
Overview
Understanding what happens inside a language model
Language models turn a sequence of words into a prediction about what comes next. This project explores that process by building the main components directly, rather than treating a transformer as a black box.
I worked from attention and embeddings up to a GPT-style model, then explored two practical adaptations: following instructions and classifying SMS messages as spam or ordinary messages.
The project connects the mathematics of language modelling with the software needed to prepare text, train a model, save its state, and generate output. It is an educational implementation of established architectures.
Technical detail
From text to next-token predictions
Text is split into tokens using the GPT-2 tokenizer. Token embeddings represent their content, while positional embeddings represent their order. A stack of transformer blocks processes those representations before a final projection produces vocabulary logits.
- Text and tokenisation
- Token + positional embeddings
- Causal transformer blocks
- Vocabulary logits
- Next-token generation
The PyTorch implementation separates multi-head attention, feed-forward layers, normalisation, and residual connections into individual modules. Causal masking prevents a position from looking at later tokens during next-token prediction.
Adapting the same foundation
Instruction fine-tuning formats instruction, input, and response examples into training sequences. The spam-classification workflow replaces the vocabulary output with a two-class head and selectively trains later layers, reusing the pretrained language representations.
The current training configuration uses 256-token windows, 12 transformer layers, 12 attention heads, and an embedding width of 768. The shorter context reduces attention memory requirements at the cost of how much preceding text the model can use.
What the training curves show
| Epoch | Training loss | Validation loss |
|---|---|---|
| 1 | 3.985 | 4.857 |
| 2 | 3.574 | 4.421 |
| 3 | 3.339 | 4.476 |
| 4 | 3.237 | 4.403 |
| 5 | 3.187 | 4.294 |
Across this five-epoch training sequence, validation loss falls from 4.857 to 4.294, while training loss falls from 3.985 to 3.187. The gap between them makes generalisation an important part of interpreting the result.
These losses describe one training sequence, not a comparison with commercial models. The current workflow uses a sequential text split; document-level holdouts and fixed generation evaluations would provide a stronger picture of performance on unfamiliar material.