Layer 4 — Model Architecture · Chapter 3
Transformers and Attention
The transformer became the dominant architecture for language models because it can connect distant pieces of information, train efficiently on parallel hardware, and scale to very large datasets and parameter counts. Its central mechanism is attention.
What attention does
Attention allows each token or input element to ask: “Which other elements are relevant to me right now?” The system creates a weighted mixture of information from those elements. The weights change with the input, so the same architecture can form different relationships in different contexts.
Queries, keys, and values
Each element is transformed into three roles:
- Query: what this element is looking for.
- Key: what this element offers for matching.
- Value: the information passed forward if the match is strong.
Matching queries against keys produces attention weights. Those weights combine the values. Multiple attention heads perform this process in parallel, allowing the model to track several kinds of relationship at once.
Position must be added
Attention by itself does not inherently know whether a word came first or last. Transformers therefore add positional information. Different designs encode position through learned vectors, rotations, relative distances, or other mechanisms.
The transformer block
A typical block alternates attention with a feed-forward transformation, surrounded by residual connections and normalization. Stacking many blocks creates progressively richer representations.
Why transformers scaled so well
- Training can process many sequence positions in parallel.
- The same design works across text, images, audio, code, biology, and mixed modalities.
- Model size, data, and computation can be increased without changing the basic blueprint.
- The architecture fits modern accelerators and distributed training systems.
Important limitations
Standard attention compares many pairs of tokens, which can make long contexts expensive. A transformer also has no persistent memory by default; information outside its current context is absent unless another mechanism retrieves or stores it. Finally, predicting the next token does not automatically create reliable planning, factuality, or causal understanding.
| Strength | Corresponding limitation |
|---|---|
| Flexible relationships across a sequence | Attention cost can rise quickly with context length. |
| General-purpose architecture | General design may be inefficient for highly structured tasks. |
| Strong pattern learning at scale | Learned patterns can imitate reasoning without guaranteeing it. |
| One model can handle many tasks | Control, verification, and specialization often require external systems. |
The transformer is a powerful information mixer, not a complete theory of intelligence.