Transformer Architecture

Definition. The neural network design in which every token attends to every other token in the context through scaled dot-product self-attention, replacing recurrence with parallel matrix operations. A block consists of multi-head self-attention, a feed-forward network, residual connections and layer normalisation, stacked dozens of times. Decoder-only variants apply a causal mask so a token can only attend to earlier tokens, which is what makes autoregressive generation possible.

How it works. You rarely implement a transformer, but you reason about it constantly when tuning systems. Attention cost grows quadratically with sequence length, so doubling context quadruples the attention compute. That single fact drives chunking strategy in RAG, the choice of sliding-window attention in long-document models, and why prefill (processing the prompt) is compute-bound while decode (generating tokens) is memory-bandwidth-bound.

Example. An engineering team ingests 200-page contracts. A naive approach sends the whole document, roughly 90,000 tokens, in one call. Attention cost scales with the square of that, and the request either exceeds the context limit or costs several dollars per document. Chunking into 1,500-token sections with 200-token overlap, embedding each, and retrieving the top eight sections per question reduces per-query cost by more than an order of magnitude and improves answer precision because the model is not distracted by irrelevant clauses.

Use case. Understanding the architecture matters for capacity planning, cost modelling, context budgeting, and diagnosing failure modes such as lost-in-the-middle recall, where information placed in the centre of a long context is retrieved less reliably than information at the start or end. It also explains why prompt ordering and document placement change accuracy.

Warnings. Do not assume a large advertised context window is usable at full length. Effective recall degrades well before the hard limit, and providers bill for every token sent whether or not the model attends to it usefully.

Limitations. Transformers have no inherent notion of order beyond positional encodings, no recurrence, and no external memory. They cannot update their weights from a conversation, so anything that must persist has to be written to an external store and re-injected. Quadratic attention remains the binding constraint on very long inputs despite linear-attention and state-space alternatives.

← Browse all glossary terms