---
title: "Transformer"
url: "https://mahouse.world/glossary/transformer"
type: "glossary term"
topic: "Language models"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-06"
---

# Transformer

The transformer is the neural-network architecture, introduced by Google researchers in 2017, that underlies every modern language model and most vision, speech and multimodal models: it processes a sequence of tokens in parallel through stacked layers of self-attention — in which every token weighs its relation to every other — and feedforward networks. It replaced the recurrent networks that processed sequences step by step, trained far faster on GPUs, and scaled to hundreds of billions of parameters.

**Also known as:** Transformer architecture, Transformer model, Attention-based model

## Key facts

- **Used at:** language models (GPT, Claude, Gemini, Llama), text encoders (BERT and successors), vision transformers, speech models (Whisper), protein models (AlphaFold's core), multimodal models, code models, and, in fintech, sequence models over transactions and events where the largest firms build their own
- **Binding:** none; an architecture
- **Typical structure:** tokens are embedded into vectors; positional encodings are added; each of N layers (12 in BERT-base, 96 in GPT-3, more since) applies multi-head self-attention followed by a feedforward network, with residual connections and normalisation; a decoder model attends only to earlier tokens and predicts the next; an encoder attends to all; the original had both for translation
- **Typical scaling:** parameters from 100 million (2018) to over a trillion (2024, with mixture-of-experts); training compute from GPU-days to hundreds of millions of dollars; context windows from 512 tokens to over a million; the engineering of long context — efficient attention, KV caching, sparse and sliding-window variants — is the response to the quadratic cost
- **Parties:** the 2017 authors (most of whom founded AI companies), the labs that scaled it, the hardware makers whose chips are designed around its matrix multiplications, and everyone building on the models

## How a transformer works in practice

Feed a transformer the sentence "the acquirer withdrew its offer after the target's numbers came in". Each word becomes a vector; positions are added. In the first attention layer, every word computes how much to attend to every other — "its" attends strongly to "acquirer" — and updates its representation with a weighted mix of the others'; a feedforward network transforms each; the next layer does it again on richer representations; after dozens of layers, the representation of the last token encodes enough of the sentence's meaning to predict what comes next, and, in a model trained on trillions of tokens, to answer a question about it, translate it, or continue it in the style of a legal memo. Training adjusts the billions of weights so that the predictions match the text; nothing about syntax, entities or meaning was programmed.

The architecture's consequences are the shape of the AI economy. Parallel training on GPUs made scale affordable, and scale produced capabilities — in-context learning, reasoning, code — that smaller models lacked; the labs' race has been a race to train larger transformers on more data with more compute. The quadratic attention cost is why long documents are expensive and why retrieval remains the economical design. The same architecture serves text, images, audio and proteins, which is why one lab's expertise transfers across them and why the multimodal models arrived. For a fintech, the transformer is the mechanism inside every language and vision model it rents, and the architecture it would use for a sequence model on its own event data; for an investor, it is the reason the AI value chain concentrates in compute and in the few organisations that can train at the frontier.

For a practitioner, the transformer is the reason to keep context to what the task needs and to retrieve rather than stuff; for an acquirer of an AI company, "we built our own transformer" means a pre-trained open model fine-tuned, unless the diligence finds a training run that cost nine figures.

## Example

A payments company with 3 billion transactions a year builds a transformer over customer event sequences to score account-takeover risk: each event (login, device change, payee added, transfer) is a token with attributes; the model attends over the last 200 events and outputs a risk score. It is small by language-model standards — 20 million parameters, trained in a day on one GPU on 18 months of labelled sequences — and it outperforms the feature-engineered gradient-boosted model on this task (AUC 0.95 against 0.91) because the sequence and its timing carry the signal that hand-built features missed. The attention weights give investigators the three events that drove each alert. Inference at 4 milliseconds fits the authorisation budget; the model runs on every login and payee change, and the tabular model keeps the bulk of transaction scoring. The model card records the architecture, the data, the validation and the explanation method.

## Common mistakes

- Assuming long context is free; attention cost is quadratic.
- Reading "transformer" in a pitch as proprietary technology; the architecture is public and the pre-trained weights are usually someone else's.
- Sequence models for tabular problems that have no sequence.
- Overlooking that the same architecture serves text, vision and audio, so one vendor's models converge.
- Building a language model when fine-tuning an open one would do.

## Related terms

- [Attention Mechanism](https://mahouse.world/glossary/attention-mechanism.md)
- [Large Language Model](https://mahouse.world/glossary/large-language-model.md)
- [Pre-Training](https://mahouse.world/glossary/pre-training.md)
- [Tokenization](https://mahouse.world/glossary/tokenization.md)
- [Model Parameters](https://mahouse.world/glossary/model-parameters.md)
- [Mixture of Experts](https://mahouse.world/glossary/mixture-of-experts.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov