---
title: "Attention Mechanism"
url: "https://mahouse.world/glossary/attention-mechanism"
type: "glossary term"
topic: "Model architecture"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Attention Mechanism

The attention mechanism is the component of a neural network that, for each element of the output being produced, computes a weighting over the elements of the input — which words in a sentence, which transactions in a history matter for this step — and combines them accordingly. Self-attention is the core of the transformer (2017); it is why language models can relate a word to one fifty pages earlier, and why cost grows with the square of context length.

**Also known as:** Attention, Self-attention, Multi-head attention, Attention layer

## Key facts

- **Used at:** every transformer — language models, vision transformers, speech and protein models — and in recommendation and fraud models that weigh a customer's event history
- **Binding:** none; a technical building block
- **Typical mechanics:** each input element is projected into a query, a key and a value; the attention weight between two elements is the similarity of one's query with the other's key, normalised across all elements; the output is the weighted sum of values; multi-head attention runs several such projections in parallel so different heads can attend to different relations (syntax, coreference, position)
- **Typical cost:** attention over n elements computes n² similarities, so doubling the context quadruples that part of the computation and memory; the engineering of long-context models — FlashAttention, sparse and sliding-window attention, KV caching — is about paying less for it
- **Parties:** the researchers who introduced and refined it, the hardware makers whose chips are optimised for its matrix products, and everyone who pays for context tokens

## How the attention mechanism works in practice

Read the sentence "the acquirer withdrew its offer after the target's numbers came in, and it paid the break fee". To know what "it" refers to, a reader looks back to "the acquirer"; attention does the same, computing for the token "it" a high weight on "acquirer" and low weights on the rest, and the representation of "it" that flows to the next layer carries the acquirer's meaning. Across dozens of layers and heads, a transformer builds up relations of this kind — subject to verb, entity to attribute, question to the passage that answers it — with no rule written by anyone; the weights were learned from data.

The practical consequences are in what models can do and what they cost. They can use a long context — a whole contract, a year of transactions, a long conversation — because attention reaches any position directly, where earlier recurrent networks had to pass information step by step and lost it. They cost more the longer the context, because every token attends to every other; a 100,000-token document is not ten times a 10,000-token one but far more, which is why long-context pricing is what it is and why retrieval (give the model the relevant pages, not the whole archive) remains the economical design. And they can be inspected: attention weights show which inputs a model used for an output, which is a debugging aid, though not a full explanation.

For a fintech, attention is the mechanism behind the models it rents, and the reason two engineering rules hold: keep the context to what the task needs, and retrieve rather than stuff.

## Example

A fintech builds a transaction-monitoring model that reads a customer's last 500 events — payments, logins, device changes, support contacts — and scores the current transaction. A gradient-boosted model over hand-built features (counts, averages, recency) reaches an AUC of 0.91. A small transformer with attention over the event sequence reaches 0.94: it learns, without feature engineering, that a password reset followed by a new payee followed by a large transfer is the pattern, and the attention weights on those three events are what the investigator sees when a case is opened. The cost: inference is four times slower, which the team handles by running the transformer only on the 8% of transactions the fast model marks as uncertain.

## Common mistakes

- Sending an entire archive as context when retrieval would give the model the relevant pages at a fraction of the cost.
- Reading attention weights as a complete explanation of a model's output.
- Assuming long-context ability means equal accuracy at every position; models still miss things in the middle.
- Sequence models everywhere; for tabular data with good features, gradient boosting often wins.
- Ignoring the quadratic cost when planning throughput for long documents.

## Related terms

- [Transformer](https://mahouse.world/glossary/transformer.md)
- [Large Language Model](https://mahouse.world/glossary/large-language-model.md)
- [Hidden Layer](https://mahouse.world/glossary/hidden-layer.md)
- [Model Parameters](https://mahouse.world/glossary/model-parameters.md)
- [Compute](https://mahouse.world/glossary/compute.md)
- [Inference](https://mahouse.world/glossary/inference.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov