Compute
Compute, in AI, is the processing capacity consumed by training and running models — measured in floating-point operations (a frontier training run in 2024–2025 used on the order of 10^25 to 10^26 FLOPs), in accelerator-hours and in dollars — and it is the resource the industry is constrained by. Capability has scaled with it for a decade, and the EU AI Act uses a compute threshold to define the models it regulates most.
- Also known as
- AI compute, Training compute, FLOPs
- Used at
- pre-training (the bulk of a frontier model's compute), fine-tuning and alignment, inference at scale (which now rivals training in total spend), and in fintech the far smaller runs of fraud, credit and language models
- Binding
- the EU AI Act's systemic-risk threshold for general-purpose models is defined by training compute (10^25 FLOPs); US export controls restrict the chips that supply it; cloud contracts allocate it
- Typical scaling
- the compute used to train frontier models grew about 4–5× a year through the 2010s and 2020s; the "scaling laws" relate model performance to compute, data and parameters, and have guided lab strategy; inference compute per query has risen with "reasoning" models that think longer
- Typical cost
- frontier training runs from $100m (2023) to over $1bn (2025); inference for a popular model costs the operator more per year than training did; a fintech's compute for its own models is thousands to low millions a year; API usage is priced per million tokens
- Parties
- the labs and hyperscalers buying and building it, the chip makers and foundries supplying it, the data-centre and power industries housing it, the regulators counting it, and every company deciding whether to rent, buy or use an API
How compute works in practice
Training a model is arithmetic at scale: each training step runs the model forward on a batch of data and backward to compute gradients, updating billions of parameters, repeated for trillions of tokens. The total is the model's training compute — parameters × tokens × a constant, in FLOPs — and it maps to accelerator-hours at a given hardware efficiency and to dollars at a given rental rate. A model with ten times the parameters trained on ten times the data costs a hundred times the compute, which is why frontier runs reached the hundreds of millions and why the labs measure themselves in gigawatts of data-centre capacity. Inference is the same arithmetic without the backward pass, run once per query for millions of users, which at scale costs more than the training did.
The constraint shaped the industry's structure. Compute is bought from the hyperscalers or built with them; the labs' funding rounds are compute commitments; the chips come from one designer and one foundry; and the data centres need power the grid was not built for. Regulators, needing a proxy for "capable enough to worry about", chose compute — the AI Act's 10^25 FLOPs — because it is measurable and correlates with capability, and reporting obligations follow the threshold.
For a fintech the scale is different by orders of magnitude, and the decisions are: rent accelerators to train models on its own data (fraud, credit, personalisation), or pay per token for a vendor's model and put the compute in the operating budget. Both are compute; neither needs a data centre.
Example
A fintech's AI budget for a year: $600,000 of API usage for a vendor's language model across its support bot and document tools (about 2 billion tokens a month at blended rates); $180,000 of rented accelerator time for training and retraining its fraud and credit models; $90,000 for inference infrastructure serving those models; and a data science team whose salaries dwarf all three. A board member, having read that a lab raised $6bn "for compute", asks whether the fintech is underinvesting. The CTO's reply: the fintech's total compute is 0.0002% of a frontier run, its models are small because its problems are, and the $600,000 of tokens is the line that will grow — because every product team wants a model in its feature, and the meter runs per word.
Common mistakes
- Reading frontier compute numbers as relevant to a company's own models. - API costs treated as free at pilot scale and discovered at production scale. - Training on rented accelerators without tracking cost per experiment. - Assuming more compute fixes a model whose problem is data. - Diligence on an AI company that values the model without the compute bill that produced it and the one that serves it.