Inference
Inference is the use of a trained model to produce an output for a new input — a fraud score for a transaction, a reply to a customer's message, a document's summary — as opposed to training, which produced the model. It is where AI meets production: served behind an API, measured in latency and cost per query, and at scale the larger part of the compute bill, since a model is trained once and queried millions of times.
- Also known as
- Model inference, Serving, Prediction (AI)
- Used at
- every deployed model — a fintech's fraud engine scoring 3,000 transactions a second, a support bot answering a message, a credit model deciding an application, a vendor's language model behind an API priced per token
- Binding
- no rule names it, but the outcomes are regulated, and the EU AI Act's obligations on high-risk systems apply at inference — logging, human oversight, accuracy monitoring
- Typical costs
- language-model APIs priced per million tokens (from cents for small models to tens of dollars for the largest); self-hosted inference priced by accelerator-hours; a fraud model's inference on CPUs costs fractions of a cent per transaction; "reasoning" models that think longer multiply the tokens per query
- Typical engineering
- batching queries to keep accelerators busy, quantisation (running the model at lower precision), caching, distillation to smaller models for the common cases, and latency targets — a fraud decision in 50 milliseconds, a chat reply that starts in a second
- Parties
- the model's operator (the lab or the company hosting it), the cloud and accelerator providers, the application that calls it, and the users whose queries are the load
How inference works in practice
A card transaction arrives at a fintech's fraud engine. The model — a gradient-boosted tree or a small neural network trained last week — takes the transaction's features and returns a score in 20 milliseconds, on a CPU, at a cost of about a hundredth of a cent. Three thousand a second, all day: cheap, fast, and the whole reason the model exists. A customer's message arrives at the support bot: the fintech's application assembles a prompt — the message, the retrieved fee schedule, instructions — and sends it to a vendor's language model over an API; 1,500 tokens in, 300 out, a reply in two seconds, at a few cents. Two million messages a month is $60,000 — the inference bill for that product.
The economics differ from training's and now dominate. A frontier model costs hundreds of millions to train once; serving it to a hundred million users costs the operator more every year than the training did, which is why the labs price per token and optimise serving hard: batching, quantisation, speculative decoding, caching, distilling large models into small ones that answer the easy questions. For a company using models over an API the same logic applies at its scale: a model's cost is the tokens it consumes multiplied by the queries the product generates, and a feature that seemed free at pilot becomes a line item when every customer uses it.
For a buyer of an AI product company the inference cost per user against the revenue per user is the unit economics; a product whose serving cost exceeds its price is a subsidy, however good the model.
Example
A fintech launches an AI assistant in its app. Pilot: 5,000 users, 40,000 queries a month, 800 tokens average per query in and out, at a blended $6 per million tokens — $190 a month. Launch: 800,000 users, 9 million queries a month — $43,000 a month, and rising with a "reasoning" feature that triples tokens per query. The team optimises: a small model classifies queries and answers the 60% that are simple (balance, last transaction) at a tenth of the cost; the large model handles the rest; a cache serves repeated questions; prompts are trimmed by a third. Cost settles at $21,000 a month for 9 million queries — $0.0023 each — against a subscription tier the assistant helped sell. The CFO's dashboard gains a line called "tokens" between "cloud" and "payments processing".
Common mistakes
- Piloting on a large model and launching without a cost model for production volumes. - Ignoring latency until customers do; a fraud score that takes a second is a decline. - Running every query through the largest model when a small one answers most. - Self-hosting a model for control at a cost far above the API, or using an API for data that should not leave the company. - Diligence on an AI product that reads the model quality and not the serving cost per user.