---
title: "Inference"
url: "https://mahouse.world/glossary/inference"
type: "glossary term"
topic: "AI infrastructure"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Inference

Inference is the use of a trained model to produce an output for a new input — a fraud score for a transaction, a reply to a customer's message, a document's summary — as opposed to training, which produced the model. It is where AI meets production: served behind an API, measured in latency and cost per query, and at scale the larger part of the compute bill, since a model is trained once and queried millions of times.

**Also known as:** Model inference, Serving, Prediction (AI)

## Key facts

- **Used at:** every deployed model — a fintech's fraud engine scoring 3,000 transactions a second, a support bot answering a message, a credit model deciding an application, a vendor's language model behind an API priced per token
- **Binding:** no rule names it, but the outcomes are regulated, and the EU AI Act's obligations on high-risk systems apply at inference — logging, human oversight, accuracy monitoring
- **Typical costs:** language-model APIs priced per million tokens (from cents for small models to tens of dollars for the largest); self-hosted inference priced by accelerator-hours; a fraud model's inference on CPUs costs fractions of a cent per transaction; "reasoning" models that think longer multiply the tokens per query
- **Typical engineering:** batching queries to keep accelerators busy, quantisation (running the model at lower precision), caching, distillation to smaller models for the common cases, and latency targets — a fraud decision in 50 milliseconds, a chat reply that starts in a second
- **Parties:** the model's operator (the lab or the company hosting it), the cloud and accelerator providers, the application that calls it, and the users whose queries are the load

## How inference works in practice

A card transaction arrives at a fintech's fraud engine. The model — a gradient-boosted tree or a small neural network trained last week — takes the transaction's features and returns a score in 20 milliseconds, on a CPU, at a cost of about a hundredth of a cent. Three thousand a second, all day: cheap, fast, and the whole reason the model exists. A customer's message arrives at the support bot: the fintech's application assembles a prompt — the message, the retrieved fee schedule, instructions — and sends it to a vendor's language model over an API; 1,500 tokens in, 300 out, a reply in two seconds, at a few cents. Two million messages a month is $60,000 — the inference bill for that product.

The economics differ from training's and now dominate. A frontier model costs hundreds of millions to train once; serving it to a hundred million users costs the operator more every year than the training did, which is why the labs price per token and optimise serving hard: batching, quantisation, speculative decoding, caching, distilling large models into small ones that answer the easy questions. For a company using models over an API the same logic applies at its scale: a model's cost is the tokens it consumes multiplied by the queries the product generates, and a feature that seemed free at pilot becomes a line item when every customer uses it.

For a buyer of an AI product company the inference cost per user against the revenue per user is the unit economics; a product whose serving cost exceeds its price is a subsidy, however good the model.

## Example

A fintech launches an AI assistant in its app. Pilot: 5,000 users, 40,000 queries a month, 800 tokens average per query in and out, at a blended $6 per million tokens — $190 a month. Launch: 800,000 users, 9 million queries a month — $43,000 a month, and rising with a "reasoning" feature that triples tokens per query. The team optimises: a small model classifies queries and answers the 60% that are simple (balance, last transaction) at a tenth of the cost; the large model handles the rest; a cache serves repeated questions; prompts are trimmed by a third. Cost settles at $21,000 a month for 9 million queries — $0.0023 each — against a subscription tier the assistant helped sell. The CFO's dashboard gains a line called "tokens" between "cloud" and "payments processing".

## Common mistakes

- Piloting on a large model and launching without a cost model for production volumes.
- Ignoring latency until customers do; a fraud score that takes a second is a decline.
- Running every query through the largest model when a small one answers most.
- Self-hosting a model for control at a cost far above the API, or using an API for data that should not leave the company.
- Diligence on an AI product that reads the model quality and not the serving cost per user.

## Related terms

- [Compute](https://mahouse.world/glossary/compute.md)
- [AI Accelerator](https://mahouse.world/glossary/ai-accelerator.md)
- [Pre-Training](https://mahouse.world/glossary/pre-training.md)
- [Guardrails](https://mahouse.world/glossary/guardrails.md)
- [Model Parameters](https://mahouse.world/glossary/model-parameters.md)
- [Hallucination](https://mahouse.world/glossary/hallucination.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov