Model Parameters
Model parameters are the learned numbers inside a neural network — the weights that multiply inputs and the biases added to them — set by training and fixed at inference; their count is the customary, if crude, measure of a model's size, from millions in a fraud model to hundreds of billions in a frontier language model. More parameters mean more capacity, more data and compute to train, more memory to run, and, past what the data supports, overfitting.
- Also known as
- Parameters, Weights, Parameter count, Model size
- Used at
- every model description and comparison ("a 7B model"), hardware sizing (a parameter in 16-bit precision is 2 bytes of memory, so a 70-billion-parameter model needs 140 GB to load at that precision), and, in policy, the proxies for capability — the EU AI Act uses training compute (10²⁵ FLOPs), which scales with parameters times tokens
- Binding
- none on the count itself; the compute threshold above is the regulatory use
- Typical scale
- a credit or fraud network, 10⁵–10⁷ parameters; a small language model for on-device use, 1–4 billion; open-weight mid-size, 7–70 billion; frontier models, hundreds of billions to over a trillion (with mixture-of-experts making the active count much smaller)
- Typical relations
- scaling laws (2020–2022) found loss improves predictably with parameters, data and compute together, and that a model of a given size is best trained on roughly 20 tokens per parameter (Chinchilla); the trend since has been to train smaller models on far more data, so a 2025 8-billion-parameter model beats a 2022 175-billion one
- Parties
- the labs that choose the size, the hardware makers whose memory decides what can be served, and the buyers for whom the count is a proxy for cost and capability
How model parameters work in practice
A fraud model with 300 inputs, two hidden layers of 256 and 128 units and one output has 300×256 + 256 + 256×128 + 128 + 128 + 1 ≈ 110,000 parameters; each is a floating-point number learned by backpropagation, and the model is the set of them plus the architecture that says how they connect. A language model is the same at a scale of billions: every parameter takes part in every forward pass, so the count sets the computation per token (about two operations per parameter) and the memory to hold them. Training sets them; fine-tuning adjusts them (or, with LoRA, adds a small number of new ones); quantisation stores them in fewer bits; distillation trains a smaller set to imitate a larger one.
The count's usefulness as a measure declined as the relationships changed. A 2020 model's quality rose with size because everyone trained on similar data for similar durations; by 2024, training data quality and quantity, post-training, and architecture (mixture of experts, longer context) mattered as much, and small models trained well overtook large ones trained early. For a buyer the count answers two questions reliably — what hardware will serve this, and roughly what a token will cost — and one unreliably: how good it is, for which the answer is the evaluation on the company's own task.
For a fintech, the practical rule is that the parameter count is a capacity-planning number, not a quality number, and that the smallest model that passes the evaluation set is the one to run.
Example
A fintech's data team proposes a frontier model (hundreds of billions of parameters, hosted, $15 per million output tokens) for a merchant-support assistant. The platform lead asks for an evaluation instead of a size: on 800 real conversations, the frontier model scores 91% on the resolution rubric; a hosted 70-billion open-weight model scores 88% at $0.80 per million; a fine-tuned 8-billion model, trained on 3,000 of the company's own conversations, scores 90% at $0.15 per million and can run on one GPU the company already has. Annual cost at forecast volume: $210,000, $11,000 and $4,000 in inference respectively, plus $12,000 of fine-tuning for the small one. The 8-billion model ships; the frontier model handles the 4% of conversations it escalates.
Common mistakes
- Choosing a model by parameter count rather than by an evaluation on the actual task. - Sizing hardware for a model without checking precision and the active parameters of a MoE. - Assuming a larger model is safer or more accurate for a narrow task; often the fine-tuned small one wins. - Confusing parameters with training data; a model's knowledge is bounded by what it was trained on, not its size. - Reporting parameter counts in diligence without training data, compute and evaluation results.