---
title: "Mixture of Experts"
url: "https://mahouse.world/glossary/mixture-of-experts"
type: "glossary term"
topic: "Model architecture"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Mixture of Experts

A mixture of experts is a neural-network architecture in which certain layers are replaced by several parallel sub-networks — the experts — and a small router chooses, for each input token, which few experts process it; the total parameter count grows with the number of experts while the computation per token stays that of the few that ran. Mixtral, DeepSeek-V3 and Gemini use it; a model can have hundreds of billions of parameters and run like a small one.

**Also known as:** MoE, Sparse mixture of experts, Sparse model, Expert routing

## Key facts

- **Used at:** the largest language models, where it decouples capacity (parameters) from inference cost (active parameters); the open-weight Mixtral 8×7B (2023) and DeepSeek-V3 (2024, 671 billion total, 37 billion active per token) made the pattern public
- **Binding:** none; the EU AI Act's compute threshold for general-purpose models with systemic risk counts training compute, which for a MoE model is that of the active parameters — a detail regulators have noted
- **Typical mechanics:** in a transformer, the feedforward block of some or all layers becomes N experts; a router computes a score per expert for each token and sends it to the top-k (usually one or two); the outputs are combined by the router's weights; training adds a load-balancing loss so experts are used evenly and none collapses
- **Typical trade-offs:** better quality per unit of compute than a dense model of the same active size; more memory, since all experts must be loaded even though few run; harder to train stably; harder to serve on small hardware because the total parameters must fit somewhere
- **Parties:** the labs (Google's early work, Mistral, DeepSeek, OpenAI), the inference providers who must host all experts, and buyers reading "parameters" in a model's name without knowing how many are active

## How a mixture of experts works in practice

A dense 70-billion-parameter model runs all 70 billion weights on every token. A mixture-of-experts model with 8 experts of 7 billion each in its feedforward layers has around 47 billion parameters total (experts plus the shared attention layers) but runs about 13 billion per token — the shared layers plus two experts. The router, a small learned layer, sends a token about legal language to experts that have come to specialise in it and a token of code to others; nobody assigned the specialisms, they emerged from the load-balanced training. The result is a model that scores like a larger dense model and costs like a smaller one per token — with a hardware condition: all 47 billion parameters must sit in GPU memory, because any token might need any expert.

The consequences for buyers of model access are in the price and the fine print. A MoE model's API price reflects its active parameters, which is why the frontier models got cheaper per token even as their capabilities grew. Self-hosting one requires memory for the whole model — a 671-billion-parameter MoE needs the hardware of a 671-billion-parameter model to load, even though it computes like a 37-billion one — which keeps it in the data centre. And "parameters" in a model's name became ambiguous: the honest figure is the pair, total and active.

For AI M&A the architecture is one of the reasons inference economics improved through 2024–2025, and a diligence detail when a company's "proprietary model" turns out to be a fine-tuned open MoE.

## Example

A fintech runs a document-extraction workload of 2 million pages a month through a hosted dense 70-billion-parameter model at $0.90 per million input tokens: about $5,400 a month. A MoE model with comparable benchmark quality is priced at $0.27 per million input tokens; on the fintech's own evaluation set of 1,000 pages, it scores within 0.5 points of the dense model on field accuracy. Switching saves $3,800 a month. The engineering lead also evaluates self-hosting the open-weight MoE: the model needs four 80 GB GPUs just to load, against a single GPU for a dense 13-billion model of lower quality; at the fintech's volume the hosted API is cheaper than the hardware, and the decision is to stay hosted and revisit at ten times the volume.

## Common mistakes

- Comparing models by total parameters when one is a MoE and its active count is a fraction.
- Planning to self-host a MoE model on the hardware its active size suggests.
- Assuming an expert corresponds to a domain; the specialisms are emergent and mixed.
- Reading the cheaper price as lower quality; evaluate on the company's own set.
- Missing that training-compute thresholds may count active parameters.

## Related terms

- [Model Parameters](https://mahouse.world/glossary/model-parameters.md)
- [Large Language Model](https://mahouse.world/glossary/large-language-model.md)
- [Transformer](https://mahouse.world/glossary/transformer.md)
- [Compute](https://mahouse.world/glossary/compute.md)
- [Inference](https://mahouse.world/glossary/inference.md)
- [Foundation Model](https://mahouse.world/glossary/foundation-model.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov