Pre-Training
Pre-training is the first and by far the most expensive phase of building a modern AI model: training it on a very large, broadly unlabelled corpus — trillions of tokens of text, billions of images — with a self-supervised objective such as predicting the next token, so that it learns the general structure of language, images or code before any specific task. The result is a base or foundation model; fine-tuning and preference training adapt it. Few can afford it.
- Also known as
- Pretraining, Base-model training, Self-supervised pre-training
- Used at
- every foundation model — language, vision, speech, code, protein — and every model derived from one; a fintech never pre-trains but always depends on someone's pre-training
- Binding
- the EU AI Act's obligations on general-purpose AI providers attach to the pre-trained model (training-data summary, copyright policy, evaluations, and, above 10²⁵ FLOPs of training compute, systemic-risk duties); copyright litigation over pre-training data is ongoing in several jurisdictions
- Typical scale
- GPT-3 (2020) trained on about 300 billion tokens; Llama 3 (2024) on 15 trillion; frontier runs by 2025 in the tens of trillions, on tens of thousands of GPUs for months; compute costs from tens of millions to several hundred million dollars per run
- Typical objective
- next-token prediction for language (the model sees text and predicts what comes next; the loss is cross-entropy); masked prediction for some encoders; contrastive objectives for image–text models. No labels are written; the data is its own supervision
- Typical outputs
- a base model that continues text but does not follow instructions or converse; the capabilities that emerge from scale (in-context learning, reasoning, code) appear here; post-training shapes them into a product
- Parties
- the labs with the capital and compute, the data sources whose content is the corpus, the regulators, and the downstream builders who inherit the model's knowledge, biases and cut-off date
How pre-training works in practice
Assemble a corpus: web crawls filtered for quality, books, code, papers, licensed datasets, transcripts — deduplicated, cleaned, and mixed in proportions that are themselves a research question. Tokenise it. Train a transformer to predict each next token given the preceding ones, over the whole corpus, once or a few times, on a cluster of thousands of accelerators running for months, with engineers watching the loss curve and intervening on spikes and failures. The model that emerges has read more than any person could and can complete any text in its style; it also knows nothing after its data cut-off, has absorbed every bias and error in the corpus, and will produce anything it was trained on, including things it should not. Post-training addresses the last; nothing addresses the first two except the next pre-training run.
For a fintech, pre-training is background with consequences. The cut-off date is why the model does not know last quarter's regulation. The corpus is why it writes fluent English about payments and stumbles on a niche scheme rule that appeared in few documents. The data provenance is a supply-chain question: a model trained on content its provider had no right to use carries legal risk that may reach the deployer. And the cost is why the market has few providers and why "our own model" in a startup's pitch nearly always means a fine-tuned open one — a diligence point in every AI acquisition.
For AI M&A, the pre-trained model is the asset that the licence-and-hire deals of 2024 were structured around: the talent that could pre-train and the weights they had produced.
Example
A fintech's AI lead evaluates three ways to get a model for contract analysis: pre-train a domain model on the company's 2 million documents ("so it knows our language"), fine-tune an open base model, or prompt a hosted frontier model with retrieval. The pre-training estimate: even a small 1-billion-parameter model on 2 billion tokens needs a cluster for weeks, a team with experience the company lacks, and produces a model with no general ability — about $600,000 and a worse result than the alternatives. Fine-tuning an open 8-billion model on 20,000 contract–summary pairs: $2,000 and a week, 90% on the evaluation set. Retrieval with a hosted model: no training, 92%, at a higher per-document cost. The company chooses retrieval, notes the vendor's training-data terms and cut-off date in the vendor file, and closes the pre-training question.
Common mistakes
- Proposing to pre-train a domain model when fine-tuning or retrieval would beat it at a fraction of the cost. - Ignoring the cut-off date; the model does not know what happened after it. - Treating a fine-tuned open model as "proprietary" in a pitch or a diligence. - Overlooking the provider's training-data provenance and its legal exposure. - Expecting a base model to follow instructions; that is post-training's job.