---
title: "CLIP"
url: "https://mahouse.world/glossary/clip"
type: "glossary term"
topic: "Multimodal AI"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# CLIP

CLIP — Contrastive Language-Image Pre-training — is a model released by OpenAI in 2021 that learned to place images and their text descriptions into a shared embedding space, trained on 400 million image-caption pairs by pulling matching pairs together and pushing mismatched ones apart. It made zero-shot image classification possible, became the text encoder guiding diffusion image generators, and established the recipe every multimodal model since has used to connect vision and language.

**Also known as:** Contrastive Language-Image Pre-training, CLIP model, Image-text embedding

## Key facts

- **Used at:** zero-shot image classification and retrieval, content moderation, the text-conditioning of Stable Diffusion and DALL·E 2, image search, and — in fintech — document classification (is this a passport, a utility bill, a bank statement?), receipt and invoice understanding, and the vision side of multimodal assistants
- **Binding:** no regulation of the method; the training data's provenance (web-scraped images and captions) is the subject of copyright litigation, and a diligence question for any model built on similar data
- **Typical mechanics:** an image encoder (a vision transformer or a ResNet) and a text encoder (a transformer) trained jointly so that the cosine similarity between an image's embedding and its caption's is high and between mismatched pairs low — a contrastive objective over batches of thousands of pairs; at inference, an image's embedding is compared with the embeddings of candidate texts
- **Typical results:** zero-shot ImageNet accuracy matching supervised models without seeing ImageNet's training labels; robustness to distribution shift that supervised models lacked; open-weight versions (OpenCLIP) trained on public datasets (LAION) reproduced it
- **Parties:** OpenAI (the original), the open-source reproductions, the generative-model builders who adopted its encoders, and the companies applying it to their own images and documents

## How CLIP works in practice

Training: take a batch of 32,000 images and their captions from the web. Encode every image and every caption. The objective rewards the model when each image's embedding is closest to its own caption's and far from the other 31,999 — a contrastive loss. Across hundreds of millions of pairs the model learns a space where "a photo of a dog" and photos of dogs sit together, without anyone labelling dogs. Inference: to classify an image among categories nobody trained for, write the categories as sentences ("a photo of a passport", "a photo of a utility bill"), embed them, embed the image, and pick the closest — zero-shot classification, from words.

The recipe spread. Diffusion generators use a CLIP-style text encoder to turn a prompt into the guidance that shapes the image; multimodal language models connect a vision encoder trained this way to a language model; retrieval systems embed images and text into the same space and search across both. For a fintech the practical uses are unglamorous and valuable: sorting uploaded documents by type before OCR, matching a receipt image to a transaction description, flagging images that do not belong (a selfie uploaded as a bank statement), and the vision component of an assistant that reads what a customer photographs.

For AI M&A, CLIP is a marker of provenance: models trained on web-scraped image-text data carry the licensing questions that the copyright suits are deciding, and a buyer of a vision-AI company asks what its encoders were trained on.

## Example

A lender's onboarding flow asks applicants to upload proof of address. Thirty per cent of uploads are the wrong document type — a passport, a selfie, a screenshot of an app — and each is rejected by a human two days later. The team adds a CLIP-based classifier: the uploaded image is embedded and compared with text descriptions of the twelve accepted document types and eight common mistakes; wrong uploads are rejected instantly with a message naming what was received and what is needed. No training data was labelled; the categories were written as sentences in an afternoon. Wrong-document rejections at review fall from 30% to 4%, median onboarding time from three days to one, and the classifier's confidence threshold routes the ambiguous 6% to the human queue it used to flood.

## Common mistakes

- Deploying a zero-shot classifier without measuring its accuracy on the company's own documents.
- Treating web-trained embeddings as neutral; their biases are the web's.
- Ignoring training-data provenance in diligence on a vision-AI vendor.
- Using CLIP for fine-grained tasks (reading a number on a document) it was not built for; OCR and document models do that.
- Assuming the open-weight reproductions match the original's behaviour exactly.

## Related terms

- [Multimodal Model](https://mahouse.world/glossary/multimodal-model.md)
- [Latent Space](https://mahouse.world/glossary/latent-space.md)
- [Foundation Model](https://mahouse.world/glossary/foundation-model.md)
- [Diffusion Model](https://mahouse.world/glossary/diffusion-model.md)
- [Zero-Shot Learning](https://mahouse.world/glossary/zero-shot-learning.md)
- [Training Data](https://mahouse.world/glossary/training-data.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov