---
title: "Multimodal Model"
url: "https://mahouse.world/glossary/multimodal-model"
type: "glossary term"
topic: "Multimodal AI"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Multimodal Model

A multimodal model is an AI model that works across more than one type of data — reading images and text together, hearing audio, watching video, and producing text, images or speech in return — rather than one modality alone. GPT-4V, Gemini and the open-weight vision-language models joined a vision encoder to a language model; for a fintech that means a model that reads a bank statement, a receipt, an ID document or a customer's screenshot, and answers in words.

**Also known as:** Multimodal AI, Vision-language model, VLM, Multimodal LLM

## Key facts

- **Used at:** document understanding (statements, invoices, IDs, contracts), customer support with screenshots and photos, accessibility, video and audio analysis, voice assistants, and the perception layer of agents that act on a screen
- **Binding:** the EU AI Act's transparency rules for AI-generated content and its high-risk obligations where the model decides about people; biometric uses (face recognition) have their own restrictions; data-protection law on the images and audio processed
- **Typical architecture:** a vision encoder (often CLIP-derived) turns an image into embeddings that are projected into the language model's token space, so the language model attends to image content as if it were text; audio and video encoders join the same way; "natively multimodal" models are trained on all modalities from the start rather than bolted together
- **Typical strengths and limits:** strong at describing, classifying and extracting from images and documents, reading charts and screenshots, reasoning across text and image; weaker at precise measurement, small text, counting, and spatial detail; hallucination applies to images as to text — the model may report a figure that is not there
- **Parties:** the labs building them, the companies applying them to documents and support, the identity and fraud vendors using and defending against them, and the customers whose images are processed

## How a multimodal model works in practice

A customer photographs a rejected transfer screen and types "why did this fail?". The image goes to the model's vision encoder, which produces a sequence of embeddings; those are placed alongside the text's tokens in the language model's context; the model attends across both, reads the error code on the screen, recognises the fintech's interface, and answers: "The transfer was rejected because the IBAN's country is not supported on your plan — here is how to check". Nothing was OCR'd in the old sense; the model read the image the way it reads text, and reasoned over both. The same model, shown a bank statement, extracts the transactions into a table; shown an invoice, matches it to a purchase order; shown a chart, describes the trend.

The engineering questions are where it is accurate enough. Document extraction from clean images is strong; small print, handwritten amounts and dense tables need verification; a model asked for a total may invent one, so the output is checked against the source or constrained to fields the model must cite. Identity documents are a special case: the model can read and classify them, but the fraud question — is this document real, is this face live — is the identity vendor's forensic stack, not the general model's. Voice adds another modality and another attack surface (cloned voices), and the multimodal model that talks is the one that customers will trust most and that needs the tightest guardrails.

For fintech M&A, document-AI companies built on OCR and templates were the acquisition targets of 2018–2022; the multimodal models made much of their technology a feature of a vendor's API, which is a valuation question for the ones that remain.

## Example

A business-lending fintech underwrites on bank statements uploaded as PDFs and photos from 400 banks in 30 formats. Its OCR-and-template pipeline handles 60% automatically; the rest go to a team of 20 who key figures by hand at a cost of €18 per application. It replaces the pipeline with a multimodal model prompted to extract transactions into a schema with a citation to the page and line for every figure, plus a verification pass that recomputes balances from the extracted transactions and flags mismatches. Automation rises to 91%; the manual team handles the 9% flagged, mostly handwritten or damaged documents; cost per application falls to €4; and the verification pass catches a doctored statement the old pipeline had passed — the balances did not add up, which the model's extraction made checkable.

## Common mistakes

- Trusting extracted figures without a verification that recomputes them from the source.
- Using a general multimodal model as an identity-document forensic tool.
- Sending customers' images and documents to a vendor whose data terms allow training on them.
- Assuming the model reads small print and handwriting as well as it describes a photograph.
- Document-AI vendors valued on technology the labs' APIs now include.

## Related terms

- [CLIP](https://mahouse.world/glossary/clip.md)
- [Large Language Model](https://mahouse.world/glossary/large-language-model.md)
- [Foundation Model](https://mahouse.world/glossary/foundation-model.md)
- [Diffusion Model](https://mahouse.world/glossary/diffusion-model.md)
- [Hallucination](https://mahouse.world/glossary/hallucination.md)
- [Identity Verification](https://mahouse.world/glossary/identity-verification.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov