End-to-End Learning
End-to-end learning is training a single model to map raw input directly to the desired output — pixels to a document's fields, audio to a transcript, a transaction record to a fraud score — rather than building a pipeline of separately engineered stages, each with its own logic and errors. It trades the pipeline's interpretability and modularity for a model that optimises the whole task at once and often does better, provided there is enough labelled data at the ends.
- Also known as
- End-to-end training, End-to-end model, Pipeline-free learning
- Used at
- speech recognition (audio to text without separate acoustic and language models), document understanding (image to structured fields without OCR-then-parse), machine translation, autonomous driving debates (end-to-end versus modular stacks), and increasingly the language-model approach to any task once handled by a pipeline
- Binding
- none directly; the EU AI Act's and model-risk expectations of traceability bear on end-to-end models, which lack the intermediate outputs a pipeline exposes
- Typical trade-offs
- end-to-end models avoid error propagation between stages and exploit signals the pipeline's designers did not think of, but need far more labelled examples of the final output, are harder to debug (no intermediate to inspect), and cannot be improved one stage at a time; pipelines are interpretable, testable per stage, and can use small datasets per stage, at the cost of compounding errors and hand-engineering
- Typical outcome
- end-to-end wins where data is abundant and the task is perceptual (speech, vision); pipelines persist where data is scarce, intermediates are required (a regulated decision needs the extracted figure, not only the verdict), or components change independently
- Parties
- the research groups that demonstrated end-to-end wins in speech and vision, the engineers choosing an architecture, and the risk teams asking what the model saw on the way to its answer
How end-to-end learning works in practice
A payslip-verification pipeline runs five stages: detect the document, correct its orientation, OCR the text, parse the fields with rules and a small model, and compute a verdict. Each stage is testable; each has its own error rate; and a 4% error at each compounds to 18% at the end. The end-to-end alternative trains one model on 50,000 payslip images labelled with the final structured output — employer, gross pay, net pay, period — and lets the model learn whatever internal representation gets there. Its error is 6% on the same test set, because it never commits to a wrong OCR reading it cannot revisit. Its cost is the 50,000 labels, the opacity of the route from pixels to fields, and the fact that adding a new field means retraining rather than adding a rule.
The compromise that fintech teams usually reach is a hybrid: an end-to-end model for the perceptual part (image to structured fields, where the output is checkable against the image) feeding a rules-based decision stage that the regulator can read. The multimodal language models pushed the boundary further — one prompt from image to answer — and the same governance answer applies: use the model for extraction, cite the source for every figure, and decide in code. Where the pipeline's intermediates are themselves required outputs (the extracted income must be recorded), end-to-end from image to verdict is not an option, whatever its accuracy.
For a fintech, the question is not which is more elegant but where the errors, the labels and the audit trail need to be.
Example
A remittance company's transaction-monitoring pipeline has four stages: enrich the transaction, compute 300 features, score with a gradient-boosted model, apply rules. Detection rate 72% at a 5% alert rate. The data team trains an end-to-end sequence model on 18 months of raw event streams labelled with confirmed outcomes: detection rises to 81% at the same alert rate. The compliance officer's objection is that the new model gives no features to explain an alert. The resolution: the end-to-end model runs as a scorer whose output becomes one input to the existing rules and feature layer; alerts carry both the sequence model's attention over events and the features that fired; and the SAR narrative is written from the latter. Detection 79%, explanations intact, and the model-risk file records why the two-point cost was taken.
Common mistakes
- End-to-end from input to regulated decision with no intermediate a reviewer can read. - Choosing end-to-end without enough labelled final outputs; the pipeline's small-data stages were the point. - Keeping a pipeline whose compounding errors an end-to-end model would remove, out of habit. - No per-stage tests, so a pipeline's failures are as opaque as a model's. - Treating the multimodal prompt as the whole system; extraction is a stage, the decision is another.