Tensor Processing Unit
A tensor processing unit is Google's application-specific chip for neural-network arithmetic — designed from 2015 for the matrix multiplications that training and inference consist of, deployed in Google's data centres in pods of thousands, and rented through Google Cloud. It is the best-known of the custom accelerators the hyperscalers built to reduce dependence on Nvidia's GPUs, and the chip Google's own models are trained and served on.
- Also known as
- TPU, Google TPU, Tensor core
- Used at
- Google's own model training and serving; customers of Google Cloud renting TPU capacity (Anthropic among the largest, alongside its use of other accelerators); the "Tensor" chips in Google's phones are a different, smaller product with the same name
- Binding
- available only through Google Cloud — no TPUs are sold as hardware — which ties users to that cloud and its software stack (JAX, TensorFlow, PyTorch support via XLA)
- Typical generations
- TPU v1 (2015, inference only), v2 and v3 (training, pods), v4 (2021), v5e and v5p (2023), v6 "Trillium" (2024), with each generation's pods scaling to thousands of chips interconnected by Google's own network; per-chip performance competitive with contemporary Nvidia GPUs, with the pod's interconnect a strength
- Typical rationale for custom chips
- the hyperscalers spend billions a year on accelerators; a chip designed for their workloads, made by TSMC, costs them less per unit of compute than Nvidia's margin allows, and gives them supply independent of one vendor — Amazon (Trainium, Inferentia), Microsoft (Maia) and Meta (MTIA) followed
- Parties
- Google (design and operation), TSMC (manufacture), Broadcom (design partnership), cloud customers, and the labs choosing between TPU and GPU capacity
How a tensor processing unit works in practice
A neural network's layers multiply inputs by weight matrices. A TPU's core is a systolic array — a grid of multiply-accumulate units through which data flows in lockstep, so a matrix multiplication proceeds without the memory traffic a general processor would need — surrounded by high-bandwidth memory and a fast interconnect to neighbouring chips. Training a large model spreads the work across a pod; Google's software (XLA) compiles the model's computation graph to the chips and manages the communication. The user, a cloud customer, sees a virtual machine with TPUs attached and a framework that runs on them; the pod, the interconnect and the compiler are Google's.
The strategic story is supply. Nvidia's GPUs became the default for AI because of CUDA's ecosystem as much as the hardware, and every large buyer of AI compute became dependent on one supplier with a 70%-plus gross margin. Google, which had built TPUs for its own search and ads workloads, offered them to cloud customers as an alternative; the other hyperscalers designed their own; and the labs — Anthropic, OpenAI, Meta — signed for capacity across chips to avoid dependence on any one. For the AI economy the custom chips are a check on Nvidia's pricing and a reason the cloud providers' AI margins differ.
For a fintech the TPU is a line on a cloud pricing page — relevant if it trains large models on Google Cloud, irrelevant if it uses models over an API.
Example
A fintech on Google Cloud trains a transaction-classification model. On the cloud's GPU instances the run costs $2,600; on a TPU v5e slice, with the model's code adapted to run through XLA, $1,700 — a saving that took an engineer two weeks of porting to achieve, which the team judges worthwhile for a model it retrains weekly. Its language-model workloads use Gemini over an API, served on TPUs the fintech never sees. When the fintech considers moving clouds a year later, the TPU-specific code is one of the switching costs its CTO lists, and the reason its next model is written to run on either chip.
Common mistakes
- Assuming custom chips are interchangeable with GPUs without the porting work. - Choosing an accelerator on per-chip benchmarks rather than the pod's throughput on the actual workload. - Locking a training pipeline to one cloud's chip and discovering it in a migration. - Reading the phone "Tensor" chip as the data-centre TPU. - Believing a fintech's model choices depend on any of this when an API would do.