Tensor
In one sentence A tensor is a grid of numbers with any number of dimensions, and it is the container every deep learning library moves around.
Updated
A tensor is a grid of numbers with any number of dimensions. It is the single data structure that deep learning libraries pass from layer to layer.
Start from things you already know and keep adding a dimension. One number is a temperature reading. A row of numbers is the week's temperatures. A table of numbers is temperatures for seven cities across seven days. A stack of those tables is the same thing for twelve months. Nothing new is introduced at any step — a tensor is that idea with no upper limit on how many dimensions you stack.
| Dimensions | Common name | Example |
|---|---|---|
| 0 | Scalar | A single loss value, 2.31 |
| 1 | Vector | One embedding, 768 numbers |
| 2 | Matrix | A batch of 32 embeddings, (32, 768) |
| 3 | — | A batch of token sequences, (32, 128, 768) |
| 4 | — | A batch of images, (32, 3, 224, 224) |
Reading a shape is a real skill
That last one is worth decoding, because you will read shapes like it every day: 32 images, 3 colour channels, 224 pixels tall, 224 wide. PyTorch puts channels before height and width; TensorFlow usually puts them last. Mixing the two conventions is a classic afternoon lost.
Tensors carry two properties beyond their numbers, and both cause errors when they disagree. The shape must line up for an operation to be legal, which is what mat1 and mat2 shapes cannot be multiplied is telling you. The device says whether the data sits in CPU memory or GPU memory, and two tensors on different devices cannot be combined — that one produces expected all tensors to be on the same device.
The reason everything is a tensor is hardware. A GPU is built to run the same arithmetic on thousands of numbers at the same time, so work expressed as whole-tensor operations runs hundreds of times faster than the equivalent Python loop.
Where to go next
- Full lesson: PyTorch basics
- Related terms: parameter, cnn, embedding, inference