Autoencoder

An autoencoder is a neural network trained to compress data into a small set of numbers and then rebuild the original from those numbers.

Packing for a two-day trip with one small bag is the same problem. You cannot take the whole cupboard, so you take the things that matter and accept that you will manage without the rest. If you can unpack at the other end and still be dressed properly, your packing captured what was essential. An autoencoder is trained by being marked on exactly that: how close the rebuilt version is to the original.

The network is two halves joined at a narrow waist. The encoder shrinks the input down to a short code, often called the latent vector. The decoder expands that code back out. Because the waist is far too small to store a copy, the only way to score well is to learn the structure of the data — what faces have in common, what handwritten digits have in common.

The shape of it#

Image 784 numbers → Encoder → code: 32 numbers → Decoder → Image 784 numbers
                                  ▲
                        everything must fit through here

That squeeze is useful beyond compression. Feed a noisy image in and train against the clean one, and you get a denoiser. Feed normal machine sensor readings in during training, and later a reading that rebuilds badly is probably a fault — that is anomaly detection, and it works because the model never learned to represent failures. The variational autoencoder adds structure to the latent space so you can sample new points from it, an idea that reappears inside modern image generators, which do their work in a compressed latent space rather than on raw pixels.

Where to go next#

• Full lesson: Autoencoders
• Related terms: embedding, gan, tensor, parameter