Self-supervised learning

Self-supervised learning creates its own labels by hiding part of the data and training the model to reconstruct the hidden part.

Cover the last word of a newspaper sentence with your thumb and guess it. "The minister announced the new ___." You check by lifting your thumb. Nobody wrote an answer key — the sentence is the answer key. Repeat this across every sentence ever printed and you have unlimited exam questions with guaranteed correct answers, for free.

That is the trick powering modern AI. Predicting the next token trains GPT-style models; predicting masked-out middle words trained BERT; matching images to their captions trained CLIP. In each case the supervision is manufactured from raw data, so the labelling bottleneck of supervised-learning vanishes — the training set becomes "all text we can gather".

The deep insight is why this works so well. Guessing the hidden word forces the model to learn grammar, facts, and reasoning, because those are what make the guess accurate. The task is a pretext; the knowledge is the point.

In the standard recipe, self-supervised pretraining builds general capability, then a small supervised stage — fine-tuning or instruction-tuning — shapes it for actual use.

Where to go next#

• Full lesson: How LLMs work
• Related terms: pretraining, supervised-learning, llm, bert