Learn AI Create the Future — with Pranay
Main site

Inside a Transformer Section 081

Mixture of Experts

How a model can have hundreds of billions of parameters but only use a few of them for each word you type.

Every lesson in this section is written by .

10 of 10 lessons published Three reading levels on every lesson

Start with “What a mixture of experts really is”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. What a mixture of experts really is
  2. The router
  3. Load balancing and expert collapse
  4. Expert capacity and dropped tokens
  5. Shared and fine-grained experts
  6. Active parameters vs total parameters
  7. Serving a MoE across GPUs
  8. A working MoE layer in PyTorch
  9. Mixture of depths
  10. Early exit and layer skipping