Deep Learning Section 028
Autograd in Depth
PyTorch keeps a record of every calculation so it can work out who caused the error. This section is what that record really is.
10 of 10 lessons published Three reading levels on every lesson
Lessons in order
Work top to bottom. Each lesson assumes the one above it.
- How the autograd graph is built and freed
- requires_grad and leaf tensors
- detach, no_grad and inference_mode
- Calling backward on a non-scalar output
- In-place operations and when they break autograd
- Gradient accumulation for large batches
- Gradient clipping
- Writing a custom autograd Function
- Checking your gradients with gradcheck
- Jacobians, vjp and jvp with torch.func