Autograd in Depth

detach, no_grad and inference_mode

Three ways to switch the recorder off — detach cuts one tensor loose, no_grad silences a block of code, inference_mode locks the door for speed.

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

detach, no_grad and inference_mode all stop PyTorch's recording — one for a single tensor, one for a stretch of code, one for good.

Think of a courtroom stenographer typing every word into the official record. Three different requests stop her pen. "Strike that last remark" removes one statement. "Off the record for a moment" pauses her for a stretch of conversation. And adjourning the court ends recording entirely — nothing said after can ever enter the record.

PyTorch records your calculations the same way, and offers the same three requests.

Why it exists

Recording costs memory — every recorded step saves values for the backward walk. But much of what a program computes never needs a backward walk. Evaluating on test data. Making predictions for a user. Copying a number into a log.

Recording those is pure waste, sometimes most of your memory. So you tell the recorder when to stop:

  • detach() — strike one tensor from the record. Same numbers, no history.
  • no_grad — off the record for this block of code.
  • inference_mode — court adjourned: fastest, and permanent for what is made inside.

How it works

recorded world              off the record
    x ── y = x*3 ──────────  d = y.detach()      one tensor cut loose
    x ──────────  with no_grad:  z = x*3         a block unrecorded
    x ──────────  with inference_mode:  w = x*3  unrecorded AND locked

A real example you have seen

Every chatbot answer you have ever received was computed with the recorder off. Training happened months earlier in a lab; your conversation is pure prediction. Serving millions of users with the recorder on would need vastly more memory for zero benefit — inference mode is why the answer arrives fast and cheap.

Remember this

  • detach(): one tensor, freed from its history.
  • no_grad: a block of code, nothing recorded.
  • inference_mode: like no_grad, faster — but its tensors can never rejoin training.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch

Outputs verified with torch 2.5.1, CPU.

The three switches, side by side

switches.py
import torch

x = torch.tensor(2.0, requires_grad=True)

y = x * 3
d = y.detach()                  # same values, cut out of the recording
print(y.requires_grad, d.requires_grad)

with torch.no_grad():           # a room where nothing is recorded
    z = x * 3
print(z.requires_grad)

with torch.inference_mode():    # a stricter, faster room
    w = x * 3
print(w.requires_grad, w.is_inference())
Output
True False
False
False True

The walkthrough

detach() does not copy. It returns a new tensor object sharing the same memory, with the history link removed:

shared.py
import torch

a = torch.ones(3, requires_grad=True)
b = a.detach()          # NOT a copy: b shares memory with a
b[0] = 99.
print(a)                # a changed too
Output
tensor([99.,  1.,  1.], requires_grad=True)

Editing a detached tensor edits the original — and can corrupt values autograd saved for a pending backward. When you want an independent, history-free copy, the idiom is y.detach().clone().

no_grad is for code you control around a computation — the standard wrapper for validation loops and for optimiser updates. Everything inside produces untracked tensors; everything outside behaves normally. Note what it does not do: it does not switch dropout or batch-norm to evaluation behaviour — that is model.eval(), a separate and independent switch. Every serious evaluation loop uses both.

inference_mode is no_grad with the door welded shut. It skips autograd bookkeeping that no_grad still pays for (version counters — see in-place operations), making it the right default for pure prediction. The price is strictness:

python
x = torch.tensor(2.0, requires_grad=True)
with torch.inference_mode():
    t = torch.ones(3) * 2
out = t * x            # using an inference tensor in recorded code
Output
RuntimeError: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.

A tensor born in inference mode can never participate in training, even later. The error tells you the escape hatch — .clone() outside the block — but treat needing it as a sign the tensor was made in the wrong place.

Where detach earns its living

The canonical use is a value that should influence the numbers but not receive blame. Real examples you will meet: the target network in DQN-style reinforcement learning, teacher outputs in distillation, and metrics — running_loss += loss.detach() keeps a graph-free number, where forgetting the detach keeps the entire recording alive and leaks memory every step.

Common mistakes

Detaching somewhere inside the model and killing learning upstream. Everything before a detach receives zero gradient, silently. If some layers never learn, hunt for a stray .detach(), .item(), .numpy(), or an int cast on the path — all of them cut the recording.

model.eval() versus no_grad confusion. eval() changes layer behaviour (dropout off, batch-norm uses running statistics); no_grad changes recording. One without the other is a real, common bug in both directions: eval-only still burns memory; no_grad-only validates a model with dropout still firing.

Using inference_mode for anything that might feed training later. Cached features, precomputed embeddings — if any downstream code differentiates through them, make them under no_grad instead, or clone on the way out.

.data in modern code. Old tutorials use tensor.data where detach() belongs. .data skips safety checks autograd relies on and can produce silently wrong gradients; in torch 2.x code it is a red flag.

Try it yourself

Train the tiny network from PyTorch basics for a few steps, once accumulating loss and once loss.detach(), and watch memory with psutil — the difference is the recording. Then take a two-layer model, detach between the layers, train, and confirm layer 1's weights never move.

What to learn next

Researcher — Mathematics and papers.

Semantics in graph terms

detach() creates an alias node with grad_fn=None sharing storage: the computational graph is cut at that edge, so the adjoint at the detach point is defined as zero into the upstream subgraph. Formally, for L = f(g(x).detach()), we assert ∂L/∂x = 0 — detach implements the stop-gradient operator of the broader literature (sg[·] in many papers).

Stop-gradient is load-bearing in several method families: target networks in deep Q-learning (Mnih et al., 2015, Human-level control through deep reinforcement learning, Nature 518); the straight-through estimator, where forward uses a non-differentiable quantity and backward pretends identity — x + sg[q(x) − x] in VQ-VAE (van den Oord et al., 2017, Neural Discrete Representation Learning; STE per Bengio et al., 2013, arXiv:1308.3432); and non-collapse mechanisms in self-supervised learning, where BYOL/SimSiam depend on asymmetric stop-gradients (Chen and He, 2021, Exploring Simple Siamese Representation Learning — their ablation shows collapse without it).

no_grad versus inference_mode, mechanically

no_grad sets a thread-local flag suppressing node construction; outputs are untracked, but remain ordinary tensors with version counters, and can be used in recorded computations later (as constants). inference_mode (stable since torch 1.10; unchanged in 2.x) additionally skips version-counter maintenance and view-tracking metadata. The engine cannot verify such tensors were unmodified since any hypothetical save — so it refuses to save them for backward at all, which is the error in the developer block. Guarantees: strictly less bookkeeping per op, hence the measurable speedup on small-op-dominated inference workloads; magnitude is model- and version-dependent, so benchmark rather than assume.

torch.set_grad_enabled(bool) is the parameterised form of no_grad; decorator forms of both contexts exist. A related tool, torch.no_grad() around parameter updates, is what keeps optimiser steps out of the graph — otherwise w_{t+1} = w_t − η·g would make every future loss differentiate through the optimisation history (deliberately done in meta-learning, e.g. MAML, Finn et al., 2017, which omits the no_grad to differentiate through SGD itself).

Aliasing discipline

Because detach shares storage, in-place writes through either alias are observable through both; autograd's version-counter check catches mutations that would invalidate saved tensors, raising at backward time (see in-place operations). detach() inside inference_mode yields an inference tensor regardless; cloning is the only export path. The safe patterns reduce to two: detach() when downstream is read-only; detach().clone() when anyone might write.

References

  • PyTorch docs, Autograd mechanics, §"Locally disabling gradient computation" — the normative comparison table of the three mechanisms.
  • Mnih et al. (2015), Nature 518 — target networks.
  • van den Oord et al. (2017), arXiv:1711.00937 — stop-gradient in VQ-VAE.
  • Chen and He (2021), CVPR — stop-gradient preventing representational collapse.

What to learn next