Debugging a distributed job that hangs

Setup#

pip install torch

Captured on CPU with the gloo backend and two processes. Reproducing a real hang is the point, so both scripts below deliberately deadlock and then time out.

Reproducing the classic hang#

from datetime import timedelta
import torch
import torch.distributed as dist

# a short timeout turns a silent hang into a loud error you can read
dist.init_process_group(backend="gloo", timeout=timedelta(seconds=10))
rank = dist.get_rank()

t = torch.ones(1)
dist.all_reduce(t)
print(f"rank {rank}: first all_reduce done")

if rank == 0:                      # THE BUG: only rank 0 makes this call
    dist.all_reduce(t)
    print("rank 0: second all_reduce done")

print(f"rank {rank}: reached the end")
dist.destroy_process_group()

torchrun --nproc_per_node=2 hang_demo.py

rank 1: first all_reduce done
rank 1: reached the end
rank 0: first all_reduce done
Traceback (most recent call last):
  File "hang_demo.py", line 14, in <module>
    dist.all_reduce(t)
RuntimeError: [.../gloo/transport/uv/unbound_buffer.cc:67] Timed out waiting
10000ms for recv operation to complete

That error took ten seconds to arrive because of the timeout argument. Remove it and the default gloo wait is 30 minutes of complete silence. The path inside the brackets is a build path and differs between platforms and versions; the words after it are the part that matters.

Read the log the way an investigator would. Rank 1 printed "reached the end" and rank 0 did not. The rank that is missing from the last log line is the one still waiting, and the rank that finished early is usually the one that skipped a call.

Finding the culprit by name#

from datetime import timedelta
import time
import torch.distributed as dist

dist.init_process_group(backend="gloo", timeout=timedelta(seconds=30))
rank = dist.get_rank()

if rank == 1:
    time.sleep(25)          # pretend rank 1 is stuck in a slow dataloader

try:
    dist.monitored_barrier(timeout=timedelta(seconds=5))   # gloo only
except RuntimeError as e:
    print(f"rank {rank}:", str(e).splitlines()[0])

rank 0: [Rank 0]: Rank 1 failed to pass monitoredBarrier in 5000 ms

monitored_barrier is an ordinary barrier that reports which rank failed to arrive. It is gloo-only, so on a GPU job you create a small side process group for it:

gloo_group = dist.new_group(backend="gloo")     # alongside your nccl group
dist.monitored_barrier(group=gloo_group, timeout=timedelta(seconds=60))

Drop that call at a few suspected points in your loop and the hang stops being anonymous.

A checklist that finds most hangs#

1. Set a timeout at startup, permanently.

dist.init_process_group(backend="nccl", timeout=timedelta(minutes=10))

2. Turn on the NCCL watchdog so a stuck collective aborts with a rank named, rather than spinning:

export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
export TORCH_NCCL_TRACE_BUFFER_SIZE=2000    # keeps the last 2000 collectives per rank
export NCCL_DEBUG=WARN                      # INFO when you need the topology dump

3. Turn on the distributed consistency checks:

export TORCH_DISTRIBUTED_DEBUG=DETAIL

This verifies that every rank passes matching shapes and dtypes to each collective, and reports mismatches instead of hanging. It is slow, so use it while diagnosing and remove it afterwards.

4. Look at where the Python code actually is. py-spy dump --pid <pid> prints a live stack trace of a running process without stopping it. Run it against one process per rank; the rank whose stack differs is the offender.

pip install py-spy
py-spy dump --pid 12345

5. Search your own code for these five patterns:

The uneven-batches case is the sneaky one. It comes free with DistributedSampler in training, and it comes straight back if you write a custom loader or filter samples inside the loop.

Hangs that are not deadlocks#

Not everything that stops is a missed collective.

Dataloader workers deadlocking. A num_workers > 0 loader that forks after CUDA has been initialised can wedge. Symptom: the hang happens on the first batch, before any collective. Try num_workers=0 to confirm — see dataloader workers and speed.

A crashed rank you were not told about. torchrun should tear the group down, but an out-of-memory kill by the operating system can leave a zombie. Check dmesg for the OOM killer and your job's exit codes per rank.

A network path that never opened. If the hang is at init_process_group rather than in the loop, it is connectivity, not logic. Test the port directly with nc -zv <master_addr> 29500.

Genuine slowness mistaken for a hang. One straggler rank running at a tenth of the speed of the others makes every step slow, not stuck. Print per-rank step times before you conclude anything.

Common mistakes#

Debugging at full scale. Reproduce on 2 ranks on one machine with gloo first. Almost every logic hang reproduces there, in seconds, with readable errors.

Leaving the default 30-minute timeout. You lose half an hour per attempt, and the eventual message is less useful than one from a short timeout.

Adding barrier() calls to "fix" it. A barrier in the wrong place creates a fresh deadlock. Barriers are diagnosis, not treatment.

Treating a NCCL error's line number as the fault site. NCCL calls are asynchronous, so the error surfaces at whatever line the CPU had reached. The flight recorder buffer, not the traceback, tells you which collective actually failed.

Try it yourself#

Change hang_demo.py so both ranks call all_reduce twice but rank 1 passes torch.ones(2) instead of torch.ones(1). Predict what happens, then run it with and without TORCH_DISTRIBUTED_DEBUG=DETAIL and compare the messages.

What to learn next#

• all_reduce, all_gather and broadcast — the calls that must match across every rank.
• Training across several machines — the environment where these hangs get expensive.
• Memory leaks in training loops — the other slow failure that looks like nothing is wrong.