Learn AI Create the Future — with Pranay
Main site

Deep Learning

Gradient descent

Gradient descent finds good weights by repeatedly taking a small step in the direction that lowers the error fastest.

Read these first

On this page 7
  1. Why this had to be invented
  2. How it works
  3. An analogy for the step size
  4. Where you have already seen this
  5. What is honestly hard here
  6. Remember this
  7. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Gradient descent is a way to find the bottom of a valley by always stepping in the steepest downhill direction.

Picture yourself on a hillside in thick morning fog. You want to reach the lowest point, but you can see barely a metre ahead.

You can still feel the ground under your feet. Shift your weight around and you can tell which way slopes downward. So you take a step that way, then feel again, then step again.

You never see the valley floor. You reach it anyway, one careful step at a time. That is gradient descent, and it is how nearly every AI model on earth was trained.

Why this had to be invented

Backpropagation, the step that works out which knob caused the mistake, gives the network a direction for every knob. It does not say how far to move.

Turning a knob all the way in the suggested direction is a bad idea. That direction is only reliable very close to where you stand. "Downhill" on a mountain changes as you walk.

So the network takes a small step, then recalculates. The size of that step is the single most important setting in all of deep learning.

How it works

   feel the slope  →  step downhill a little  →  feel the slope again
          ↑                                              │
          └──────────────────────────────────────────────┘
                        repeat, thousands of times

That step size has a name: the learning rate — how far you move on each step.

Getting it wrong is the most common reason training fails.

  too small:   . . . . . . . . . . . . . . . →  still walking, hours later
  about right: .   .   .    .     .      .   →  arrives, steadily
  too large:      ↗       ↘        ↗         →  bounces over the valley, never lands

The "too large" case surprises people. A step big enough to cross the valley lands you higher up the far side. The next step throws you further still. Instead of settling, the model flies apart, and the error grows every round.

An analogy for the step size

Think about pouring tea into a cup from a height. Pour gently and it takes a while, but the cup fills neatly. Pour hard and most of it splashes out.

You want the largest pour that does not splash. Finding it takes a few tries, and even experienced people run small experiments to settle it.

Where you have already seen this

Nothing you use displays gradient descent to you. But your keyboard suggestions, your bank's fraud alerts and every chatbot arrived at their settings this way.

It is one of the most widely run procedures in computing. It rests on the idea of feeling for downhill.

What is honestly hard here

Real error landscapes are not smooth bowls with one low point. They have ridges, flat plains and many dips, in millions of directions at once.

Nobody can picture that, including researchers. There is no guarantee of finding the very lowest point. In practice, models settle in a dip that is good enough rather than perfect.

That sounds like a weakness. It turns out to work remarkably well, and understanding exactly why is still an open research question. Anyone claiming this is a solved matter is not being straight with you.

Remember this

  • Gradient descent takes small steps downhill, over and over.
  • The learning rate is the step size, and it is the setting most worth tuning.
  • Too large is worse than too small: the error grows instead of shrinking.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy

Two experiments below. The first shows the step-size failure directly. The second checks gradient descent against an answer we can compute exactly.

gradient_descent.py
import numpy as np

# ---- part 1: walking downhill on one number ----
# The hill is f(w) = (w - 3)^2. Its lowest point is at w = 3, but pretend we do not know that.


def height(w):
    return (w - 3.0) ** 2


def slope(w):
    return 2.0 * (w - 3.0)          # the derivative, worked out by hand


print("lr = 0.1  (a sensible step size)")
w = 0.0
for step in range(6):
    print(f"  step {step}:  w = {w:8.4f}   height = {height(w):8.4f}   slope = {slope(w):8.4f}")
    w = w - 0.1 * slope(w)
for _ in range(44):
    w = w - 0.1 * slope(w)
print(f"  ...after 50 steps: w = {w:.5f}   (the true lowest point is w = 3)")

print("\nlr = 1.1  (steps too big - it climbs the walls instead)")
w = 0.0
for step in range(6):
    print(f"  step {step}:  w = {w:12.4f}   height = {height(w):14.4f}")
    w = w - 1.1 * slope(w)

# ---- part 2: fitting a real line ----
x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.1, 4.9, 7.2, 8.9, 11.1])        # roughly y = 2x + 1

w, b = 0.0, 0.0
lr = 0.02
n = len(x)
print("\nfitting y = w*x + b by gradient descent")
for step in range(1, 2001):
    pred = w * x + b
    err = pred - y
    mse = float(np.mean(err ** 2))
    gw = float(2.0 * np.mean(err * x))          # slope of the error wrt w
    gb = float(2.0 * np.mean(err))              # slope of the error wrt b
    w -= lr * gw
    b -= lr * gb
    if step in (1, 10, 100, 500, 2000):
        print(f"  step {step:4d}:  w = {w:.4f}   b = {b:.4f}   mse = {mse:.5f}")

# The exact answer, from the closed-form least-squares formula.
A = np.vstack([x, np.ones_like(x)]).T
exact_w, exact_b = np.linalg.lstsq(A, y, rcond=None)[0]
print(f"\ngradient descent found: w = {w:.4f}  b = {b:.4f}")
print(f"exact algebra says:     w = {exact_w:.4f}  b = {exact_b:.4f}")
Output
lr = 0.1  (a sensible step size)
  step 0:  w =   0.0000   height =   9.0000   slope =  -6.0000
  step 1:  w =   0.6000   height =   5.7600   slope =  -4.8000
  step 2:  w =   1.0800   height =   3.6864   slope =  -3.8400
  step 3:  w =   1.4640   height =   2.3593   slope =  -3.0720
  step 4:  w =   1.7712   height =   1.5099   slope =  -2.4576
  step 5:  w =   2.0170   height =   0.9664   slope =  -1.9661
  ...after 50 steps: w = 2.99996   (the true lowest point is w = 3)

lr = 1.1  (steps too big - it climbs the walls instead)
  step 0:  w =       0.0000   height =         9.0000
  step 1:  w =       6.6000   height =        12.9600
  step 2:  w =      -1.3200   height =        18.6624
  step 3:  w =       8.1840   height =        26.8739
  step 4:  w =      -3.2208   height =        38.6984
  step 5:  w =      10.4650   height =        55.7256

fitting y = w*x + b by gradient descent
  step    1:  w = 1.0048   b = 0.2816   mse = 57.57600
  step   10:  w = 2.1133   b = 0.6172   mse = 0.04779
  step  100:  w = 2.0634   b = 0.8110   mse = 0.02408
  step  500:  w = 2.0042   b = 1.0248   mse = 0.01444
  step 2000:  w = 2.0000   b = 1.0400   mse = 0.01440

gradient descent found: w = 2.0000  b = 1.0400
exact algebra says:     w = 2.0000  b = 1.0400

Reading the results

Part one, lr = 0.1. The slope column shrinks every step: -6.0, -4.8, -3.84. Steps get smaller automatically as the ground flattens, with no extra logic. Gradient descent slows down near the bottom for free.

Part one, lr = 1.1. The height column tells the story: 9.0, 12.96, 18.66, 26.87. It goes up every step, and w alternates in sign while growing. This is divergence, and it looks exactly like this in a real training log — a loss that climbs and then becomes nan.

There is a precise threshold here. For a bowl of this shape, any learning rate below 1.0 converges and anything above it diverges. Real networks have a threshold too; we cannot compute it, so we find it by experiment.

Part two. Gradient descent lands on w = 2.0000, b = 1.0400, and the closed-form least-squares solution agrees to four decimals. When an exact answer exists, gradient descent finds it. We use gradient descent for neural networks precisely because no exact answer exists there.

Notice the mse stops improving after step 500 — 0.01444 to 0.01440. It has not stalled; it has arrived. The residual error is the noise in the data, which no straight line can remove.

Common mistakes

Loss becomes nan after a few steps. Learning rate too high, almost every time. Divide it by ten and rerun before changing anything else.

Loss decreases painfully slowly. Learning rate too low, or unscaled features. If one feature ranges over thousands and another over fractions, the error surface becomes a long narrow canyon and progress crawls. Standardise your inputs.

Tuning the learning rate by guessing. Run a range test instead: start at 1e-7, multiply by 1.1 each batch, and plot loss against learning rate. Pick the value about one order of magnitude below where the loss starts rising. This takes a couple of minutes and beats a day of guessing.

Assuming a lower training loss is better. It is not, past a point. Track validation loss and stop when that stops improving — see overfitting-and-underfitting.

In practice, you use an optimiser

Nobody writes w -= lr * gw in production. Here is the identical problem in PyTorch, starting from the same zeroed weights so the numbers line up.

gradient_descent_torch.py
import torch
import torch.nn as nn

x = torch.tensor([[1.], [2.], [3.], [4.], [5.]])
y = torch.tensor([[3.1], [4.9], [7.2], [8.9], [11.1]])

model = nn.Linear(1, 1)                 # one input, one output: w*x + b
nn.init.zeros_(model.weight)            # start from zero, matching the NumPy version above
nn.init.zeros_(model.bias)

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.02)

for step in range(1, 2001):
    pred = model(x)
    loss = loss_fn(pred, y)

    optimizer.zero_grad()               # gradients accumulate by default, so clear them first
    loss.backward()                     # backpropagation fills in every .grad
    optimizer.step()                    # one downhill step for every parameter

    if step in (1, 10, 100, 500, 2000):
        print(f"  step {step:4d}:  w = {model.weight.item():.4f}   "
              f"b = {model.bias.item():.4f}   mse = {loss.item():.5f}")

print(f"\nPyTorch found: w = {model.weight.item():.4f}  b = {model.bias.item():.4f}")
Output
  step    1:  w = 1.0048   b = 0.2816   mse = 57.57600
  step   10:  w = 2.1133   b = 0.6172   mse = 0.04779
  step  100:  w = 2.0634   b = 0.8110   mse = 0.02408
  step  500:  w = 2.0042   b = 1.0248   mse = 0.01444
  step 2000:  w = 2.0000   b = 1.0400   mse = 0.01440

PyTorch found: w = 2.0000  b = 1.0400

Compare these numbers against the hand-written loop above. They match to every printed digit, at every checkpoint. PyTorch is doing exactly what you wrote by hand — no more, no less.

Those three calls, in that order, are the core of every PyTorch training loop you will ever read:

  optimizer.zero_grad()  →  loss.backward()  →  optimizer.step()
     clear old grads         compute new ones     move downhill

For real networks, swap SGD for AdamW and start around lr=3e-4. AdamW gives each parameter its own effective step size, which is why it copes with problems that plain SGD needs careful tuning to handle.

Try it yourself

In part one, set the learning rate to exactly 1.0 and predict what happens before running. Then try 0.99. The behaviour at the boundary is worth seeing once.

What to learn next

Researcher — Mathematics and papers.

The update, and what guarantees it carries

For an objective $f : \mathbb{R}^{d} \to \mathbb{R}$:

$$ w_{t+1} = w_t - \eta \nabla f(w_t) $$

Where $\eta > 0$ is the step size and $\nabla f(w_t)$ is the gradient at iterate $t$.

Assume $f$ is $L$-smooth, meaning its gradient is Lipschitz continuous with constant $L$:

$$ \left| \nabla f(x) - \nabla f(y) \right| \le L \left| x - y \right| \quad \forall x, y $$

Then the descent lemma gives $f(w_{t+1}) \le f(w_t) - \eta\left(1 - \tfrac{L\eta}{2}\right) \left| \nabla f(w_t) \right|^2$. Progress is guaranteed whenever $0 < \eta < \tfrac{2}{L}$, and the optimal fixed step is $\eta = \tfrac{1}{L}$.

The toy problem above makes this concrete: $f(w) = (w-3)^2$ has $f''= 2$, so $L = 2$ and the divergence threshold is $\eta = 2/L = 1.0$. The observed behaviour matches exactly.

Rates, for reference:

AssumptionRateIterations for $\epsilon$
Convex, $L$-smooth$f(w_t) - f^* = O(1/t)$$O(1/\epsilon)$
$\mu$-strongly convex, $L$-smooth$O!\left(\left(1 - \tfrac{\mu}{L}\right)^{t}\right)$$O!\left(\kappa \log \tfrac{1}{\epsilon}\right)$
Non-convex, $L$-smooth$\min_t \left| \nabla f(w_t) \right|^2 = O(1/t)$$O(1/\epsilon^2)$ to a stationary point

Where $\mu$ is the strong-convexity constant and $\kappa = L/\mu$ is the condition number. Deep networks satisfy only the last row: convergence to a stationary point, with no claim about its quality.

Stochastic gradient descent

Full-batch gradients cost $O(n)$ per step. SGD (Robbins and Monro, 1951) substitutes a mini-batch estimate:

$$ w_{t+1} = w_t - \eta_t \nabla f_{\mathcal{B}_t}(w_t), \qquad \mathbb{E}\left[\nabla f_{\mathcal{B}}\right] = \nabla f $$

The estimator is unbiased with variance $\sigma^2 / |\mathcal{B}|$. Convergence for a fixed $\eta$ stalls at a noise floor proportional to $\eta \sigma^2$, which is why decaying schedules exist. The Robbins–Monro conditions $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$ guarantee convergence to a stationary point.

The gradient noise is not purely a cost. It is widely believed to aid generalisation by biasing solutions toward flatter minima — see Keskar et al. (2017), On large-batch training for deep learning, though the flat-minima/generalisation link remains contested (Dinh et al., 2017).

The optimisers you will actually use

Momentum (Polyak, 1964) accumulates an exponentially weighted average of past gradients:

$$ v_{t+1} = \beta v_t + \nabla f(w_t), \qquad w_{t+1} = w_t - \eta \, v_{t+1} $$

Where $\beta \in [0, 1)$ is the momentum coefficient, typically $0.9$. This damps oscillation across a narrow canyon while accumulating speed along it, improving the strongly-convex rate from $O(\kappa)$ to $O(\sqrt{\kappa})$ in the Nesterov variant (Nesterov, 1983).

Adam (Kingma and Ba, 2015) keeps running estimates of the first and second moments:

$$ m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2 $$

$$ \hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}, \qquad w_{t+1} = w_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} $$

Where:

  • $g_t = \nabla f_{\mathcal{B}_t}(w_t)$ — the mini-batch gradient.
  • $\beta_1 = 0.9$, $\beta_2 = 0.999$ — decay rates for the two moments.
  • $\epsilon = 10^{-8}$ — a small constant preventing division by zero.
  • $\hat{m}_t, \hat{v}_t$ — bias-corrected estimates; without correction both are biased toward zero early in training, since $m_0 = v_0 = 0$.

Dividing by $\sqrt{\hat{v}_t}$ gives each parameter its own effective step size, which is why Adam tolerates badly scaled problems that plain SGD cannot.

AdamW (Loshchilov and Hutter, 2019), Decoupled weight decay regularization, applies weight decay directly to the parameters rather than through the gradient. In Adam, L2 regularisation added to the loss gets divided by $\sqrt{\hat{v}_t}$ along with everything else, which weakens decay exactly on the parameters with large gradients. AdamW is the correct default for transformers.

Practical notes that matter at scale

  • Warmup. Adam's second-moment estimate is unreliable in the first few hundred steps, when the sample is small. A linear warmup over 1–10k steps prevents the early instability. Liu et al. (2020), On the variance of the adaptive learning rate, analyse this.
  • Cosine decay from the peak to near zero is the standard schedule for large model training.
  • Gradient clipping by global norm, typically at $1.0$, prevents rare large batches from destroying a run.
  • Linear scaling rule (Goyal et al., 2017): when multiplying batch size by $k$, multiply the learning rate by $k$ as well. Holds up to roughly 8k examples per batch, then breaks down.
  • Adam stores two extra states per parameter. For a model in fp32, optimiser state alone costs $8$ bytes per parameter — often more memory than the weights themselves.

What to learn next