Deep learning questions, with answers

Setup#

Questions 6–15. Two are answered with runnable NumPy.

pip install numpy

CPU-only, under two seconds, no downloads. Outputs from NumPy 2.4; the seeds make them reproducible.

Q6. Show me the vanishing gradient problem — don't recite it#

The strongest possible answer builds a 30-layer network and measures the correction signal arriving at each depth, once with sigmoid activations, once with ReLU:

import numpy as np


def sigmoid(z):
    return 1 / (1 + np.exp(-z))


width, depth = 64, 30


def gradient_reaching_each_layer(kind):
    rng = np.random.default_rng(0)                 # identical weights for both runs
    h = rng.normal(size=width)
    tape = []                                      # backprop needs W and the local slope
    for _ in range(depth):
        W = rng.normal(0, np.sqrt(2 / width), (width, width))   # He initialisation
        z = W @ h
        if kind == "relu":
            h = np.maximum(z, 0)
            local = (z > 0).astype(float)          # relu slope: exactly 1 where active
        else:
            h = sigmoid(z)
            local = h * (1 - h)                    # sigmoid slope: never above 0.25
        tape.append((W, local))

    grad = np.ones(width)                          # the gradient arriving from the loss
    norms = {}
    for layer in range(depth, 0, -1):
        W, local = tape[layer - 1]
        grad = W.T @ (grad * local)                # one chain-rule step, walking down
        norms[layer] = np.linalg.norm(grad)
    return norms


sig = gradient_reaching_each_layer("sigmoid")
relu = gradient_reaching_each_layer("relu")
print("gradient size reaching layer:    sigmoid       relu")
for layer in (30, 20, 10, 1):
    print(f"{layer:>27}    {sig[layer]:>9.1e}  {relu[layer]:>9.1e}")

gradient size reaching layer:    sigmoid       relu
                         30      2.0e+00    8.5e+00
                         20      2.6e-05    2.0e+01
                         10      4.1e-10    2.4e+01
                          1      1.2e-14    3.2e+01

With sigmoid, the signal shrinks fourteen orders of magnitude on its way down — layer 1 receives numerical silence and cannot learn. The mechanism: backprop multiplies one local slope per layer, sigmoid's slope never exceeds 0.25, and thirty numbers below 0.25 multiplied together is nearly nothing. ReLU's slope is exactly 1 on active units, so the product survives — 32 at the bottom, thirty layers down.

Then name the full toolkit, because one activation swap is not the whole cure: careful initialisation (Q16 territory), residual connections (Q14), normalisation layers (Q8), and gradient clipping for the exploding twin, where the product grows instead of shrinking — common in RNNs.

Q7. Why not initialise all weights to zero?#

Because every neuron in a layer would then compute the same output, receive the same gradient, and take the same update — forever. The layer collapses into one neuron photocopied. This is the symmetry problem: identical starting points can never differentiate.

Random initialisation breaks the tie, letting different neurons specialise in different features. The follow-up — "random with what size?" — is Q16, and the demo above quietly used its answer (np.sqrt(2 / width), He initialisation).

Q8. What does batch normalisation do, and why does training differ from inference?#

Batch norm re-centres and re-scales each layer's pre-activations, using the mean and spread of the current batch — the group of examples processed together. It then lets the network undo the operation via two learned parameters per channel, so nothing is forced. In practice it allows higher learning rates, makes training far less sensitive to initialisation, and speeds convergence.

The trap inside the question is train-versus-inference. At inference there may be no batch — one image arrives alone — so the layer switches to averages collected during training. Forgetting to flip that switch (model.eval() in PyTorch) is a classic real-world bug: the model behaves differently one example at a time than in batches. Mentioning that bug unprompted is the strong version. Why batch norm works is genuinely contested — that story is Q17.

Q9. Explain dropout — and why it is off at test time#

Dropout randomly silences a fraction of neurons on every training step, a different set each time. No neuron can rely on a specific partner existing, so the network cannot build fragile co-dependent circuits — it must spread knowledge redundantly. It behaves like training many thinned networks that share weights, then averaging them.

At test time you want the full ensemble's stable answer, not a random thinned network — so nothing is dropped, and activations are rescaled to keep their expected size consistent with training (standard "inverted dropout" does the rescaling during training instead). The one-line summary interviewers reward: "noise at training time, deterministic average at test time."

Q10. Adam versus SGD — what does Adam actually track?#

Adam keeps two running averages per parameter: the recent gradient direction (momentum) and the recent gradient magnitude. It divides one by the other, so parameters with consistently large gradients take careful steps and parameters with faint gradients take relatively bigger ones — a per-parameter learning rate, tuned automatically.

Trade-offs, which are the marks: Adam is robust to learning-rate choice and dominates in practice for transformers and sparse-gradient problems. Plain SGD with momentum, tuned well, has matched or beaten it on classic vision benchmarks and was long the choice for state-of-the-art CNNs. The senior detail: modern practice uses AdamW, which fixes how weight decay interacts with Adam's scaling — Q18 has the equations. Foundations: gradient-descent.

Q11. The loss is not going down. Walk me through your checklist#

Ordered hypotheses, cheapest first:

1. Overfit one batch. Train on ten examples; a healthy network memorises them in seconds. Failure here means a bug — wrong labels, wrong loss, frozen weights — not a tuning issue. This is the single highest-value trick in the round.
2. Learning rate. Too high: loss spikes, oscillates, or turns nan. Too low: loss creeps down at a crawl. Sweep powers of ten.
3. Data pipeline. Look at actual batches with your eyes. Shuffled labels, un-normalised inputs and duplicate-leaking splits all live here.
4. Gradient flow. Print gradient norms per layer, like Q6's demo — silence at early layers or explosions at late ones each has its fix.

The framing sentence: "loss not falling is a symptom with about four common diseases; I test them in order of how cheap the test is."

Q12. How many parameters in a conv layer — and why so few?#

The formula in words: channels-in times kernel-height times kernel-width times channels-out, plus one bias per output channel. Interviewers want the arithmetic done and the contrast drawn:

in_ch, out_ch, k = 3, 64, 3
conv = in_ch * k * k * out_ch + out_ch          # weights + one bias per filter
print(f"conv 3x3, 3 -> 64 channels: {conv:,} parameters")

h, w = 224, 224                                  # a dense layer doing the same job
dense = (in_ch * h * w) * (out_ch * h * w)
print(f"dense layer on the same image: {dense:,} parameters")

conv 3x3, 3 -> 64 channels: 1,792 parameters
dense layer on the same image: 483,385,147,392 parameters

1,792 against 483 billion. The gap is the answer to "why convolutions": the parameter count is independent of image size, because the same 3×3 detector slides everywhere — Q3's weight sharing, quantified. Follow-up to expect: parameters are not activations. The conv layer's output is huge (64 channels at full resolution), which is why memory during training is dominated by activations, not weights.

Q13. What actually changes when I increase the batch size?#

Mechanically: each update averages more examples, so the gradient estimate is less noisy, and the hardware runs more efficiently. The naive conclusion — "bigger is better" — is the trap.

What to say: larger batches take fewer, cleaner steps per epoch, and typically need a raised learning rate to compensate; the practical recipe is scaling the learning rate with the batch size, with a warm-up period. Very large batches were long associated with worse generalisation ("sharp minima"); later work showed much of the gap closes with proper tuning, but the noise of small batches does act as a mild regulariser. The honest close: "batch size is mostly an engineering choice — I pick the largest that fits memory, then retune the learning rate."

Q14. Why do residual connections let us train 100-layer networks?#

A residual connection adds a layer's input directly to its output, so the layer learns only the change it should make, not the whole transformation. Two consequences, and naming both is the full answer.

For gradients: the addition creates a highway — during backprop, the correction signal flows through the identity path untouched, sidestepping Q6's shrinking product entirely. For optimisation: learning "change nothing" means pushing weights towards zero, which is easy — so extra layers can only help, never strand the network. Before ResNets, deeper networks often trained worse than shallow ones on training data itself — an optimisation failure, not overfitting — and residual connections removed it. They are also why transformers stack to enormous depth.

Q15. Why do softmax and cross-entropy appear together?#

Softmax turns raw scores into probabilities; cross-entropy punishes the model by how little probability it gave the true class. Paired, something beautiful happens: the gradient at the output collapses to predicted probability minus true label — the cleanest error signal possible, large exactly when the model is confidently wrong.

The pairing is also numerically deliberate: computed separately, a confident wrong prediction can produce a log of nearly zero and overflow; fused (as every framework's CrossEntropyLoss does, via the log-sum-exp trick), it is stable. The mistake this question fishes for: putting an explicit softmax layer before a fused loss, applying it twice — a real, common, silent bug. Deriving the collapse is Q19's warm-up; the loss family lives in loss-functions.

Common mistakes in this round#

Reciting cures without mechanisms. "Use ReLU, use batch norm, use residuals" without why reads as cargo cult. Every "use X" needs its failure story.

Never having watched a training curve. Q11 exposes this instantly. Train small things and break them on purpose; the pytorch-basics lesson is the place to start.

Treating contested questions as settled. Why batch norm helps, whether large batches generalise worse — these have live literatures (see the researcher tab). Presenting folklore as fact costs marks against a well-read interviewer.

Try it yourself#

In vanishing_gradients.py, swap He initialisation for rng.normal(0, 0.01, ...) and predict what happens to the ReLU column before running — small weights shrink the product too, activation choice notwithstanding. Then reduce depth to 5 and see how much the sigmoid column recovers. Shallow networks are why sigmoid survived for decades.

What to learn next#

• Backpropagation — the machinery underneath half of these questions.
• ML coding interviews — where these ideas get implemented under a timer.
• Transformers — the architecture the next question set is built on.