Reading and Reimplementing Papers
Decoding the notation
Paper notation is compression, not difficulty — a small vocabulary of symbols covers most ML papers, and a shape audit catches the places where you decoded it wrong.
- 12 min read
- 3 reading levels
- Published
Read these first
On this page 6
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Paper notation is shorthand for things you already understand — a compressed code, not a harder kind of thinking.
A cricket scorecard reads "10-2-31-4". Four numbers, no words. Anyone who follows the game reads it instantly: ten overs bowled, two maidens, thirty-one runs conceded, four wickets taken. To someone who has never been told the order, it is a meaningless string.
The information was never hard. The code had to be learned once.
Research papers work the same way. A line that looks like a wall of Greek letters is usually saying something you could describe in one English sentence. The barrier is vocabulary, and vocabulary is finite.
Why papers are written this way
Space, and precision.
Writing "take the average, over every example in the training set, of the difference between the predicted value and the true value, squared" costs a paragraph. The compressed form costs one line, and every reader who knows the code reads it the same way.
That second part matters more than the space saving. English is ambiguous. "The average error over the batch" could mean three different things. Notation pins down exactly which, including what is being averaged over and in what order.
What the pieces do
Every symbol in an ML paper is doing one of four jobs.
- Naming a thing — a number, a list of numbers, or a table of numbers.
- Saying which one — a small mark below a name meaning "the third example" or "the current step".
- Saying what to do — add these up, take the average, take the biggest.
- Saying which is an estimate — a small hat above a name, meaning "our guess at" rather than "the true".
That is close to the whole game. A page that looks terrifying is these four jobs, repeated.
How it works
a line of a paper → an English sentence → a line of code
looks like a wall "average the squared ((pred - true) ** 2).mean()
of symbols difference over all
the examples"Decode in that direction, always. Symbols to English first, English to code second. Going straight from symbols to code is where mistakes hide.
The honest bit
Some notation is genuinely badly written. Authors reuse the same letter for two things, forget to define a symbol, or use a convention from their subfield without saying so.
When you are stuck on a line for twenty minutes, the odds are not that you are slow. The odds are that the line is under-specified. This is confusing for almost everyone the first time. Read it twice, then go looking for the paper's code — that is not giving up, it is the standard move.
Remember this
- Notation is a code to learn once, not a harder kind of maths.
- Every symbol names a thing, picks one out, does something, or marks an estimate.
- Decode symbols → English → code, never symbols → code.
What to learn next
- Turning an equation into tensor code — the next step once the symbols are decoded.
- Tensor shapes and broadcasting — the check that catches most decoding errors.
- einsum in PyTorch — writing index notation directly as code.
Developer — Code and libraries.
The vocabulary that covers most papers
| In the paper | What it says | In code |
|---|---|---|
| $x_i$ or $x^{(i)}$ | the $i$-th example | X[i] |
| $x^{(i)}_j$ | feature $j$ of example $i$ | X[i, j] |
| $\theta$ or $w$ | every learnable parameter, bundled | model.parameters() |
| $\nabla_\theta L$ | the gradient of $L$ with respect to every parameter | loss.backward() |
| $\mathbb{E}_{x \sim \mathcal{D}}[f(x)]$ | the average of $f$ over the data | f(X).mean() |
| $\sum_{i=1}^{n}$ | add over the examples | .sum(axis=0) |
| $\odot$ | multiply elementwise | a * b |
| $a^\top b$ or $\langle a, b \rangle$ | dot product | a @ b |
| $\lVert x \rVert_2$ | length of a vector | np.linalg.norm(x) |
| $\mathbb{1}{c}$ | 1 when $c$ holds, else 0 | c.astype(float) |
| $\hat{x}$ | an estimate or corrected version of $x$ | x_hat |
| $\beta_1^t$ | $\beta_1$ raised to the power $t$ | beta1 ** t |
The last row is the one that bites, and the paper we are working through says so explicitly.
Setup
pip install scikit-learnVerified with numpy 1.26.4 and scikit-learn 1.7.2 on CPU.
Decoding Adam's Algorithm 1
Kingma and Ba's paper defines nine symbols in ten lines of pseudocode. Here is the whole cast, with the shapes made visible — the fastest way to check you read them correctly.
import numpy as np
theta = np.zeros((2, 3)) # the parameters, one number per weight
g = np.ones((2, 3)) * 0.4 # the gradient, same shape as the parameters
m = np.zeros_like(theta) # first-moment estimate, same shape again
v = np.zeros_like(theta) # second-moment estimate, same shape again
alpha, beta1, beta2, eps = 0.001, 0.9, 0.999, 1e-8 # four plain numbers
for name, val in [("theta", theta), ("g", g), ("m", m), ("v", v)]:
print(f"{name:6s} shape {val.shape} dtype {val.dtype}")
for name, val in [("alpha", alpha), ("beta1", beta1), ("beta2", beta2), ("eps", eps)]:
print(f"{name:6s} scalar {val}")
print("\nbias-correction divisor 1 - beta1^t, under two readings of the superscript:")
for t in range(1, 6):
power = 1 - beta1 ** t # the paper's meaning: beta1 to the power t
index = 1 - beta1 # the misreading: 'the t-th beta', a constant
print(f" t={t} power-of-t {power:.4f} misread-as-index {index:.4f}")theta shape (2, 3) dtype float64 g shape (2, 3) dtype float64 m shape (2, 3) dtype float64 v shape (2, 3) dtype float64 alpha scalar 0.001 beta1 scalar 0.9 beta2 scalar 0.999 eps scalar 1e-08 bias-correction divisor 1 - beta1^t, under two readings of the superscript: t=1 power-of-t 0.1000 misread-as-index 0.1000 t=2 power-of-t 0.1900 misread-as-index 0.1000 t=3 power-of-t 0.2710 misread-as-index 0.1000 t=4 power-of-t 0.3439 misread-as-index 0.1000 t=5 power-of-t 0.4095 misread-as-index 0.1000
Two things fall out.
Four arrays share one shape. Once you see that m and v are the same shape as the parameters, Algorithm 1 stops being abstract. It is elementwise arithmetic on four same-shaped arrays plus four scalars, repeated. Nothing in it is harder than that.
The superscript trap is real, and it is silent. Both readings agree at step 1, then diverge forever. Misread it and your optimiser still runs, still decreases the loss, and quietly takes wrong-sized steps for the rest of training. The Adam paper heads this off in the text by stating that these superscripts denote powers — a sentence worth reading in every paper that uses one.
Check your reading against a reference
Whenever the quantity you decoded already exists in a library, decode it and compare. Here is the standard multi-class loss, written out from its usual form, checked against scikit-learn.
import numpy as np
from sklearn.metrics import log_loss
rng = np.random.default_rng(0)
n, k = 5, 3 # n = number of examples, k = number of classes
z = rng.normal(size=(n, k)) # z[i, j] is the score for example i, class j
labels = rng.integers(0, k, n) # the true class of each example
# The written form: L = - (1/n) * SUM_i log( exp(z[i, y_i]) / SUM_j exp(z[i, j]) )
p = np.exp(z) / np.exp(z).sum(axis=1, keepdims=True) # the inner fraction
picked = p[np.arange(n), labels] # z[i, y_i]: one pick per row
mine = -np.mean(np.log(picked)) # the 1/n SUM_i and the minus
print("shapes: z", z.shape, " p", p.shape, " picked", picked.shape)
print("my reading of the formula:", round(float(mine), 6))
print("sklearn log_loss :", round(float(log_loss(labels, p, labels=range(k))), 6))shapes: z (5, 3) p (5, 3) picked (5,) my reading of the formula: 1.088654 sklearn log_loss : 1.088654
Six decimal places of agreement means the decoding is right. Disagreement would have told you which part — a wrong axis, a missed minus sign, a mean where a sum belonged.
The walkthrough
Two subscripts, two axes. The inner sum runs over classes (axis=1), the outer over examples (axis=0). Getting these the wrong way round produces a number rather than an error, which is why the reference check matters.
p[np.arange(n), labels] is the code for a subscript that depends on another subscript. Wherever a paper writes an index inside an index, expect fancy indexing or a gather in the implementation.
Shapes are the cheapest possible check. Before running anything, predict the shape of each intermediate result. A shape that surprises you means the decoding is wrong, and you learn it in a second rather than after an hour of training. See tensor shapes and broadcasting.
Common mistakes
Assuming a superscript is an index. Powers, transposes, layer numbers and sample indices all appear as superscripts. Find the sentence in the paper that says which, and if there is none, test both readings numerically.
Missing an implicit sum. A repeated index in physics-influenced notation, or a bare product of matrices, may hide a summation. If your shapes come out wrong by one axis, look for a sum you did not perform.
Trusting the paper's own consistency. Papers redefine symbols between sections, especially between the method and the appendix. Build a symbol table as you read and note the section each definition came from.
Confusing a random variable with one of its values. Capital letters often denote the random quantity, lowercase a particular value. Mixing them up turns a statement about distributions into a statement about one data point.
Try it yourself
Take the definition of any metric you use daily — precision, recall, or mean squared error — and write it out from the symbols, then check it against the scikit-learn implementation to six decimals. Then break it on purpose: swap axis=1 for axis=0 in the softmax line above, and see how plausible the wrong number looks.
What to learn next
- Turning an equation into tensor code — the next step once the symbols are decoded.
- Tensor shapes and broadcasting — the check that catches most decoding errors.
- einsum in PyTorch — writing index notation directly as code.
Researcher — Mathematics and papers.
Where notation is genuinely ambiguous
A symbol table is not enough, because several conventions are contested across subfields. The ones that cause real bugs:
- Row versus column vectors. Optimisation and statistics papers usually treat $x$ as a column vector and write $w^\top x$; most deep-learning code stores examples as rows and computes
X @ W. Every transpose in a paper needs re-derivation against your memory layout, and a shape check is the only reliable arbiter. - Gradient layout. $\partial L / \partial W$ may be laid out as the Jacobian or its transpose. The convention is rarely stated, and the two differ by a transpose that a square weight matrix will hide until the first non-square layer.
- Indexing origin. Mathematical notation is one-based, code is zero-based. Adam's $t$ starts at 1 and appears in $\beta_1^t$; a zero-based loop makes the first bias correction divide by zero.
- Expectation subscripts. $\mathbb{E}{x \sim \mathcal{D}}$, $\mathbb{E}{q_\phi(z \mid x)}$ and a bare $\mathbb{E}$ can appear in the same derivation over different distributions. In variational objectives, which distribution the expectation is taken under determines whether a gradient estimator is even valid.
- Overloaded $p$. The same letter serves as a density, a mass function, and a model's predicted probability, frequently within one equation.
- Norm subscripts. $\lVert \cdot \rVert$ without a subscript means the 2-norm in most ML papers and the Frobenius norm when applied to a matrix, but not universally.
Reading tensor-shaped notation
Modern papers describe operations over tensors with several free indices, where the written form is often less clear than the code. Two habits help.
Annotate every symbol with its shape in the margin, using named dimensions rather than letters: (batch, heads, seq, dim). Shape suffixes carried into your implementation catch the majority of decoding errors before a single value is computed.
Rewrite index-heavy expressions in Einstein summation. An expression with explicit sums over repeated indices maps directly onto np.einsum or torch.einsum, and the string itself becomes a machine-checkable statement of your reading. See einsum in PyTorch. The translation forces you to commit to which indices are summed and which are free, which is exactly the ambiguity you are trying to resolve.
Building a durable symbol table
For any paper you intend to implement, keep a table with four columns: symbol, meaning, shape, and the equation number where it was defined. Two things make it worth the effort. Symbols that never appear in the table were never defined, which is a finding about the paper rather than about you. And symbols with two rows are overloaded, which tells you exactly where to expect a bug.
The same table becomes the variable-naming scheme for the implementation, so that a reviewer can hold the paper beside the code and match them line by line. That correspondence is the subject of the next lesson, and it is worth far more than idiomatic naming.
When the notation is wrong
Published equations contain errors: sign slips, missing normalising constants, indices off by one. The reliable detectors are dimensional analysis (does every term have the same units and shape?), limiting cases (does the expression reduce to something known when a term goes to zero?), and numerical comparison against the authors' released code. When the text and the code disagree, the code is what produced the reported numbers.
What to learn next
- Turning an equation into tensor code — the next step once the symbols are decoded.
- Tensor shapes and broadcasting — the check that catches most decoding errors.
- einsum in PyTorch — writing index notation directly as code.