What is a neural network?

Formal definition#

A feedforward neural network (a multilayer perceptron, MLP) is a parameterised function built by composing affine maps with pointwise non-linearities:

$$ h^{(0)} = x, \qquad h^{(l)} = \phi^{(l)}!\left(W^{(l)} h^{(l-1)} + b^{(l)}\right), \qquad f_\theta(x) = h^{(L)} $$

Where:

• $x \in \mathbb{R}^{d_0}$ — the input vector, with $d_0$ features.
• $L$ — the number of layers; $l \in {1, \dots, L}$ indexes them.
• $W^{(l)} \in \mathbb{R}^{d_l \times d_{l-1}}$ — the weight matrix of layer $l$.
• $b^{(l)} \in \mathbb{R}^{d_l}$ — the bias vector of layer $l$.
• $\phi^{(l)}$ — a non-linear function applied elementwise (ReLU, tanh, GELU).
• $h^{(l)} \in \mathbb{R}^{d_l}$ — the activations, or hidden representation, at layer $l$.
• $\theta = {W^{(l)}, b^{(l)}}_{l=1}^{L}$ — every trainable parameter, collected.

The non-linearity is what makes the composition non-trivial. If every $\phi^{(l)}$ were the identity, then $f_\theta(x) = W^{(L)} \cdots W^{(1)} x + \tilde{b}$, which is a single affine map of rank at most $\min_l d_l$. Depth would buy nothing but a rank constraint.

Cost#

For a layer mapping $d_{l-1} \to d_l$ with batch size $B$:

• Parameters: $d_l d_{l-1} + d_l$.
• Forward FLOPs: $\approx 2 B d_l d_{l-1}$ (one multiply and one add per weight per sample).
• Backward FLOPs: $\approx 2 \times$ the forward cost, since gradients flow to both inputs and weights.

The rule of thumb for training a dense model is six FLOPs per parameter per token, decomposed as two forward and four backward.

Activation memory, not parameter memory, is usually the binding constraint during training: storing $h^{(l)}$ for the backward pass costs $O(B \sum_l d_l)$.

Universal approximation, stated honestly#

Cybenko (1989), Approximation by superpositions of a sigmoidal function, and Hornik (1991), Approximation capabilities of multilayer feedforward networks, establish that a network with one hidden layer and a non-polynomial activation is dense in $C(K)$ for compact $K \subset \mathbb{R}^n$, under the sup norm.

Three caveats matter, and they are routinely dropped when this theorem is quoted:

1. It is an existence result. It asserts that suitable weights exist; it says nothing about gradient descent finding them.
2. The required width can be exponential in the input dimension. Existence at any cost is a weak guarantee.
3. It says nothing about generalisation — approximating a function on the training set is not the same as approximating the data-generating distribution.

Depth genuinely helps, and there are separation theorems that prove it. Telgarsky (2016), Benefits of depth in neural networks, exhibits functions computable by a network of depth $k$ that require width exponential in $k$ at depth $O(1)$. Eldan and Shamir (2016) give a $3$-layer versus $2$-layer separation.

Parameter counting in practice#

For an MLP with layer widths $[d_0, d_1, \dots, d_L]$, total parameters are $\sum_{l=1}^{L} (d_l d_{l-1} + d_l)$.

A concrete case: widths $[784, 256, 128, 10]$ gives $784 \cdot 256 + 256 = 200{,}960$, plus $256 \cdot 128 + 128 = 32{,}896$, plus $128 \cdot 10 + 10 = 1{,}290$ — a total of $235{,}146$ parameters. Note that the first layer holds 85% of them, which is why input dimensionality dominates small dense models.

The biological analogy is weak#

The "neuron" name is historical. McCulloch and Pitts (1943) proposed a threshold logic unit as a model of neural activity, and Rosenblatt (1958) built the perceptron on that framing.

Modern artificial neurons diverge sharply from biology: real neurons communicate in discrete spikes with timing-dependent plasticity, there is no known biological mechanism implementing symmetric weight transport for backpropagation (the "weight transport problem", raised by Grossberg, 1987), and cortical connectivity is nothing like dense all-to-all layers. Treat the name as a label, not as evidence.

Where to read next#

• Rumelhart, Hinton and Williams (1986), Learning representations by back-propagating errors — the paper that made depth trainable.
• Goodfellow, Bengio and Courville, Deep Learning (2016), chapter 6 — the standard reference treatment of feedforward networks.
• Zhang et al. (2017), Understanding deep learning requires rethinking generalization — networks fit random labels perfectly, so classical capacity arguments do not explain why they generalise.

What to learn next#

• How neural networks learn — where the weights actually come from.
• Activation functions — why that bend between layers is not optional.
• Gradient descent — the downhill walk that does the tuning.