Build a handwritten digit recogniser

Formulas here are written in plain text, since the site renders no maths typesetting library.

The classification head#

For K classes, the network outputs a vector of logits z in R^K, converted to a distribution by softmax:

softmax(z)_k  =  exp(z_k - max_j z_j)  /  sum_i exp(z_i - max_j z_j)

The subtraction of max_j z_j is mathematically a no-op and numerically essential: without it, exp overflows for logits above roughly 709 in float64. Every production implementation does this.

Training minimises categorical cross-entropy over N examples:

L  =  -(1/N) * sum_n  sum_k  y_nk * log( p_nk )

• y_nk is 1 if example n has true class k, else 0 (one-hot).
• p_nk is the softmax probability the model assigns example n to class k.

With one-hot targets this collapses to -(1/N) * sum_n log p_n,c(n), the negative log-likelihood of the correct class. The gradient with respect to the logits is exactly p - y, which is why softmax and cross-entropy are always implemented as a fused operation.

Why an MLP is the wrong prior for images#

The model above flattens an 8x8 grid into a 64-vector. That discards the spatial arrangement entirely: permute the 64 pixels with any fixed permutation, retrain, and you get the same accuracy. The network never knew the pixels were adjacent.

Convolutional networks (LeCun et al., 1989; LeCun et al., 1998) impose two structural priors:

• Translation equivariance. A convolution commutes with translation, so a feature detector learned at one location applies at every location. An MLP must learn each location separately.
• Locality and weight sharing. A k x k kernel over C_in input channels and C_out output channels has k^2 * C_in * C_out + C_out parameters regardless of image size. A dense layer over a H x W image scales with H * W.

Concretely, for 28x28 MNIST: a dense layer to 128 units costs 784 * 128 + 128 = 100,480 parameters. A 3x3 convolution to 32 channels costs 9 * 1 * 32 + 32 = 320 parameters and produces a 32-channel feature map. Two orders of magnitude fewer parameters, and better accuracy.

Note that convolution gives equivariance, not invariance. Pooling and augmentation supply approximate invariance. This distinction is the subject of Cohen and Welling (2016) on group-equivariant networks.

Dataset provenance and its limits#

load_digits is the Optical Recognition of Handwritten Digits set (Alpaydin and Kaynak, 1998), 1,797 samples in scikit-learn's copy. Preprocessing: 32x32 binary bitmaps, partitioned into non-overlapping 4x4 blocks, with the count of on-pixels per block forming an 8x8 integer matrix in [0, 16]. This block-count step is itself a hand-designed dimensionality reduction, and part of why simple classifiers do so well.

MNIST (LeCun, Cortes and Burges, 1998) is derived from NIST Special Database 3 and 1, size-normalised to a 20x20 box and centred by centre of mass in a 28x28 field. State of the art sits around 99.8% test accuracy, which means the benchmark carries roughly 20 informative examples. It is saturated and should not be used to compare modern methods.

Recht et al. (2018, 2019) constructed fresh test sets for CIFAR-10 and ImageNet following the original collection protocols, and observed accuracy drops of 3 to 15 points across a wide range of models, with the drop increasing for higher-scoring models. The ranking of models was largely preserved, so this is distribution shift from protocol details rather than pure adaptive overfitting — but it bounds how much any absolute number on a reused test set should be trusted.

Calibration#

The 77%-confident error in the developer block is not an anomaly. Guo et al. (2017), On Calibration of Modern Neural Networks, show that deeper and wider networks are systematically overconfident, and that this worsened as architectures grew, even as accuracy improved.

Expected calibration error partitions predictions into M confidence bins and measures the gap between confidence and accuracy:

ECE  =  sum_{m=1..M}  ( |B_m| / n ) * | acc(B_m) - conf(B_m) |

B_m is the set of predictions whose confidence falls in bin m, acc(B_m) their empirical accuracy, conf(B_m) their mean predicted confidence.

Temperature scaling — dividing logits by a single scalar T fitted on a validation set by minimising NLL — repairs most of the gap at essentially zero cost and without changing the argmax, so accuracy is unaffected. It should be the default final step for any classifier whose probabilities are consumed downstream.

For a reject option, the decision-theoretic result is Chow's rule (Chow, 1970): under a known cost d for rejection and 0/1 loss for errors, the optimal policy rejects when max_k p(k|x) < 1 - d. This is optimal only for calibrated posteriors, which closes the loop back to why calibration matters.

Cost#

For an MLP with layer widths [d_0, ..., d_L] and batch size B:

• Parameters: sum_l ( d_l * d_{l-1} + d_l ).
• Forward FLOPs: about 2 * B * sum_l d_l * d_{l-1}.
• Backward: roughly twice the forward cost.

For the (64, 64, 10) network above: 64*64 + 64 = 4,160 plus 64*10 + 10 = 650, giving 4,810 parameters. That is small enough to run on a microcontroller, which is precisely why digit recognition was deployable in the 1990s on hardware weaker than a modern washing machine.

For a convolution producing H_out x W_out x C_out from C_in channels with a k x k kernel, FLOPs are about 2 * H_out * W_out * C_out * C_in * k^2 per image. Compute scales with spatial size while parameters do not, which is the opposite of the dense case and drives entirely different engineering trade-offs.

What production handwriting recognition looks like#

Single-digit classification is a solved subproblem, not the product. A cheque or address reader is a pipeline:

1. Detection and layout analysis — locate text regions, correct skew and perspective.
2. Binarisation and normalisation — adaptive thresholding (Sauvola or Niblack) under uneven lighting.
3. Sequence recognition — not per-character classification. Segmentation-free CTC (Graves et al., 2006) or attention-based encoder-decoders read a whole line, because character boundaries in cursive handwriting are ill-defined. This "Sayre's paradox" — you cannot segment without recognising, nor recognise without segmenting — is why per-character pipelines were abandoned.
4. Language modelling — a lexicon or n-gram prior over valid PIN codes, amounts or names, which corrects a large fraction of visual errors.
5. Rejection and human routing, with cost-sensitive thresholds set per field.

The recognition network is often the least troublesome component. Steps 1, 2 and 4 usually determine whether the system works.

Papers#

• LeCun, Boser, Denker et al., Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation, 1989.
• LeCun, Bottou, Bengio and Haffner, Gradient-Based Learning Applied to Document Recognition, Proc. IEEE, 1998 — introduces LeNet-5 and MNIST.
• Alpaydin and Kaynak, Optical Recognition of Handwritten Digits, UCI Machine Learning Repository, 1998 — the dataset used here.
• Chow, On Optimum Recognition Error and Reject Tradeoff, IEEE Trans. Information Theory, 1970.
• Graves, Fernández, Gomez and Schmidhuber, Connectionist Temporal Classification, ICML 2006.
• Cohen and Welling, Group Equivariant Convolutional Networks, ICML 2016 — arxiv.org/abs/1602.07576
• Guo, Pleiss, Sun and Weinberger, On Calibration of Modern Neural Networks, ICML 2017 — arxiv.org/abs/1706.04599
• Recht, Roelofs, Schmidt and Shankar, Do ImageNet Classifiers Generalize to ImageNet?, ICML 2019 — arxiv.org/abs/1902.10811

What to learn next#

• Convolutional neural networks — the architecture built for images, and the next version of this project.
• Image classification — the same task on photographs instead of 8x8 grids.
• Model evaluation — confusion matrices, precision and recall in depth.