Why AI needs mathematics

The problem, stated once#

Supervised learning is empirical risk minimisation. Samples are drawn independently from an unknown joint distribution over inputs and labels. Choose the parameters that minimise average loss on those samples:

R_emp(theta)  =  (1/n) * sum_{i=1..n}  L( f(x_i; theta),  y_i )

• n is the number of training examples.
• x_i is the i-th input, y_i its true label.
• theta is the full parameter vector of the model.
• f(x; theta) is the model's prediction for input x.
• L(a, b) is the loss, a non-negative function measuring how bad prediction a is against truth b.

The quantity actually wanted is the true risk, the expected loss over the whole distribution D:

R(theta)  =  E_{(x,y) ~ D} [ L( f(x; theta), y ) ]

Everything difficult in machine learning lives in the gap R(theta) - R_emp(theta). Statistics is the study of that gap.

Where each branch enters#

Linear algebra supplies the model class and the cost model. A dense layer maps a batch X of shape (B, d_in) to

Z  =  X @ W + b          W has shape (d_in, d_out),  b has shape (d_out,)

with a forward cost of about 2 * B * d_in * d_out floating-point operations, counting a multiply-accumulate as two. Backward is roughly twice forward, so a training step costs about three times a forward pass. This is the arithmetic behind every FLOP budget you will read in a model card.

Calculus supplies the descent direction. Gradient descent is:

theta_{t+1}  =  theta_t  -  eta * grad_theta R_emp(theta_t)

• eta is the learning rate, a positive scalar.
• grad_theta R_emp is the vector of partial derivatives of the empirical risk with respect to every entry of theta.

Take an L-smooth convex objective, meaning its gradient does not change faster than a constant L. The classical condition for monotone decrease is then eta < 2/L. The divergence shown in the Developer block is exactly this bound being violated.

Reverse-mode automatic differentiation computes the full gradient within a small constant factor of one forward pass. That single result is what makes deep learning computationally possible.

Probability supplies the loss functions, which are not arbitrary. Maximum likelihood estimation chooses

theta_MLE  =  argmax_theta  sum_{i=1..n}  log p( y_i | x_i ; theta )
           =  argmin_theta  -(1/n) * sum_{i=1..n}  log p( y_i | x_i ; theta )

where p(y | x; theta) is the model's assumed conditional density or mass function. Two familiar losses fall straight out:

• Assume y | x is Gaussian with mean f(x; theta) and fixed variance. The negative log-likelihood reduces to mean squared error plus a constant.
• Assume y | x is categorical with class probabilities from a softmax. The negative log-likelihood is cross-entropy.

So "use MSE for regression, cross-entropy for classification" is not a convention. It is a statement about which noise model you are assuming.

Bayes' theorem gives the machinery for updating belief with evidence:

P(A | B)  =  P(B | A) * P(A) / P(B)

• P(A) is the prior, belief before seeing B.
• P(B | A) is the likelihood of the evidence under A.
• P(A | B) is the posterior, belief after seeing B.

This underlies naive Bayes, Bayesian optimisation for hyperparameter search, Thompson sampling, and variational inference.

Statistics supplies the guarantee that any of this transfers. With a hypothesis class of finite VC dimension d and n samples, uniform convergence gives a bound of the form

R(theta)  <=  R_emp(theta)  +  O( sqrt( (d + log(1/delta)) / n ) )

holding with probability at least 1 - delta. The bound is loose for modern overparameterised networks. Explaining why they generalise anyway remains an open research area; see Zhang et al., 2017.

The qualitative lesson survives. Generalisation improves with more data and degrades with more model capacity. And any claim about a model needs a confidence interval, not a point estimate.

The minimum you should be fluent in#

References#

• Deisenroth, M. P., Faisal, A. A., Ong, C. S. Mathematics for Machine Learning. Cambridge University Press, 2020. Freely available from the authors, and the best single starting point.
• Goodfellow, I., Bengio, Y., Courville, A. Deep Learning. MIT Press, 2016. Chapters 2 to 4 are a compact survey of exactly this material.
• Shalev-Shwartz, S., Ben-David, S. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. The generalisation bounds, done properly.
• Boyd, S., Vandenberghe, L. Convex Optimization. Cambridge University Press, 2004.
• Robbins, H., Monro, S. "A Stochastic Approximation Method." Annals of Mathematical Statistics 22(3), 400–407, 1951. The origin of stochastic gradient descent.
• Rumelhart, D. E., Hinton, G. E., Williams, R. J. "Learning representations by back-propagating errors." Nature 323, 533–536, 1986.
• Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O. "Understanding deep learning requires rethinking generalization." ICLR, 2017.

What to learn next#

• Vectors and matrices — the objects and their costs.
• Derivatives and gradients — reverse-mode differentiation in detail.
• Optimization — momentum, Adam, and convergence conditions.