Logistic regression

The model#

Logistic regression is the generalised linear model for a Bernoulli response with the canonical logit link.

p(x)      = P(y = 1 | x) = sigma( w^T x + b )
sigma(z)  = 1 / (1 + exp(-z))
logit(p)  = log( p / (1 - p) ) = w^T x + b

• x — feature vector in R^d
• w — weight vector in R^d, b — intercept scalar
• z = w^T x + b — the linear predictor, often called the logit or the raw score
• sigma — the logistic (sigmoid) function, mapping R onto the open interval (0, 1)
• p / (1 - p) — the odds of the positive class

The logit link is what makes the model linear in a space where linearity is defensible. Probability is bounded and its effects saturate; log-odds is unbounded and additive. exp(w_j) is therefore an odds ratio: the multiplicative change in odds per unit change in feature j, holding the rest fixed.

Useful identities:

sigma(-z)      = 1 - sigma(z)
sigma'(z)      = sigma(z) * (1 - sigma(z))
d/dz log sigma(z) = 1 - sigma(z)

The decision boundary {x : w^T x + b = 0} is a hyperplane. Logistic regression is a linear classifier; the non-linearity acts on the output, never on the boundary.

Estimation#

There is no closed form. Unlike ordinary least squares, the score equations are non-linear in w.

Maximum likelihood minimises the negative log-likelihood, which is the binary cross-entropy or log loss:

L(w, b) = - SUM_i [ y_i * log p_i  +  (1 - y_i) * log (1 - p_i) ]

• p_i = sigma(w^T x_i + b) — predicted probability for observation i
• y_i in {0, 1} — the observed label

The gradient has a form worth memorising, because it is identical to the linear regression gradient with p_i in place of the fitted value:

grad_w L = SUM_i ( p_i - y_i ) * x_i  =  X^T (p - y)
Hessian  H = X^T S X,     S = diag( p_i * (1 - p_i) )

Since p_i(1 - p_i) > 0 strictly, H is positive semi-definite, so L is convex. Any local minimum is global. With an L2 penalty added, L becomes strictly convex and the minimiser is unique.

Convexity is the reason logistic regression is trustworthy in a way neural networks are not. There is one answer, and every solver finds the same one.

Newton's method on this objective is iteratively reweighted least squares:

w_{t+1} = w_t + (X^T S X)^{-1} X^T (y - p)

Each Newton step is a weighted least-squares solve with weights S. This is the classical statistical algorithm, and it is what statsmodels and R's glm use.

Cost and solver choice#

IRLS / Newton   O(n d^2 + d^3) per iteration    quadratic convergence, few iterations
L-BFGS          O(n d) per iteration            limited-memory quasi-Newton, default in sklearn
SAGA            O(d) per sample-step            variance-reduced SGD, handles L1 and huge n
liblinear       coordinate descent              strong on small n with L1

Newton is superb for d in the hundreds and hopeless past a few thousand, because of the d^3 solve. scikit-learn defaults to lbfgs, which never forms the Hessian.

Note the deviation from statistical convention: scikit-learn penalises by default with C = 1.0, where C = 1 / lambda. Coefficients from sklearn and from statsmodels.Logit will not match unless you set penalty=None.

Separation#

If the classes are linearly separable, the likelihood has no finite maximiser. Pushing ||w|| towards infinity drives L towards zero monotonically.

Albert and Anderson (1984) classified the three regimes — complete separation, quasi-complete separation, and overlap — and showed the MLE exists and is unique only in the third. Standard errors under separation are meaningless — the problem is not that they are large.

Remedies, in rough order of preference:

• Penalisation. Any lambda > 0 restores a finite unique solution. This is why scikit-learn rarely appears to fail on separable data.
• Firth's correction (1993). Penalises the likelihood by the Jeffreys prior, log |I(w)|^{1/2}. It removes the O(1/n) first-order bias and always yields finite estimates. Heinze and Schemper (2002) made it the standard recommendation in biostatistics.
• Weakly informative priors. Gelman et al. (2008) propose independent Cauchy priors with scale 2.5 on standardised inputs.

Separation in observational data is frequently a leakage signal. Check the offending feature before you reach for a correction.

Calibration and the proper scoring property#

Log loss is a strictly proper scoring rule: its expectation is uniquely minimised by reporting the true conditional probability. Fitting by maximum likelihood therefore targets calibration directly, which is why logistic regression is usually better calibrated out of the box than SVMs, naive Bayes, or boosted trees.

A specific consequence of including an intercept: at the optimum the gradient with respect to b vanishes, giving

SUM_i p_i  =  SUM_i y_i

The predicted positive count matches the observed positive count exactly on the training set. This is why Platt scaling (1999) — fitting a one-dimensional logistic regression to another model's scores — is such an effective post-hoc calibrator.

Under case-control or otherwise non-representative sampling, only the intercept is biased; the slopes remain consistent. Prentice and Pyke (1979) established this, and it is why logistic regression is the standard tool in epidemiology. Correct by subtracting log( (pi_1 / pi_0) * ((1 - rho_1) / (1 - rho_0)) ), where pi are sampling fractions and rho are population rates.

Multi-class#

Two standard extensions:

Multinomial (softmax):  P(y = k | x) = exp(z_k) / SUM_j exp(z_j)
One-vs-rest:            K independent binary fits, argmax over their scores

The softmax parameterisation is over-identified — adding a constant to every z_k leaves the probabilities unchanged — so the solution is determined only up to a shift, fixed either by a reference category or by the penalty. One-vs-rest produces scores that are not jointly normalised, so its probabilities require renormalisation and are less trustworthy. Since version 1.5, scikit-learn defaults to multinomial for solver='lbfgs'.

Relationship to other models#

Generative counterpart. Gaussian naive Bayes and linear discriminant analysis assume class-conditional Gaussians with shared covariance, and imply exactly this logistic posterior. Logistic regression estimates that posterior directly without the Gaussian assumption. Ng and Jordan (2001) showed the practical consequence: the generative model has higher asymptotic error but approaches it as O(log d / n) rather than O(d / n), so naive Bayes wins on small n and loses on large n.

Loss-function view. With labels in {-1, +1} the objective is log(1 + exp(-y * z)). Compare hinge loss max(0, 1 - y*z), which the SVM minimises. Log loss is smooth and never exactly zero, so every point keeps contributing a gradient. Hinge loss is exactly zero beyond the margin, which is what makes the SVM solution sparse in its support vectors.

Maximum entropy. Multinomial logistic regression is the maximum-entropy distribution subject to matching empirical feature expectations. The NLP literature of the 1990s and 2000s calls it MaxEnt for this reason; it is the same model.

Single-layer network. A neural network with no hidden layer, a sigmoid output and cross-entropy loss is logistic regression. See what is a neural network.

Key references#

• Berkson, J. (1944). Application of the Logistic Function to Bio-Assay. JASA 39(227). Introduces the term "logit".
• Cox, D. R. (1958). The Regression Analysis of Binary Sequences. JRSS-B 20(2).
• Nelder, J. & Wedderburn, R. (1972). Generalized Linear Models. JRSS-A 135(3). The unifying framework.
• Prentice, R. & Pyke, R. (1979). Logistic Disease Incidence Models and Case-Control Studies. Biometrika 66(3).
• Albert, A. & Anderson, J. (1984). On the Existence of Maximum Likelihood Estimates in Logistic Regression Models. Biometrika 71(1).
• Firth, D. (1993). Bias Reduction of Maximum Likelihood Estimates. Biometrika 80(1).
• Platt, J. (1999). Probabilistic Outputs for Support Vector Machines. Advances in Large Margin Classifiers.
• Ng, A. & Jordan, M. (2001). On Discriminative vs. Generative Classifiers. NeurIPS 14.
• Gelman, A. et al. (2008). A Weakly Informative Default Prior Distribution for Logistic and Other Regression Models. Annals of Applied Statistics 2(4).

Current state#

Logistic regression is not obsolete and is not a teaching toy. It remains the required model wherever a coefficient must be defended: credit scoring under regulatory scrutiny, clinical prediction, epidemiology, and A/B test analysis.

Three properties keep it there. Convexity guarantees a reproducible fit. The proper scoring rule delivers calibrated probabilities. Odds ratios carry a meaning that survives contact with a non-technical audience.

It is also the correct baseline for any tabular binary task. Report it beside your gradient-boosted model. If the gap is a fraction of a point, the simpler model is the better engineering decision.

What to learn next#

• Classification — the wider family, and the shapes of boundary other models can draw.
• Model evaluation — choosing the cut-off, and why accuracy misleads.
• Decision trees — what to reach for when a straight boundary is not enough.