Build a face recognition system

Formulas here are written in plain text, since the site renders no maths typesetting library.

Eigenfaces#

Let X be an n x d matrix of n vectorised face images with d = 4096 pixels, and mu the mean face. Centre the data as Xc = X - mu. PCA seeks the orthonormal basis U_k of k directions maximising retained variance, given by the top-k eigenvectors of the sample covariance:

S  =  (1 / (n - 1)) * Xc^T Xc          (d x d)
S u_i  =  lambda_i u_i,     lambda_1 >= lambda_2 >= ...

• u_i is the i-th eigenvector, reshaped to 64x64 for display as an eigenface.
• lambda_i is the variance captured along u_i; explained_variance_ratio_[i] = lambda_i / sum_j lambda_j.

A face is then represented by its k coordinates z = U_k^T (x - mu), and reconstructed as x_hat = mu + U_k z.

The Turk-Pentland trick. When d >> n, forming the d x d matrix S is wasteful and its rank is at most n - 1. Instead compute eigenvectors v_i of the n x n Gram matrix Xc Xc^T; then u_i = Xc^T v_i / || Xc^T v_i ||. For our case that is a 300x300 eigenproblem rather than 4096x4096. scikit-learn achieves the same effect via randomized SVD (Halko, Martinsson and Tropp, 2011) at O(n d k).

Whitening. whiten=True rescales each coordinate to unit variance, z_i / sqrt(lambda_i). Without it the RBF kernel's isotropic distance is dominated by the first few high-variance components — which encode illumination, not identity. This single flag is often the difference between a working and a broken eigenface pipeline.

Why PCA is the wrong objective, and Fisherfaces#

PCA maximises total variance, which is unsupervised. It does not know about identity. Belhumeur, Hespanha and Kriegman (1997) observed that for face data the largest variance directions correspond to lighting, so PCA preferentially retains exactly the nuisance factor.

Fisher's linear discriminant instead maximises the ratio of between-class to within-class scatter:

W*  =  argmax   | W^T S_B W |  /  | W^T S_W W |
          W

S_B  =  sum_c  n_c (mu_c - mu)(mu_c - mu)^T
S_W  =  sum_c  sum_{x in c}  (x - mu_c)(x - mu_c)^T

mu_c and n_c are the mean and count of class c. S_W is singular when n < d, so the standard construction is Fisherfaces: project with PCA to n - C dimensions first, then apply LDA. LDA yields at most C - 1 discriminant directions, which is a hard ceiling.

The modern formulation: metric learning#

Contemporary systems do not classify. They learn an embedding f: image -> R^128 (or 512) such that Euclidean or cosine distance encodes identity, then compare against stored templates. This is what makes enrolment cheap: adding a person stores one vector and retrains nothing.

FaceNet (Schroff, Kalenichenko and Philbin, 2015) trains directly on the triplet loss:

L  =  sum over triplets  max( 0,  || f(a) - f(p) ||^2  -  || f(a) - f(n) ||^2  +  alpha )

a is an anchor image, p a different image of the same person, n an image of a different person, and alpha the required margin. Embeddings are L2-normalised to the unit hypersphere. The practical difficulty is triplet mining: most random triplets already satisfy the constraint and produce zero gradient, so semi-hard negative mining within each batch is essential.

ArcFace (Deng, Guo, Xue and Zafeiriou, 2019) replaced triplets with a margin applied inside the softmax, which removed the mining problem:

L  =  -(1/N) sum_i  log  [  e^{s * cos(theta_yi + m)}
                           / ( e^{s * cos(theta_yi + m)}  +  sum_{j != yi} e^{s * cos(theta_j)} ) ]

• theta_j is the angle between the normalised embedding and the normalised class-j weight vector.
• m is an additive angular margin, typically 0.5 radians.
• s is a scale factor, typically 64, compensating for the bounded range of cosine.

The additive angular margin enforces a constant geodesic separation on the hypersphere, which is geometrically cleaner than the multiplicative margin of SphereFace or the cosine margin of CosFace. ArcFace and its descendants remain the standard backbone objective.

Detection followed a parallel path: Viola and Jones (2001) with Haar features and AdaBoost cascades, then MTCNN (Zhang et al., 2016) for joint detection and landmark alignment, then single-shot detectors such as RetinaFace (Deng et al., 2019).

Open-set recognition, and why the metric matters#

The developer block demonstrates the closed-set assumption failing. Formally, closed-set identification assumes the probe identity is in the gallery, so argmax over gallery scores is well-defined. Open-set identification does not, and requires a reject region.

Report TAR at a fixed FAR — true accept rate at a false accept rate, typically FAR = 1e-4 or 1e-6 — never accuracy. Accuracy at a single operating point is uninterpretable because it entangles the threshold with the class balance.

For watchlist deployments the base rate dominates. With a false accept rate of 1e-5 and 100,000 people scanned per day, expect roughly one false alarm per day even if the watchlist is empty. Scale to a busy transport hub and the alarms overwhelm the true detections. This is the false positive paradox, and no improvement in FAR of the magnitude achievable today removes it — only reducing the scanned population does.

Demographic differentials#

Buolamwini and Gebru (2018), Gender Shades, audited three commercial gender-classification APIs and found error rates up to 34.7% for darker-skinned women against 0.8% for lighter-skinned men. They also introduced the Pilot Parliaments Benchmark, constructed for balance across skin type and gender, because existing benchmarks were not.

Grother, Ngan and Hanaoka (2019), NIST FRVT Part 3: Demographic Effects (NISTIR 8280), evaluated 189 algorithms from 99 developers and confirmed the pattern at scale. Their key findings: false positive differentials across demographic groups spanned factors of 10 to 100 in many algorithms; the direction and magnitude varied with the training data's origin; and false positives were far more demographically variable than false negatives. Since false positives are the errors that cause wrongful accusations, that asymmetry is exactly the wrong way round.

The mechanism is not mysterious — unrepresentative training distributions, plus image pipelines (exposure, contrast) tuned on lighter skin. It is measurable, and it is fixable in direction if not fully in magnitude. The obligation is therefore to measure and publish per-group rates rather than an aggregate.

Security and governance#

Presentation attacks. A printed photo, a screen replay or a silicone mask defeats a matcher that has no liveness check. ISO/IEC 30107-3 defines the evaluation framework. Report attack presentation classification error rate alongside matching accuracy, or the accuracy figure is not meaningful for access control.

Template protection. Embeddings are not anonymous. Model inversion can reconstruct a recognisable face from an embedding (Mai et al., 2018, on face reconstruction from deep templates). Treat templates as biometric data with the same protections as the images, and prefer cancellable-biometric schemes that allow a compromised template to be revoked and reissued.

Legal position. Face data is special-category personal data under GDPR Article 9, requiring an explicit lawful basis. Illinois BIPA requires written consent before collection and has produced substantial settlements. The EU AI Act classifies remote biometric identification as high-risk with significant restrictions on real-time use in public spaces. India's Digital Personal Data Protection Act, 2023 governs processing of personal data including biometrics, with consent and purpose-limitation obligations. Determine the applicable regime before collecting a single face.

Papers#

• Sirovich and Kirby, Low-Dimensional Procedure for the Characterization of Human Faces, JOSA A, 1987.
• Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience, 1991.
• Belhumeur, Hespanha and Kriegman, Eigenfaces vs. Fisherfaces, IEEE TPAMI, 1997.
• Viola and Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, CVPR 2001.
• Huang, Ramesh, Berg and Learned-Miller, Labeled Faces in the Wild, UMass TR 07-49, 2007.
• Taigman, Yang, Ranzato and Wolf, DeepFace, CVPR 2014.
• Schroff, Kalenichenko and Philbin, FaceNet, CVPR 2015 — arxiv.org/abs/1503.03832
• Zhang, Zhang, Li and Qiao, Joint Face Detection and Alignment using MTCNN, IEEE SPL, 2016 — arxiv.org/abs/1604.02878
• Buolamwini and Gebru, Gender Shades, FAccT 2018.
• Deng, Guo, Xue and Zafeiriou, ArcFace, CVPR 2019 — arxiv.org/abs/1801.07698
• Grother, Ngan and Hanaoka, Face Recognition Vendor Test Part 3: Demographic Effects, NISTIR 8280, 2019.

What to learn next#

• OpenCV — the image handling this project rests on.
• Convolutional neural networks — the architecture behind every modern face embedding.
• Image classification — the general vision task, without the identity questions.