3D human pose and SMPL

The SMPL function#

Loper, Mahmood, Romero, Pons-Moll and Black (SIGGRAPH Asia 2015), SMPL: A Skinned Multi-Person Linear Model (files.is.tue.mpg.de/black/papers/SMPL2015.pdf), define

$$ M(\beta, \theta) = W\big(T_P(\beta, \theta),\; J(\beta),\; \theta,\; \mathcal{W}\big) $$

$$ T_P(\beta, \theta) = \bar{T} + B_S(\beta) + B_P(\theta) $$

• $\bar{T} \in \mathbb{R}^{3N}$ — the mean template, $N = 6890$ vertices.
• $\beta \in \mathbb{R}^{10}$ — shape coefficients; $B_S(\beta) = \sum_n \beta_n S_n$ with $S_n$ the principal components of registered body scans in the rest pose.
• $\theta \in \mathbb{R}^{3K+3}$ — axis-angle rotations for $K = 23$ joints plus a global orientation.
• $B_P(\theta) = \sum_n \big(R_n(\theta) - R_n(\theta^)\big) P_n$ — pose corrective blend shapes, linear in the elements of the joint rotation matrices relative to the rest pose $\theta^$. This linearity is the paper's central design choice: it keeps the model differentiable and cheap while capturing pose-dependent deformation.
• $J(\beta) = \mathcal{J}\,(\bar{T} + B_S(\beta))$ — a learned sparse regressor mapping the shaped surface to joint centres. Joints move with shape.
• $\mathcal{W} \in \mathbb{R}^{N \times K}$ — skinning weights, rows summing to one.

Linear blend skinning is then

$$ \bar{t}'i = \sum{k=1}^{K} w_{k,i}\, G'_k(\theta, J)\; \bar{t}_i $$

$G'_k$ is the world transform of joint $k$ with the rest-pose transform removed. The "candy wrapper" collapse is the failure mode of this weighted average of rigid transforms, and $B_P$ is the learned correction for it.

SMPL-X (Pavlakos et al., CVPR 2019, arxiv.org/abs/1904.05866) extends this to $N = 10475$ vertices and $K = 54$ joints, adding articulated hands from MANO and an expressive face from FLAME, with shape parameters trained jointly across body, face and hands.

Why monocular 3D pose is ill-posed#

A perspective camera maps $\mathbb{R}^3 \to \mathbb{R}^2$, so depth is unrecoverable without priors. Two specific ambiguities dominate:

Depth reflection. For a limb whose 2D projection is known, two 3D configurations — toward and away from the camera — project identically. With $J$ joints this generates up to $2^{J}$ consistent skeletons, pruned only by kinematic and anthropometric constraints.

Scale–depth entanglement. Under weak perspective, a body of size $s$ at depth $d$ is indistinguishable from one of size $\alpha s$ at depth $\alpha d$. Absolute metric scale is not identifiable from a single image without a known focal length or an object of known size.

A parametric body model resolves the first by construction, since $\beta$ and $\theta$ cannot express a non-anatomical body, and constrains the second by supplying a population prior over body size.

Method lineage#

Optimisation. SMPLify (Bogo et al., ECCV 2016, arxiv.org/abs/1607.08128) detects 2D joints, then minimises reprojection error over $(\beta, \theta)$ with a pose prior fitted as a Gaussian mixture over MoCap data and an interpenetration penalty. Accurate, slow, and dependent on initialisation.

Direct regression. HMR (Kanazawa et al., CVPR 2018, arxiv.org/abs/1712.06584) regresses $(\beta, \theta, \text{camera})$ from image features with an iterative error-feedback loop, supervised by 2D reprojection loss plus an adversarial prior discriminating real from implausible poses. The adversarial prior removed the need for paired 3D annotations, which is what made in-the-wild training viable.

Hybrid. SPIN (Kolotouros et al., ICCV 2019) runs SMPLify inside the training loop, using the optimiser's output as pseudo-ground-truth for the regressor, which in turn initialises the optimiser. Each component fixes the other's weakness.

Transformers. HMR 2.0 / 4D-Humans (Goel, Pavlakos, Rajasegaran, Kanazawa and Malik, ICCV 2023, arxiv.org/abs/2305.20091) replaces the convolutional backbone and iterative head with a plain ViT encoder and transformer decoder, reporting substantially improved robustness on unusual poses, and feeds the per-frame reconstructions into a 3D tracker that maintains identity through occlusion.

Probabilistic output. Because the problem is genuinely ambiguous, several methods output a distribution rather than a point estimate — ProHMR (Kolotouros et al., ICCV 2021) with normalising flows, and diffusion-based samplers more recently. This is the intellectually correct framing, and it is under-used in applications that want one answer.

Evaluation#

• MPJPE — mean per-joint position error in millimetres, after aligning the root joint.
• PA-MPJPE — the same after a full Procrustes alignment, removing global rotation, translation and scale. It measures pose quality only and is the fairer comparison across methods with different camera handling.
• PVE / MPVPE — per-vertex error, which additionally measures shape quality.
• Datasets — Human3.6M (indoor, MoCap), 3DPW (in the wild, IMU-derived ground truth), EMDB and AGORA (synthetic with exact ground truth).

Two cautions. Human3.6M is indoor, small-population and largely saturated; report 3DPW. And PA-MPJPE flatters methods with poor global orientation, so quote both.

Licensing#

SMPL and SMPL-X model files are distributed by the Max Planck Institute under a non-commercial licence, requiring registration and granting use for non-commercial scientific research, non-commercial education and non-commercial artistic projects. Commercial use requires a separate agreement, and this applies to the model parameters themselves, not only to the code.

The practical implication for a product team: the model file is a dependency with a licence, exactly like a proprietary library. Resolve it before it is embedded in a codebase. Open alternatives exist and differ in quality and ecosystem support; evaluate them early rather than late.

Papers#

• Loper et al., SMPL: A Skinned Multi-Person Linear Model, SIGGRAPH Asia 2015
• Bogo et al., Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image, ECCV 2016 — arxiv.org/abs/1607.08128
• Kanazawa et al., End-to-end Recovery of Human Shape and Pose, CVPR 2018 — arxiv.org/abs/1712.06584
• Pavlakos et al., Expressive Body Capture: 3D Hands, Face, and Body from a Single Image (SMPL-X), CVPR 2019 — arxiv.org/abs/1904.05866
• Kolotouros et al., Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop (SPIN), ICCV 2019 — arxiv.org/abs/1909.12828
• Kolotouros et al., Probabilistic Modeling for Human Mesh Recovery, ICCV 2021 — arxiv.org/abs/2108.11944
• Goel et al., Humans in 4D: Reconstructing and Tracking Humans with Transformers, ICCV 2023 — arxiv.org/abs/2305.20091

What to learn next#

• Vision transformers — the backbone behind the current best mesh-recovery models.
• Autoencoders — learning a compact code for a complicated object, which is what the shape parameters are.
• Linear algebra — rotations, blend weights and the principal components behind the shape space.