Image matting and background removal

The compositing equation#

Porter and Duff (1984) formalised compositing; matting inverts it. For each pixel $i$ and colour channel:

$$ I_i = \alpha_i F_i + (1 - \alpha_i) B_i, \qquad \alpha_i \in [0, 1] $$

In RGB this is 3 equations in 7 unknowns: $F \in \mathbb{R}^3$, $B \in \mathbb{R}^3$, $\alpha \in \mathbb{R}$. The problem is under-determined by 4 degrees of freedom per pixel, which is the formal version of the first table in the developer section.

Every method supplies the missing constraints:

• Trimap: $\alpha = 1$ and $\alpha = 0$ are given on most pixels; $F$ and $B$ in the unknown band are estimated from nearby known pixels.
• Known background: $B$ is measured, reducing the problem to 3 equations in 4 unknowns.
• Colour-line prior: foreground and background colours lie on a line locally.
• Learned prior: a network trained on ground-truth mattes.

Closed-form matting#

Levin, Lischinski, Weiss (2008) derive the most influential classical result. Assume $F$ and $B$ are locally approximately linear in the image colours within a small window. That is the colour-line model. Then $F$ and $B$ can be eliminated analytically, leaving a quadratic cost in $\alpha$ alone:

$$ J(\alpha) = \alpha^{\top} L \alpha + \lambda (\alpha - \hat{\alpha})^{\top} D (\alpha - \hat{\alpha}) $$

where $L$ is the matting Laplacian, a sparse symmetric positive semi-definite matrix whose entries derive from local colour covariance:

$$ L_{ij} = \sum_{k | (i,j) \in w_k} \left( \delta_{ij} - \frac{1}{|w_k|}\left(1 + (I_i - \mu_k)^{\top}\left(\Sigma_k + \frac{\varepsilon}{|w_k|}\mathbb{I}\right)^{-1}(I_j - \mu_k)\right)\right) $$

with $\mu_k$ and $\Sigma_k$ the mean and covariance in window $w_k$. Minimising gives a sparse linear system, so matting reduces to solving $(L + \lambda D)\alpha = \lambda D \hat{\alpha}$.

Two reasons this matters beyond history. The colour-line assumption is stated and falsifiable, which is rare. And the matting Laplacian reappears as a regulariser inside modern deep methods and in unrelated tasks such as image editing and depth refinement.

Learned matting#

Xu et al. (2017), Deep Image Matting, established the modern setup. An encoder-decoder takes image and trimap, followed by a refinement stage. The loss combines alpha prediction with a compositional term, penalising $\alpha F + (1-\alpha) B$ against the observed image. They released Composition-1k, built by compositing 493 foregrounds onto COCO backgrounds.

Later directions:

• IndexNet (Lu et al., 2019) learns the unpooling indices, treating upsampling itself as the thing to learn.
• Trimap-free portrait matting (MODNet, and later work) exploits a narrow domain prior to remove the trimap, at the cost of generality.
• Background matting (Sengupta et al., 2020; Lin et al., 2021) asks for a photograph of the empty background instead of a trimap. This is often much easier to obtain — a video call, a fixed security camera — and reduces the unknowns substantially.
• Matting with a mask prompt. A segmentation mask, including one from Segment Anything, can be dilated and eroded into a trimap automatically. This is the standard bridge from segmentation to matting today.

The evaluation problem#

The alphamatting.com benchmark (Rhemann et al., 2009) defines the four standard errors: sum of absolute differences, mean squared error, gradient error, and connectivity error. Gradient and connectivity exist because pixel-wise error tolerates a matte that is numerically close and visually wrong.

The deeper problem is data. Ground-truth alpha requires either careful studio capture or manual creation, so nearly all training data is composited. Networks then key on compositing cues — resampling artefacts, mismatched noise, inconsistent lighting — rather than on the boundary physics. Reported numbers on composited test sets overstate real-photograph performance, and this discrepancy is documented across the matting literature. Evaluate on real photographs from your own domain before believing any benchmark number.

Practical notes#

• Spill suppression is a separate step. Even with a perfect alpha, $F$ recovered from a green-screen shot carries a green cast. Production pipelines run explicit despill after matting.
• Estimate $F$, not only $\alpha$. Compositing with the original $I$ inside the band reintroduces the old background. Methods that predict $F$ alongside $\alpha$ composite correctly.
• Resolution matters more than in segmentation. The band is a few pixels wide, so downsampling to 512 pixels destroys the signal. Guided or patch-based inference at native resolution is standard.
• Alpha is not a probability. Alpha 0.5 means "half covered", not "50 per cent confident". Thresholding it is a category error.

Papers#

• Porter, Duff, Compositing Digital Images, SIGGRAPH 1984
• Chuang, Curless, Salesin, Szeliski, A Bayesian Approach to Digital Matting, CVPR 2001
• Levin, Lischinski, Weiss, A Closed-Form Solution to Natural Image Matting, TPAMI 2008
• Rhemann et al., A Perceptually Motivated Online Benchmark for Image Matting, CVPR 2009
• Xu, Price, Cohen, Huang, Deep Image Matting, CVPR 2017 — arxiv.org/abs/1703.03872
• Lin et al., Real-Time High-Resolution Background Matting, CVPR 2021 — arxiv.org/abs/2012.07810

What to learn next#

• Cleaning up predicted masks — the hard-mask toolbox this lesson argues past.
• How images are stored — channels, alpha and why PNG behaves differently from JPEG.
• Diffusion models — the other route to replacing a background, by generating one.