Cleaning up predicted masks

Morphology, stated properly#

For a binary set $A \subseteq \mathbb{Z}^2$ and structuring element $B$:

$$ A \ominus B = {z : B_z \subseteq A} \quad \text{(erosion)}, \qquad A \oplus B = \bigcup_{b \in B} A_b \quad \text{(dilation)} $$

$$ A \circ B = (A \ominus B) \oplus B \quad \text{(opening)}, \qquad A \bullet B = (A \oplus B) \ominus B \quad \text{(closing)} $$

Properties that justify the operations rather than only describing them:

• Idempotence. $(A \circ B) \circ B = A \circ B$. Applying opening twice changes nothing, so iterating is pointless.
• Anti-extensivity and extensivity. $A \circ B \subseteq A \subseteq A \bullet B$. Opening only removes, closing only adds.
• Duality. $(A \circ B)^c = A^c \bullet \check{B}$. Opening the foreground is closing the background with the reflected element.
• Granulometry. Opening with elements of increasing size gives a size distribution of the shapes present. This is the principled way to choose a size floor, rather than by eye: plot area removed against element size and look for the knee.

Erosion and dilation are separable for rectangular elements, so a $k \times k$ operation costs $O(2k)$ per pixel rather than $O(k^2)$. scipy.ndimage and OpenCV both exploit this.

Thresholding is a decision problem#

The 0.5 default assumes equal costs for false positives and false negatives and a calibrated model. Both assumptions usually fail.

With cost $c_{FP}$ and $c_{FN}$, the Bayes-optimal threshold on a calibrated probability is $t^* = \frac{c_{FP}}{c_{FP} + c_{FN}}$. Dice-trained networks are typically poorly calibrated (Mehrtash et al., 2020), which breaks that formula. It is a further argument for keeping a cross-entropy term, as discussed in Dice and other segmentation losses.

In practice, sweep the threshold on validation data against the metric you actually report, and fix it per class. Reporting a single dataset-level threshold when class frequencies differ by orders of magnitude leaves easy accuracy on the table.

Conditional random fields#

Krähenbühl and Koltun (2011) define a fully connected pairwise CRF over all pixel pairs with Gaussian edge potentials:

$$ E(x) = \sum_i \psi_u(x_i) + \sum_{i<j} \psi_p(x_i, x_j) $$

$$ \psi_p(x_i, x_j) = \mu(x_i, x_j)\left[ w_1 \exp\left(-\frac{|p_i - p_j|^2}{2\theta_\alpha^2} - \frac{|I_i - I_j|^2}{2\theta_\beta^2}\right) + w_2 \exp\left(-\frac{|p_i - p_j|^2}{2\theta_\gamma^2}\right)\right] $$

where $p$ is position and $I$ is colour. The contribution is the inference method: mean-field updates whose message passing is a high-dimensional Gaussian filter, computable in linear time with the permutohedral lattice. That turned a hopeless $O(N^2)$ model into a practical one.

Zheng et al. (2015) unrolled the mean-field iterations as recurrent layers, making the CRF trainable end to end. Both are worth reading as examples of turning an inference procedure into an architecture, even though neither is common in current pipelines.

Alternatives that usually beat post-processing#

Post-processing patches a symptom. Three approaches attack the cause.

• Test-time augmentation. Average predictions over flips and scales. Consistently worth a point or so of IoU, at a linear cost in inference passes. It reduces speckle for the same reason self-training does: inconsistent errors cancel.
• Higher-resolution prediction. Output stride 8 rather than 32, or PointRend-style adaptive refinement at boundaries. More expensive, and it fixes the boundary rather than smoothing it.
• Boundary-aware losses. Penalise distance from the true boundary during training, as in Kervadec et al. (2019).

A good rule: if morphology improves your metric substantially, the model is underfitting the boundary and you should fix the model. If morphology improves structure but barely moves the metric, keep it, because structure is what downstream code consumes.

Evaluation for the properties morphology fixes#

• Boundary IoU (Cheng et al., 2021) restricts IoU to a band of width $d$ around the boundary, so large objects stop masking boundary error.
• Normalised surface distance / surface Dice with a tolerance, standard in medical challenges because tolerance can be stated in millimetres.
• Counting metrics. Segment count error, and the split-and-merge rates from the connectomics literature, which measure exactly what speckle and pinholes damage.

Papers#

• Krähenbühl, Koltun, Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials, NeurIPS 2011 — arxiv.org/abs/1210.5644
• Zheng et al., Conditional Random Fields as Recurrent Neural Networks, ICCV 2015 — arxiv.org/abs/1502.03240
• Cheng, Girshick, Dollár, Berg, Kirillov, Boundary IoU: Improving Object-Centric Image Segmentation Evaluation, CVPR 2021 — arxiv.org/abs/2103.16562
• Mehrtash et al., Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation, IEEE TMI 2020 — arxiv.org/abs/1911.13273

What to learn next#

• Image matting and background removal — when a hard yes-or-no mask is the wrong output entirely.
• OpenCV — the wider toolbox these operations come from.
• Model evaluation — choosing what to measure before you tune anything.