Super-resolution

The problem is one-to-many#

Take a low-resolution $\mathbf{y} = (\mathbf{x} \otimes \mathbf{k}) \downarrow_s + \mathbf{n}$. Here $\mathbf{k}$ is a blur kernel, $\downarrow_s$ is decimation by factor $s$, and $\mathbf{n}$ is noise. Many high-resolution $\mathbf{x}$ are consistent with the same $\mathbf{y}$. The task is not inversion; it is sampling or selecting from a set.

Minimising $\mathbb{E}|\mathbf{x} - \hat{\mathbf{x}}|^2$ returns the conditional mean $\mathbb{E}[\mathbf{x} \mid \mathbf{y}]$. That is the average of all consistent images, and so it is blurred by construction. This is not an approximation error. It is the exact minimiser of the stated objective.

The perception-distortion trade-off#

Blau and Michaeli (2018) prove this is a fundamental bound, not an engineering limitation. Define distortion as $\mathbb{E}[\Delta(\mathbf{x}, \hat{\mathbf{x}})]$, for any full-reference measure $\Delta$. Define perceptual quality as $d(p_{\hat{\mathbf{x}}}, p_{\mathbf{x}})$, a divergence between the output distribution and the distribution of natural images. Their result:

The perception-distortion function $P(D) = \min_{p_{\hat{\mathbf{x}}|\mathbf{y}}} d(p_{\mathbf{x}}, p_{\hat{\mathbf{x}}}) \;\text{ s.t. }\; \mathbb{E}[\Delta] \le D$ is monotonically non-increasing and convex.

Improving distortion past a point necessarily worsens perceptual quality, for every distortion measure. The PIRM 2018 challenge was designed around this, scoring methods on a perception-distortion plane rather than a single axis.

The practical reading is short. A paper claiming both the best PSNR and the best perceptual quality is doing one of two things. It is either evaluating where the bound is loose, or measuring something incorrectly.

Architecture lineage#

EDSR's batch-norm removal is worth stating because it generalises to all restoration. Batch norm normalises away the per-image contrast and colour statistics. In classification that is a feature. In restoration it is destruction of the signal. Restoration networks use no normalisation, or layer norm.

Real-ESRGAN and BSRGAN are the practical turning point. Both argue the architecture was never the bottleneck. Models trained on bicubic downscales fail on real photographs. The degradation assumed at training time does not match reality. Their solution is a randomised composition of blur, resize, noise and JPEG compression, applied in several rounds. The NTIRE 2026 mobile real-world super-resolution challenge report confirms this remains standard. Entries follow Real-ESRGAN-style degradation simulation and BasicSR engineering practice as the stable baseline.

Diffusion-based restoration#

Diffusion priors sit at the far perceptual end of the curve. StableSR, DiffBIR and SUPIR condition a pretrained text-to-image diffusion model on the degraded input. They produce textures no regression model can match. The cost is low fidelity, high compute, and hallucinated content.

The 2026 literature reports hybrid designs converging on this. They combine the structural reliability of GAN-based methods with the texture generation of diffusion models. Residual refinement holds the two ends together. Evaluation in these challenges leans on no-reference metrics: NIQE, NRQM, PI, CLIP-IQA, MUSIQ. The full-reference metrics penalise the behaviour these methods are designed for.

Evaluation, and what each metric actually rewards#

• PSNR: pure distortion. Rewards the conditional mean. Correlates poorly with human judgement above about 30 dB.
• SSIM / MS-SSIM: local structure comparison. Better than PSNR, still a distortion measure and still on the wrong side of the bound.
• LPIPS (Zhang et al., 2018): distance in the features of a pretrained network, calibrated against human judgements. Better correlated, and it is still a reference metric, so it retains a distortion character.
• NIQE, PI, MUSIQ, CLIP-IQA: no-reference. They score whether the output looks like a natural image, with no ground truth involved. A model can score well here while inventing content freely, which is both the point and the danger.

The honest report is a table with at least one metric from each of the three groups. Add the operating point on the perception-distortion plane.

The evidential problem#

Upscaling is generation, and generated faces are drawn from the training distribution. The 2020 PULSE incident is the widely cited example. A downsampled photograph of Barack Obama upscaled to a white face. The underlying cause is structural rather than a bug. The model samples a plausible face from its prior, and the prior reflects its training data.

This is a hard constraint on deployment. Super-resolution output must not be presented as recovered evidence about an individual, in forensic, medical or identification contexts. Menon et al. (2020) added a statement to PULSE's own paper acknowledging exactly this failure mode.

Papers#

• Dong et al., Image Super-Resolution Using Deep Convolutional Networks (SRCNN), 2014 — arxiv.org/abs/1501.00092
• Shi et al., Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel CNN (ESPCN), 2016 — arxiv.org/abs/1609.05158
• Ledig et al., Photo-Realistic Single Image Super-Resolution Using a GAN (SRGAN), 2017 — arxiv.org/abs/1609.04802
• Lim et al., Enhanced Deep Residual Networks for Single Image Super-Resolution (EDSR), 2017 — arxiv.org/abs/1707.02921
• Blau and Michaeli, The Perception-Distortion Tradeoff, 2018 — arxiv.org/abs/1711.06077
• Wang et al., ESRGAN, 2018 — arxiv.org/abs/1809.00219
• Liang et al., SwinIR, 2021 — arxiv.org/abs/2108.10257
• Wang et al., Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data, 2021 — arxiv.org/abs/2107.10833
• Zhang et al., Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (BSRGAN), 2021 — arxiv.org/abs/2103.14006

What to learn next#

• Watermarking and provenance — proving where an image came from, once it can be invented.
• Inpainting and outpainting — the other restoration task, with the same honesty problem.
• Convolutional neural networks — the backbone underneath every method here.