Reconstructing scans with learned priors

A scanner does not capture a picture directly — it captures raw signal that has to be reconstructed into an image, and a learned model can now help fill in the gaps.

By Pranay Mahendrakar
10 min read
3 reading levels
Published 2 Sep 2026

Read these first

On this page 6

Why it exists
How it works
Where you have already seen it
An honest warning
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A scan is not captured as a picture — it is reconstructed from raw signal.

Think about a jigsaw puzzle with a few pieces missing. You can often still guess the full picture. You use the pieces you have, and your experience of what pictures usually look like.

MRI and CT scanners do something similar, out of necessity. Measuring every possible detail takes real time, and a patient cannot always hold still that long. So scanners measure less than everything, and reconstruction fills in the rest.

Why it exists

An MRI scanner does not directly record an image. It records signal in k-space — a mathematical representation of the image, not the image itself. A full reconstruction requires converting this raw signal into a picture.

Measuring every point in k-space, fully, takes time. A patient lying still longer, in a loud scanner, is not free either. So scanners often measure only some of it, undersampling to speed up the scan. The trade-off is a reconstruction problem: turning partial, incomplete signal into a full, honest image.

Older methods filled the gaps with fairly simple mathematical assumptions. Newer methods use a model trained on many prior scans. It learns what real anatomy tends to look like. Then it uses that experience to guess the missing parts.

How it works

Full signal (ideal, but slow to collect)
        |
        v (only some of it actually collected -- faster scan)
Undersampled signal
        |
        v
[ reconstruction, guided by a learned prior ]
        |
        v
Final image, filling gaps using experience from many past scans

The image was never "hidden" in the raw signal waiting to be revealed. It is actively reconstructed, using both the real measurements and a model's learned expectations.

Where you have already seen it

Zoomed, low-resolution photos "enhanced" by phone software use a similar idea. A model fills in plausible detail, informed by many other photos it learned from.
Noise reduction on a low-light photo works the same way. It fills in a cleaner image, guided by learned expectations about what real photos look like.

An honest warning

A learned reconstruction model can produce a confident, sharp-looking image that includes detail that was never actually measured. That detail came from the model's learned expectations, not from the patient's real anatomy. Confirming that a reconstruction faithfully reflects real measurements is an active area of research. It also has to avoid confidently inventing plausible-looking, wrong detail — and that problem is not yet solved.

Remember this

A scan is reconstructed from raw signal, not captured as a picture directly.
Undersampling speeds up a scan, at the cost of a harder reconstruction problem.
A learned reconstruction model can produce a confident image containing detail that was never actually measured — a real, documented risk.
A reconstruction method needs regulatory clearance and a radiologist's ongoing scrutiny before it can be trusted on a real patient's scan.

What to learn next

Latent diffusion — the general generative technique behind many modern reconstruction models.
DICOM, windowing and image intensity — what happens right after reconstruction produces an image.
Super-resolution — a closely related "fill in missing detail" problem, outside medical imaging.

Developer — Code and libraries.

Setup

bash

pip install numpy scipy

Minimal runnable code

undersampled_reconstruction.py

import numpy as np
from scipy.ndimage import median_filter

rng = np.random.default_rng(0)

# A tiny synthetic "scan": two bright squares on a dark background. Real MRI
# does not photograph the body directly -- it measures in "k-space", the 2D
# Fourier transform of the image, one frequency line at a time.
image = np.zeros((64, 64))
image[15:30, 15:30] = 1.0
image[35:50, 35:50] = 0.7
k_space = np.fft.fftshift(np.fft.fft2(image))

# Scanning every line takes time. Undersampling means measuring only some
# lines and skipping the rest -- a faster scan, at the cost of missing data.
mask = np.zeros(64, dtype=bool)
mask[::3] = True                     # keep every third line
mask[28:36] = True                   # always keep the low frequencies (the coarse shape)
k_undersampled = k_space * mask[:, None]

# Zero-fill reconstruction: put zeros where data is missing, transform back.
zero_fill = np.abs(np.fft.ifft2(np.fft.ifftshift(k_undersampled)))

def data_consistency(estimate, k_measured, mask):
    """Force the reconstruction to match every k-space line actually measured."""
    k_est = np.fft.fftshift(np.fft.fft2(estimate))
    k_est[mask] = k_measured[mask]
    return np.abs(np.fft.ifft2(np.fft.ifftshift(k_est)))

# A crude stand-in for a "learned prior": alternate between (a) matching the
# measured data and (b) a denoising step. A real system replaces step (b)
# with a trained neural network; here it is a median filter.
estimate = zero_fill.copy()
for _ in range(15):
    estimate = median_filter(estimate, size=3)
    estimate = data_consistency(estimate, k_undersampled, mask)

def rmse(a, b):
    return np.sqrt(np.mean((a - b) ** 2))

print(f"kept {mask.sum()} of {len(mask)} k-space lines ({mask.mean():.0%})")
print(f"zero-fill reconstruction RMSE:    {rmse(zero_fill, image):.4f}")
print(f"prior-guided reconstruction RMSE: {rmse(estimate, image):.4f}")

Output

kept 28 of 64 k-space lines (44%)
zero-fill reconstruction RMSE:    0.0731
prior-guided reconstruction RMSE: 0.0260

What actually happened

Using only 44% of the k-space lines, the naive zero-fill reconstruction has an error (RMSE) of 0.0731. Alternating between a simple prior (a median filter, standing in for a trained neural network) and forcing consistency with the real measured data drops that error to 0.0260 — roughly a two-thirds reduction, from the same undersampled data.

This is the whole idea behind learned reconstruction, shown at small scale: a prior fills in plausible structure, and a data-consistency step keeps that plausible structure honest, anchored to what was actually measured. Real systems replace the median filter with a trained neural network, learned from thousands of full scans — far more powerful, and, as the honest warning above states, also far more capable of confidently inventing detail that a simple median filter never could.

Line by line, the parts that are not obvious:

np.fft.fft2 and np.fft.ifft2 move between image space and k-space — this Fourier relationship is the mathematical foundation of MRI acquisition, not an implementation detail specific to this example.
mask[28:36] = True always keeps the central, low-frequency lines. Low frequencies carry the coarse shape of the image; skipping them would destroy far more information than skipping high-frequency detail lines.
data_consistency is the step that keeps this process honest: no matter how the estimate changes, it is always snapped back to agree with the k-space lines that were genuinely measured.

Common mistakes

Judging reconstruction quality from a single image, by eye. A sharp, clean-looking image can still contain confidently invented detail — quantitative comparison against real, fully-sampled ground truth is needed to catch this, which is not always available outside carefully controlled research settings.

Skipping the data-consistency step. Without it, a learned prior is free to hallucinate any plausible-looking image, completely untethered from what was actually measured — data consistency is what keeps a reconstruction anchored to reality.

Assuming higher undersampling is always fine if the reconstruction "looks good." Push undersampling far enough, and even a strong prior cannot recover genuinely missing structure — it can only guess, and a confident-looking guess is not the same as a correct one.

Try it yourself

Change mask[::3] to mask[::6], undersampling even more aggressively. Rerun and compare both RMSE values — watch how much further the zero-fill reconstruction degrades, compared to the prior-guided one.

What to learn next

Latent diffusion — the modern generative technique behind current state-of-the-art learned reconstruction.
DICOM, windowing and image intensity — the format a reconstructed image ultimately gets stored in.
Super-resolution — the same "invent plausible missing detail" trade-off, outside medical imaging.

Researcher — Mathematics and papers.

Compressed sensing MRI

Lustig, Donoho & Pauly (2007) formalised undersampled MRI reconstruction as a compressed sensing problem, exploiting the fact that MR images are typically sparse in some transform domain (e.g. wavelets), even when not sparse in the image domain directly:

minimize_x   || F_u * x - y ||_2^2  +  lambda * || Psi * x ||_1

x — the image being reconstructed
y — the measured, undersampled k-space data
F_u — the undersampled Fourier operator (Fourier transform, restricted to the measured lines)
Psi — a sparsifying transform (e.g. a wavelet transform)
lambda — a regularisation weight balancing data fidelity against sparsity

The first term enforces agreement with real measurements (the same role as data_consistency in the developer block). The second term is the classical, hand-designed "prior" — a preference for sparse wavelet coefficients — which learned methods later replaced with a trained model.

Unrolled optimisation networks

The variational network (Hammernik et al., 2018) "unrolls" an iterative optimisation algorithm into a fixed number of neural network layers, each layer performing one data-consistency step followed by one learned regularisation step — structurally identical to the alternating loop in the developer block, but with every "denoising" step replaced by a trained convolutional block, and every step's parameters learned end to end from paired undersampled/fully-sampled training data.

Documented instabilities

Antun et al. (2020, PNAS) is a widely cited study demonstrating that certain deep-learning-based MRI reconstruction methods can be unstable: small, clinically plausible perturbations to the input (including the addition of a tiny structure not actually present) could cause the reconstruction to confidently produce or remove fine detail incorrectly, in ways that classical compressed-sensing reconstruction did not exhibit under the same perturbations. This paper is a central reference for the honesty warning in the beginner block above — it is not a hypothetical concern, but a demonstrated property of specific published methods, and remains a central motivation for the evaluation standards covered below.

Evaluation

The fastMRI challenge (Zbontar et al., 2018; Muckley et al., 2021) established a large public benchmark and standardised evaluation protocol for learned MRI reconstruction, using structural similarity (SSIM), PSNR, and — increasingly — downstream clinical task performance (e.g. whether a radiologist's diagnosis from the reconstruction matches their diagnosis from the fully-sampled reference), rather than pixel-error metrics alone, in direct response to findings like Antun et al.'s.

Key references

Lustig, M., Donoho, D. & Pauly, J. (2007). Sparse MRI: The Application of Compressed Sensing for Rapid MR Imaging. Magnetic Resonance in Medicine 58(6).
Hammernik, K. et al. (2018). Learning a Variational Network for Reconstruction of Accelerated MRI Data. Magnetic Resonance in Medicine 79(6).
Antun, V. et al. (2020). On instabilities of deep learning in image reconstruction and the potential costs of AI. PNAS 117(48).
Zbontar, J. et al. (2018). fastMRI: An Open Dataset and Benchmarks for Accelerated MRI. arXiv:1811.08839.
Muckley, M. et al. (2021). Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction. IEEE Transactions on Medical Imaging 40(9).

Current state

Diffusion-based and score-based generative priors are an active research direction for reconstruction, offering strong image quality but requiring careful evaluation for the instability risks Antun et al. documented. No learned reconstruction method is exempt from the general regulatory and validation requirements covered in validating a model before it touches patients — a reconstruction method sits directly in the imaging pipeline feeding every downstream clinical decision, which is exactly why its failure modes deserve this level of scrutiny.

What to learn next

Latent diffusion — the modern generative modelling technique underlying current reconstruction research.
Validating a model before it touches patients — the validation standard any reconstruction method needs to meet before real use.
DICOM, windowing and image intensity — returning to the start of this section, now with the full imaging pipeline in view.

🟢 Beginner — No maths. Plain English.

Why it exists#

How it works#

Where you have already seen it#

An honest warning#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

Minimal runnable code#

What actually happened#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

Compressed sensing MRI#

Unrolled optimisation networks#

Documented instabilities#

Evaluation#

Key references#

Current state#

What to learn next#

Beginner — No maths. Plain English.

Why it exists

How it works

Where you have already seen it

An honest warning

Remember this

What to learn next

Developer — Code and libraries.

Setup

Minimal runnable code

What actually happened

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

Compressed sensing MRI

Unrolled optimisation networks

Documented instabilities

Evaluation

Key references

Current state

What to learn next