Preprocessing scans for OCR

Binarisation as a decision rule#

Global thresholding assigns pixel $I(x,y)$ to ink when $I(x,y) < t$. Otsu (1979) chooses $t$ by maximising between-class variance:

$$ t^{*} = \arg\max_{t} \; \omega_0(t)\,\omega_1(t)\,\bigl(\mu_0(t) - \mu_1(t)\bigr)^2 $$

Where $\omega_0, \omega_1$ are the pixel fractions below and above $t$, and $\mu_0, \mu_1$ their mean intensities. The criterion is exactly equivalent to minimising within-class variance.

Two assumptions are baked in: the histogram is bimodal, and the classes are of comparable mass. Document images violate both — ink is typically 3–10% of pixels, and illumination widens the paper mode until it dominates. The failure in the code above is not an implementation artefact; it is the assumption being false.

Local methods#

Niblack (1986) replaces the global constant with a per-window statistic:

$$ t(x,y) = \mu_w(x,y) + k \, \sigma_w(x,y) $$

Where $\mu_w$ and $\sigma_w$ are mean and standard deviation over a window $w$, and $k \approx -0.2$. Niblack is noisy in blank regions, where $\sigma_w \to 0$ and the threshold clings to the mean.

Sauvola and Pietikäinen (2000) fix that by scaling the correction with contrast:

$$ t(x,y) = \mu_w(x,y)\left[1 + k\left(\frac{\sigma_w(x,y)}{R} - 1\right)\right] $$

Where $R$ is the dynamic range of $\sigma$ (128 for 8-bit) and $k \approx 0.2$–$0.5$. In blank regions $\sigma_w \approx 0$, so $t \to \mu_w(1-k) < \mu_w$, and nothing is marked as ink. Sauvola remains the default for degraded historical documents; scikit-image ships it as threshold_sauvola.

OpenCV's adaptiveThreshold implements the simpler $t = \mu_w - C$, which is Sauvola with the contrast term removed. It is fast and adequate for evenly-inked modern documents.

Learned binarisation now beats all of the above on damaged manuscripts. The DIBCO / H-DIBCO competition series (2009–2019) is the benchmark, and U-Net style segmentation models and conditional GANs have topped it since roughly 2017. The relevant metrics are F-measure, pseudo-F-measure, PSNR and DRD (distance reciprocal distortion).

Skew estimation#

Three families, with different cost and different failure modes.

Projection profile (Postl, 1986). Define $p_\theta(y)$ as the ink count in row $y$ after rotating by $\theta$. Maximise a spikiness objective, commonly

$$ E(\theta) = \sum_y p_\theta(y)^2 \quad \text{or} \quad \operatorname{Var}y\bigl[p\theta(y)\bigr] $$

Cost is $O(n_\theta \cdot HW)$, which is why coarse-to-fine search is standard. It is the most accurate method on ordinary text pages and the one used above.

Hough transform on the binarised image, taking the dominant line orientation. Robust to sparse text, expensive, and sensitive to ruled lines and table borders — which frequently outvote the text.

Principal component or minimum-area-rectangle on ink coordinates. Effectively $O(n)$ and appealing, but it estimates the orientation of the point cloud, not of the baselines. A single block of text whose aspect ratio is near square gives a nearly arbitrary answer, and outliers dominate. The 0.82-degree error in the output above is this effect in miniature.

Angles beyond roughly $\pm 15^\circ$ need a different treatment. Photographed pages suffer perspective, not rotation, and the correct fix is a homography from the detected page quadrilateral — see homographies and perspective warp. Curved book pages need dewarping models such as DewarpNet (Das et al., ICCV 2019) or DocTr (Feng et al., ACM MM 2021).

Does preprocessing still pay?#

For classical engines, decisively yes. Tesseract 5 with LSTM still expects near-binary input and documents a preferred x-height around 30 pixels.

For modern neural recognisers the answer inverts. CRNN and TrOCR recognisers are trained on augmented natural crops, including blur, low contrast and shadow. Hard binarisation discards intensity information those models use, and published ablations on scene-text benchmarks generally show it hurting. The reliable modern preprocessing set is geometric (deskew, dewarp, crop, resample), not photometric.

The one photometric step that survives everywhere is resolution. No architecture recovers strokes that were never sampled.

What to learn next#

• Text detection models — the stage that consumes this cleaned image.
• Tesseract in practice — the engine that gains the most from good preprocessing.
• Homographies and perspective warp — flattening a page photographed at an angle.