Recognition: OCR and Speech With CTC

Reading text from images and transcribing speech share a shape: a long sequence of frames (pixel columns or audio slices) in, a shorter sequence of characters or words out, and no labels saying which frames belong to which character. This module covers a production OCR system built on a vision transformer, the loss that trains without alignments, the same idea in speech, and how recognizers are scored.

27.1 PP-OCR and SVTR

PP-OCR is Baidu's open-source optical character recognition (OCR) system, widely used on servers and small devices. A detector finds each text line, and a recognizer reads one cropped line at a time. The recognizer, SVTR ("Single Visual model for scene Text Recognition", Du et al., 2022), is a vision transformer alone, with no language model. Like ViT (section 25.2) it attends over patches, but a text line is short and very wide and must become a left-to-right sequence, so SVTR shrinks the height step by step to a single row of vectors.

In words. A 32 by 320 crop becomes an 8 by 80 patch grid; two merges halve the height, and pooling squeezes it to one row of 80 vectors. A linear layer gives each a character distribution, and CTC (section 27.2) turns the 80 guesses into a transcript.

In words. The tensor's shape through SVTR, each stage drawn as a labelled grid: crop, eight-by-eighty patches, two height-halving merges, and a final row of eighty vectors.

Key Ideas in SVTR

1. Local and global mixing. Characters are small strokes (a bar, a loop, a dot), but words need context along the line. Local mixing attends within a small window to pick up strokes; global mixing attends across the line to relate characters.
2. Shrinking the height, not the width. After the patch embedding, SVTR never reduces the width of its $1:10$ input, so neighboring letters stay in separate positions: 80 for a 320-pixel line, plenty for one line's characters.

27.2 The Alignment Problem and the BlankRun this section

The model produces $T$ outputs, one per frame (an image column position or an audio slice), for a target of $U < T$ labels: say $T = 80$ frames for $U = 15$ characters. Nobody labels "the letter a spans frames 12 to 15".

Connectionist Temporal Classification (CTC) (Graves et al., 2006) adds one output symbol, the blank $\epsilon$ ("no character here"), defines a rule turning frame-by-frame outputs into a transcript, and trains the model to make the correct transcript likely, summed over every frame-by-frame output that produces it.

The Collapsing Operator $\mathcal{B}$

An alignment (or path) $\pi = (\pi_1, \dots, \pi_T)$ has one character or blank per frame. The collapsing operator $\mathcal{B}$ turns it into a label sequence $\mathbf{l}$ in two steps, in order:

1. Merge runs of the same symbol into one: $(a, a, b) \to (a, b)$.
2. Delete every blank: $(a, \epsilon, b) \to (a, b)$.

For example, both $(c, c, \epsilon, a, t)$ and $(\epsilon, c, \epsilon, a, a, t)$ collapse to "cat".

The order makes double letters possible: $(c, c, a, t) \to$ "cat" but $(c, \epsilon, c, a, t) \to$ "ccat". Adjacent identical symbols always merge, so a double letter needs a blank between, and without blanks "book" could never be written. Hence a minimum length: one frame per label plus one per double letter, so "cat" needs $T \ge 3$ and "book" $T \ge 5$.

In words. Three alignments collapse to the same word; the loss adds up the probability of every alignment that does.

27.3 The CTC Objective and Its RecursionRun this section

With $y_k^t$ the probability of symbol $k$ (blank included) at frame $t$, and frames independent given the input $\mathbf{x}$, an alignment's probability is the product over frames, and the transcript's is the sum over alignments that collapse to it:

$\mathcal{B}^{-1}(\mathbf{l})$ is the set of alignments that collapse to $\mathbf{l}$. The loss is the negative log-likelihood (section 3.4):

There are $K^T$ paths for $K$ symbols, so CTC computes the sum by dynamic programming, filling a table of partial sums so each shared piece is computed once: the forward-backward algorithm. The forward half:

1. Put a blank before, between and after the labels. For "cat" this gives $\epsilon\, c\, \epsilon\, a\, \epsilon\, t\, \epsilon$, which has $2|\mathbf{l}| + 1 = 7$ positions.
2. Let $\alpha_t(s)$ be the total probability of all partial alignments that cover frames 1 to $t$ and end at position $s$ of that extended sequence.
3. To get $\alpha_t(s)$, add up the ways to arrive: stay at $s$, step from $s-1$, or jump from $s-2$ over a blank. The jump is allowed only when the labels on either side of that blank differ, which is the double-letter rule again. Multiply the sum by $y^t$ for the symbol at $s$.
4. At the end, add $\alpha_T$ at the last two positions (the final label, and the final blank).

In words. The CTC table for a short word, rows the word with blanks inserted and columns the frames. Each step fills a column from the same row, the row above, or two above when the skip is allowed, times the frame's probability for that symbol. The last two cells of the last column sum to the word's probability.

The table costs $\mathcal{O}(T \cdot |\mathbf{l}|)$ instead of exponential. The backward half runs the same recursion from the end, and together they give the gradient.

Measured (lab 7, experiments/ctc_alignment.py). Two checks. First, for every $T$ from 3 to 8, listing every path by brute force and adding up the ones that collapse to "cat" gives exactly the same probability as the recursion, to the last bit. Second, counting those paths separately shows how fast they multiply: $1, 7, 84, 462, 1716, 5005$ of them at $T = 3, 4, 6, 8, 10, 12$, out of $4^T$ possible paths in total. At $T = 100$ with $|\mathbf{l}| = 20$, brute force would list $10^{134.2}$ paths, while the recursion fills in $4{,}100$ table entries.

27.4 The Same Idea in Speech

Speech is cut into overlapping frames, typically 25 milliseconds every 10, each a vector (often a mel-spectrogram slice, section 28.1). A minute of speech is 6,000 frames and perhaps 150 words, with no word boundaries labelled.

CTC was used for speech first. A CTC acoustic model is an encoder, now usually a transformer or a convolution-augmented Conformer, with a linear layer giving a distribution over characters or subword tokens, plus blank, per frame, trained with section 27.3's loss. With frames independent given the input, it decodes fast and can stream, emitting text while the speaker talks.

That independence is also its weakness: it cannot use its own output so far, so it has no built-in language model. Transducers (RNN-T) add a small prediction network over the output so far; encoder-decoder models (section 8.3) such as Whisper attend over the whole audio and write autoregressively, more accurate but hard to stream. Many production systems train a CTC loss alongside an attention decoder to get both.

27.5 Measuring Recognition: Error RatesRun this section

The edit distance is the fewest single-symbol substitutions $S$, deletions $D$ and insertions $I$ turning the output into the reference transcript. Divided by the reference length $N$:

Over characters it is the character error rate (CER), usual for OCR; over words, the word error rate (WER), usual for speech. One wrong letter is one character error but a whole word error, so WER is typically several times CER, and insertions let either exceed 100%.

Both texts are first normalized for case, punctuation, spacing and number formats, so "3" for "three" is not penalized unless it matters. As with benchmark prompts (section 13.2), the rules change the score, so error rates are comparable only under the same rules.

Exercises

Core Academic References

• Du et al. (2022): SVTR: Scene Text Recognition with a Single Visual Model. IJCAI 2022. arXiv:2205.00159
• Du et al. (2020): PP-OCR: A Practical Ultra Lightweight OCR System. arXiv:2009.09941
• Graves et al. (2006): Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. ICML 2006.
• Graves (2012): Sequence Transduction with Recurrent Neural Networks (RNN-T). arXiv:1211.3711
• Gulati et al. (2020): Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech 2020. arXiv:2005.08100
• Radford et al., OpenAI (2023): Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). ICML 2023. arXiv:2212.04356
• Levenshtein (1966): Binary Codes Capable of Correcting Deletions, Insertions, and Reversals. Soviet Physics Doklady, 10(8): 707–710.