The paper in four sentences
Older handwriting readers had two parts: a feature extractor that engineers designed by hand, and a learning part that only saw those features. The quality of the hand-made part limited everything.
A convolutional network instead learns its own features, using small pattern detectors that are slid across the whole image, so the same detector works wherever the pattern appears.
Layers of detectors alternate with layers that shrink the image, so small shifts and wobbles in the handwriting matter less and less as the signal moves up.
The network, LeNet-5, made fewer mistakes on the MNIST digits than any hand-designed method tested, and joined to other trainable parts it was reading millions of cheques for banks.
Idea · §I · Fig. 1
Stop hand-designing the features
Before this work, a typical character reader was built from two separate pieces. First, a feature extractor, written by engineers, turned the raw pixels into a short list of measurements: how many loops, where the strokes end, how slanted the character is. Second, a classifier, which could learn from examples, looked at those measurements and named the character.
Only the second piece learned. If the engineers forgot a useful measurement, the classifier could never recover it. And every new problem (a new alphabet, a new kind of form) meant designing the features again.
The paper's argument is that we now have enough data, fast enough computers, and good enough learning methods to let the machine learn all of it, straight from the pixels. The learning method is the one from backpropagation: measure the error, work out how each adjustable number contributed to it, and nudge each one to reduce it. This is called gradient-based learning, and "end to end" means it is applied to the whole system at once.
TakeawayLet learning reach all the way down to the pixels, rather than only adjusting a final classifier on top of hand-made features.
Check yourself
- In the old two-part design, which part learned from examples?
- Only the classifier. The feature extractor was fixed and designed by hand, so its blind spots could not be fixed by training.
Idea · §II-A
One small detector, slid everywhere
Why not just connect every pixel to every unit of a big network? The paper gives two reasons. A 32 by 32 image has 1,024 pixels, so a first layer of even 100 units would need over 100,000 connections, and it would need huge amounts of data to learn them. Worse, such a network has no idea that a stroke shifted one pixel to the right is still the same stroke; it would have to learn that separately for every position.
A convolutional layer fixes both. Each unit looks only at a small patch of the image, its receptive field (5 by 5 pixels in LeNet-5). And all the units in one layer map use the same weights: one small detector, copied to every position. The result is a feature map that shows where in the image the detector's pattern appears. Several detectors give several feature maps.
Try it below. Click pixels to draw, pick a detector, and watch the feature map. Hover over the feature map to see which patch each value came from.
TakeawaySharing one detector across all positions means far fewer numbers to learn, and a pattern is recognised wherever it appears.
Check yourself
- If you move your drawing two pixels to the right, what happens to the feature map?
- It moves two positions to the right as well, but otherwise looks the same. The layer reacts to shifts by shifting its output.
- Why does weight sharing reduce the amount of training data needed?
- There are far fewer independent weights to pin down, and every position in every image provides evidence about the same shared detector.
Idea · §II-A
Blur the details, keep the gist
Once a detector has found, say, an edge, its exact position does not matter much. What matters is roughly where it is compared with other features: a 7 has a horizontal bar near the top-left of a diagonal stroke. Exact positions are also unreliable, because every writer is different.
So LeNet follows each convolutional layer with a subsampling layer (today usually called pooling). It averages each 2 by 2 block of the feature map into one number, halving the width and height. After this step, a feature that moved by a pixel produces almost the same output.
Slide the shift below and compare how similar the shifted digit is to the original at each stage. Raw pixels change a lot; the averaged maps much less.
TakeawayAlternating detection with shrinking gives features that are increasingly tolerant of shifts and distortions as you go up the network.
Check yourself
- What does the network lose by subsampling, and why is that acceptable?
- It loses the exact location of each feature. For recognising a character, the rough relative positions of features are what matter, and exact positions vary between writers anyway.
Idea · §II-B · §III · Fig. 2
LeNet-5 and the MNIST test
LeNet-5 stacks these ideas: detect (6 feature maps), shrink, detect again (16 maps, each combining several of the first ones), shrink, then a few ordinary layers that make the final decision among the ten digits. Not counting the input it has seven layers, and about 60,000 adjustable numbers in total, because sharing keeps the count small.
To test it fairly the authors built a new dataset, MNIST, by mixing two collections of digits from the US National Institute of Standards and Technology: one written by Census Bureau employees and one by high-school students. It has 60,000 training digits and 10,000 test digits, with the people who wrote the test digits kept separate from those who wrote the training ones. MNIST went on to become the most famous small benchmark in machine learning.
LeNet-5 misread 0.95% of the test digits, about one in a hundred, and 0.8% when the training set was enlarged with artificially distorted copies of the digits. For comparison, a simple linear classifier got 12% wrong, and comparing each new digit with its few most similar training examples got 5% wrong.
TakeawayA small, carefully shaped network that learns everything from pixels made fewer errors than every hand-engineered approach the authors tested.
Check yourself
- Why did the authors make sure different people wrote the training and test digits?
- Otherwise a method could score well by memorising particular writers' styles, which says little about reading new people's handwriting.
- Why did adding distorted copies of the training digits help?
- It showed the network more of the natural variety in handwriting (slants, squashes, stretches), so it learned to ignore those variations.
Idea · §IV–end
From single digits to reading cheques
Reading a real cheque is harder than reading a single neat digit. You have to find the amount, cut it into characters (where does one end and the next begin?), recognise each one, and check that the result makes sense as an amount of money. Usually each of these steps was a separate program, tuned separately.
The second half of the long paper extends "learn everything" to the whole pipeline. It introduces Graph Transformer Networks: a way of describing a multi-step system in which every step passes along a set of possible interpretations (for example, several possible ways to cut a word into letters, each with a score), and the whole chain can be trained together by gradient descent, so the error at the end improves every step.
The paper's abstract reports that the resulting cheque reader was deployed commercially and "reads several million checks per day". This was one of the first large-scale commercial uses of neural networks.
TakeawayThe paper's bigger idea is not just one network, but training a whole recognition system end to end so every part serves the final goal.
Abstract, restated
Multilayer networks trained with back-propagation are a successful gradient-based learning technique and, given an appropriate architecture, can synthesise complex decision surfaces for high-dimensional patterns such as handwritten characters with minimal preprocessing. The paper reviews methods for handwritten character recognition and compares them on a standard digit-recognition task, finding that convolutional networks, designed to deal with the variability of 2D shapes, outperform all other techniques. For real document recognition, which involves field extraction, segmentation, recognition and language modelling, it proposes Graph Transformer Networks (GTN), which allow such multi-module systems to be trained globally to optimise an overall performance measure. Experiments on on-line handwriting recognition and a commercially deployed bank-cheque reader illustrate the approach. Proceedings of the IEEE 86(11), 2278–2324 (1998).
Section · §II-A
Three architectural priors
The case against a fully connected network on raw pixels (§II-A): a large first layer has tens of thousands of weights and so high capacity and data needs; memory requirements grow accordingly; and, most importantly, it has no built-in invariance to translations or local distortions, so it would need many units with similar weight patterns at different input positions. It also ignores the input topology: pixels can be permuted with no effect on a fully connected net, whereas images have strong 2D local correlations.
Convolutional networks build in three priors: local receptive fields, shared weights (each feature map applies one kernel at every location, so the layer is a convolution followed by an additive bias and a squashing function), and spatial sub-sampling. A shift of the input shifts the feature map by the same amount (equivariance), and subsampling reduces sensitivity to shifts and distortions. A subsampling unit in LeNet sums its four inputs, multiplies by a trainable coefficient, adds a trainable bias and applies the sigmoid; so each subsampling map has exactly two parameters.
Units use a scaled hyperbolic tangent,
with S setting the slope at the origin; the rationale is in the paper's Appendix A. Inputs are normalised so the foreground is 1.175 and the background slightly negative, which makes the mean input roughly zero and the variance roughly one and so speeds learning. The paper notes the fully trained system has about 340,908 connections but only about 60,000 trainable parameters because of weight sharing.
Check yourself
- Is a convolutional layer invariant or equivariant to translation?
- Equivariant: shifting the input shifts the output. Approximate invariance comes from subsampling and from the stack as a whole, not from a single convolution.
Section · §II-B · Fig. 2 · Table I
LeNet-5, layer by layer
The input is 32 × 32, larger than the characters, which are at most 20 × 20 pixels centred in a 28 × 28 field. The margin is deliberate: distinctive features such as stroke endpoints and corners can then appear near the centre of the receptive fields of the highest-level feature detectors. C3 does not connect every S2 map to every C3 map. Table I specifies a sparse scheme, which keeps the connection count bounded and "forces a break of symmetry": different C3 maps see different input subsets and so are pushed to extract different, complementary features.
Click a layer to see its shape and how its parameter and connection counts arise. Hover over a column of the C3 table to highlight which S2 maps feed it.
C5 is labelled a convolutional layer even though, with a 5 × 5 input and 5 × 5 kernels, its maps are 1 × 1, because with a larger input it would produce larger maps. F6 has 84 units, and the output is ten Euclidean radial basis function (RBF) units:
Each output measures the distance between F6's output and a prototype vector for class i. The prototypes were fixed by hand to stylised 7 × 12 bitmaps of the characters (84 = 7 × 12), coded as ±1, so that confusable characters such as O and 0 have similar codes, which helps a later linguistic stage resolve them. Because yi is a distance, lower is better: it acts as a penalty, not a score.
Section · §II-C · §III-B · Appendix C
Loss and training
The simplest criterion is the mean penalty of the correct class over the P training patterns,
which is a maximum-likelihood criterion but has two weaknesses: if the RBF prototypes were learnable it has a trivial collapsed solution (all prototypes and F6 outputs equal), and it has no competition between classes. The paper therefore uses a discriminative, MAP-style criterion that additionally pushes up the penalties of incorrect classes:
The constant j plays the role of a "rubbish" class. The second term is a log-sum-exp over negated penalties, so this is the same family of losses as today's softmax cross-entropy.
Optimiser. A stochastic version of the diagonal approximation to Levenberg–Marquardt: each parameter gets its own step size, derived from a running estimate of the diagonal of the Gauss–Newton Hessian, re-estimated on 500 samples before each pass. Global learning-rate schedule: 0.0005 for the first two passes, 0.0002 for the next three, 0.0001 for the next three, 0.00005 for the next four, and 0.00001 thereafter, for 20 passes in all. Test error stabilises near 0.95% after about ten passes; training error reaches 0.35% after 19. Training took two to three days of CPU on a single 200 MHz R10000 processor (§III-D).
Check yourself
- Why does Eq. 8 alone collapse if prototypes are learnable?
- It only rewards a small distance to the correct prototype, so making every prototype and every F6 output identical drives all penalties to zero without discriminating anything.
- How does the second term in Eq. 9 relate to softmax cross-entropy?
- With scores −yi, the loss is (minus the correct score) plus log-sum-exp of all scores, which is exactly the negative log of a softmax probability (with an extra constant class).
Section · §III-A · §III-C · Fig. 9
The MNIST comparison
MNIST was assembled from NIST Special Databases 1 and 3 (SD-3 written by Census Bureau employees, SD-1 by high-school students, and SD-3 much cleaner). The authors re-mixed them so that the 60,000 training patterns and the 10,000 test patterns each draw on both sources, with disjoint writers. Images are size-normalised to 20 × 20 preserving aspect ratio, then centred by centre of mass in a 28 × 28 field.
The comparison (Fig. 9) spans linear classifiers, k-nearest neighbours, PCA with a polynomial classifier, RBF networks, fully connected MLPs, tangent distance, several support vector machines, and three LeNets. Switch views below.
The best result, 0.7%, is boosted LeNet-4: three LeNet-4s, the second trained on a mix of patterns the first got right and wrong, the third on patterns where the first two disagree. The paper's discussion (§III-D) also weighs memory and recognition time, where the convolutional nets compare favourably with memory-hungry k-NN and SVM methods, and shows LeNet-5's robustness to noise and distortion (§III-E).
Section · §IV onward · lineage
Global training, limitations and lineage
Graph Transformer Networks (§IV–§VI). A recognition system is cast as modules that pass graphs rather than fixed-size vectors: each arc carries a hypothesis (such as a candidate segment of a word) with a penalty. Heuristic over-segmentation (§V) produces a segmentation graph; a recognition transformer runs the character recogniser on every arc; a Viterbi transformer picks the lowest-penalty path. §VI shows how to train the whole thing: Viterbi training, discriminative Viterbi training, and discriminative forward training, where the forward algorithm's soft-min over all paths replaces the single best path so that gradients flow to every hypothesis. These are close relatives of what speech recognition later called sequence-discriminative training and of CTC-style losses.
Limitations. The architecture is shaped by hand (sizes, the C3 table, the fixed RBF codes), the network is tiny by later standards, and the evidence is confined to characters and documents. Saturating tanh units and per-parameter second-order step sizes were sensible on CPUs in 1998 but did not carry over to large-scale training.
Check yourself
- Why does discriminative forward training give gradients to more hypotheses than Viterbi training?
- Viterbi uses only the single best path, so only its arcs receive gradient. The forward algorithm combines all paths with a smooth log-sum-exp, so every arc contributes in proportion to how likely its paths are.
Reference
Glossary
- Receptive field
- The input region that one unit sees.
- Feature map
- The output plane of one shared kernel applied at every position.
- Weight sharing
- Using identical weights at every position of a feature map.
- Subsampling
- Averaging (here: sum × trainable coefficient + bias) over 2 × 2 blocks to halve resolution.
- RBF unit
- Output unit computing a squared Euclidean distance to a prototype vector.
- MNIST
- Modified NIST: 60,000 training and 10,000 test digits, 28 × 28.
- GTN
- Graph Transformer Network: modules that map scored graphs to scored graphs, trainable end to end.
- Viterbi transformer
- Selects the minimum-penalty path through a graph.