Study guide · Computer vision · 2012

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton · NeurIPS 2012

Could a neural network learn to recognise objects in ordinary photographs, across a thousand categories, better than the carefully engineered systems of the day? This paper trained a much larger convolutional network than anyone had before, on two graphics cards, and cut the error in the 2012 ImageNet challenge from 26.2% to 15.3%. The margin was so large that computer vision switched to deep learning almost overnight.

The paper in four sentences

ImageNet offered 1.2 million labelled photos in 1,000 categories, enough data, for the first time, to train a really big network without it simply memorising the examples.

The authors built a convolutional network with 60 million adjustable numbers, used a very simple on/off-style switch in each unit (the ReLU) so it trained much faster, and split the work across two graphics cards.

To stop it memorising, they showed it shifted, mirrored and recoloured copies of each photo, and randomly switched off half the units in its final layers during training ("dropout").

It won the 2012 ImageNet competition with 15.3% top-5 error against 26.2% for the next best entry, a gap that convinced the field.

Idea · §1–2 · §6

A big test, and a big win

For years, computers were tested on recognising objects in small collections of photos: tens of thousands of images at most. ImageNet changed that. Its yearly competition (ILSVRC) used about 1.2 million training photos, each labelled with one of 1,000 categories: many breeds of dog, types of mushroom, cars, boats, tools and so on.

The main score is top-5 error: the system may give five guesses, and it counts as wrong only if the correct label is not among them. This is fair because many photos contain several objects, or categories that even people confuse.

In 2012 the team's network scored 15.3% top-5 error. The second-best team, which used the best hand-engineered approach of the time, scored 26.2%. In a competition where improvements had come a point or two at a time, this was a leap.

Everyday pictureImagine a quiz where you name the object in a photo and may give five answers. The previous champion missed about one photo in four. The newcomer missed about one in seven.

TakeawayWith enough labelled data and enough computing power, a learned network beat decades of hand-designed vision features by a wide margin.

Check yourself
Why does ImageNet use top-5 error rather than only counting the single best guess?
Photos often contain several objects, and some categories are very similar, so a sensible second or third guess should not count as a failure.

Idea · §3.1 · Fig. 1

A simpler switch that learns faster

Each unit in a neural network adds up its inputs and then passes the total through an activation function. The common choice was a smooth S-shaped curve (tanh) that flattens out for large inputs. Once a unit is far out on a flat part, small changes to its input barely change its output, so the learning signal that tells it how to improve becomes tiny.

AlexNet used the rectified linear unit, or ReLU: output the input if it is positive, otherwise output zero. For positive inputs it never flattens out. The paper's Fig. 1 shows a small four-layer network with ReLUs reaching a target training error on a smaller image dataset (CIFAR-10) about six times faster than the same network with tanh.

Move the slider to change a unit's total input and compare how much of a small nudge each function passes on.

The slope (dashed tangent line) is how much a small change in input changes the output, which is what learning relies on. "Layers in a row" multiplies the slope across that many units with the same input, a rough picture of a learning signal travelling back through a deep network. Illustrative: in a real network each unit has its own input.

TakeawayReLUs keep the learning signal strong wherever they are active, which made training a large network fast enough to be practical.

Check yourself
A tanh unit has an input of 3. Why does it learn slowly?
tanh is almost flat there (slope about 0.01), so changes to its weights barely change its output and the error signal reaching it is tiny.
What is the ReLU's weak spot?
For negative inputs its slope is zero, so a unit that is always "off" gets no learning signal at all.

Idea · §3.2 · §3.5 · Fig. 2

Big network, two graphics cards

The network has eight learned layers: five convolutional layers (the sliding pattern detectors introduced by LeNet) and three ordinary fully connected layers, ending in 1,000 outputs, one per category. It has about 60 million adjustable numbers and 650,000 units.

That was too big for one graphics card of the day, which had 3 GB of memory. So the authors split the network down the middle: half the feature maps on each card, and the two halves only exchange information at certain layers. Graphics cards (GPUs) are built to do thousands of simple multiplications at once, which is exactly what a convolutional network needs. Training still took five to six days.

GPU 1GPU 2 image conv1conv1 conv2conv2 conv3conv3 conv4conv4 conv5conv5 fc6fc6 fc7fc7 1000 amber: halves exchange
Redrawn after Fig. 2. Each GPU holds half of every layer. Communication happens into conv3 and in the fully connected layers; conv2, conv4 and conv5 only see maps on their own GPU.

TakeawayGraphics cards made a network this size trainable in days rather than months, and that, as much as any new idea, is what made 2012 different.

Idea · §4.1 · §4.2

Two ways to stop memorising

A network with 60 million adjustable numbers and "only" 1.2 million photos is at risk of overfitting: learning the quirks of the training photos instead of what a dog actually looks like. The paper used two defences.

Data augmentation. Every photo is stored at 256 by 256 pixels, but the network is shown a random 224 by 224 piece of it, sometimes mirrored left to right. Each photo therefore turns into many slightly different training examples, all with the same correct label. The colours are also jittered slightly. Try it:

Left: the stored 256×256 image with the random 224×224 window. Right: what the network sees. There are 32 × 32 = 1,024 window positions (the paper's count), each with or without mirroring: 2,048 versions of every photo. The drawing is a stand-in for a photo.

Dropout. During training, in the first two fully connected layers, each unit is switched off at random with probability one half, freshly for every example. No unit can rely on a particular partner being there, so each must learn something useful on its own. At test time all units are on, and their outputs are halved to keep the totals at the right scale.

A miniature with 8 + 8 hidden units (AlexNet's fully connected layers have 4,096 each). Crossed-out units are dropped for this example. Each press samples a different "thinned" network, so training effectively averages a huge family of networks that share weights.
Everyday pictureA team where any member might be off sick on a given day learns to spread knowledge around, rather than depend on one star. Dropout forces that kind of resilience on the network.

TakeawayShowing many varied copies of each photo, and randomly removing units during training, let a huge network learn general patterns instead of memorising.

Check yourself
Why mirror left to right but not upside down?
A mirrored dog is still a realistic photo of a dog; an upside-down one mostly is not, so it would teach the network about images it will never see.
Why halve the outputs at test time when using dropout?
During training only about half the units were on, so the next layer is used to receiving about half the total. Halving keeps the scale the same when all are on.

Idea · §6.1 · §7 · Figs. 3–4

What it learned, and why depth mattered

The paper looks inside. The first layer's 96 detectors (Fig. 3) turned out to be edge detectors at various angles and spacings, plus blobs of colour, much like what neuroscientists find early in animal vision. Curiously, the detectors on one graphics card were mostly colourless and those on the other mostly coloured, which happened on every run.

Near the top, photos that the network represents similarly are similar in meaning, even when the pixels are very different: elephants in different poses end up near each other (Fig. 4). And removing any one of the middle convolutional layers made results about 2% worse, which the authors took as evidence that depth itself matters.

Later refinementWithin a few years, networks got deeper and simpler. VGG (2014) used only small 3×3 filters and 16 to 19 layers. ResNet (2015) added shortcut connections and reached 152 layers, with top-5 error near 3.6% on the same challenge. Some AlexNet details were dropped: the special "local response normalisation" layer was replaced by batch normalisation, and splitting across GPUs became an engineering choice rather than part of the design. ReLUs, dropout, data augmentation and GPU training all stayed.

TakeawayThe network learned sensible visual features on its own, from edges upward, and its success made deep learning the default way to do computer vision.

Follow the idea

Built on: LeNet (1998)

Led to: ResNet (2015)