The paper in four sentences
ImageNet offered 1.2 million labelled photos in 1,000 categories, enough data, for the first time, to train a really big network without it simply memorising the examples.
The authors built a convolutional network with 60 million adjustable numbers, used a very simple on/off-style switch in each unit (the ReLU) so it trained much faster, and split the work across two graphics cards.
To stop it memorising, they showed it shifted, mirrored and recoloured copies of each photo, and randomly switched off half the units in its final layers during training ("dropout").
It won the 2012 ImageNet competition with 15.3% top-5 error against 26.2% for the next best entry, a gap that convinced the field.
Idea · §1–2 · §6
A big test, and a big win
For years, computers were tested on recognising objects in small collections of photos: tens of thousands of images at most. ImageNet changed that. Its yearly competition (ILSVRC) used about 1.2 million training photos, each labelled with one of 1,000 categories: many breeds of dog, types of mushroom, cars, boats, tools and so on.
The main score is top-5 error: the system may give five guesses, and it counts as wrong only if the correct label is not among them. This is fair because many photos contain several objects, or categories that even people confuse.
In 2012 the team's network scored 15.3% top-5 error. The second-best team, which used the best hand-engineered approach of the time, scored 26.2%. In a competition where improvements had come a point or two at a time, this was a leap.
TakeawayWith enough labelled data and enough computing power, a learned network beat decades of hand-designed vision features by a wide margin.
Check yourself
- Why does ImageNet use top-5 error rather than only counting the single best guess?
- Photos often contain several objects, and some categories are very similar, so a sensible second or third guess should not count as a failure.
Idea · §3.1 · Fig. 1
A simpler switch that learns faster
Each unit in a neural network adds up its inputs and then passes the total through an activation function. The common choice was a smooth S-shaped curve (tanh) that flattens out for large inputs. Once a unit is far out on a flat part, small changes to its input barely change its output, so the learning signal that tells it how to improve becomes tiny.
AlexNet used the rectified linear unit, or ReLU: output the input if it is positive, otherwise output zero. For positive inputs it never flattens out. The paper's Fig. 1 shows a small four-layer network with ReLUs reaching a target training error on a smaller image dataset (CIFAR-10) about six times faster than the same network with tanh.
Move the slider to change a unit's total input and compare how much of a small nudge each function passes on.
TakeawayReLUs keep the learning signal strong wherever they are active, which made training a large network fast enough to be practical.
Check yourself
- A tanh unit has an input of 3. Why does it learn slowly?
- tanh is almost flat there (slope about 0.01), so changes to its weights barely change its output and the error signal reaching it is tiny.
- What is the ReLU's weak spot?
- For negative inputs its slope is zero, so a unit that is always "off" gets no learning signal at all.
Idea · §3.2 · §3.5 · Fig. 2
Big network, two graphics cards
The network has eight learned layers: five convolutional layers (the sliding pattern detectors introduced by LeNet) and three ordinary fully connected layers, ending in 1,000 outputs, one per category. It has about 60 million adjustable numbers and 650,000 units.
That was too big for one graphics card of the day, which had 3 GB of memory. So the authors split the network down the middle: half the feature maps on each card, and the two halves only exchange information at certain layers. Graphics cards (GPUs) are built to do thousands of simple multiplications at once, which is exactly what a convolutional network needs. Training still took five to six days.
TakeawayGraphics cards made a network this size trainable in days rather than months, and that, as much as any new idea, is what made 2012 different.
Idea · §4.1 · §4.2
Two ways to stop memorising
A network with 60 million adjustable numbers and "only" 1.2 million photos is at risk of overfitting: learning the quirks of the training photos instead of what a dog actually looks like. The paper used two defences.
Data augmentation. Every photo is stored at 256 by 256 pixels, but the network is shown a random 224 by 224 piece of it, sometimes mirrored left to right. Each photo therefore turns into many slightly different training examples, all with the same correct label. The colours are also jittered slightly. Try it:
Dropout. During training, in the first two fully connected layers, each unit is switched off at random with probability one half, freshly for every example. No unit can rely on a particular partner being there, so each must learn something useful on its own. At test time all units are on, and their outputs are halved to keep the totals at the right scale.
TakeawayShowing many varied copies of each photo, and randomly removing units during training, let a huge network learn general patterns instead of memorising.
Check yourself
- Why mirror left to right but not upside down?
- A mirrored dog is still a realistic photo of a dog; an upside-down one mostly is not, so it would teach the network about images it will never see.
- Why halve the outputs at test time when using dropout?
- During training only about half the units were on, so the next layer is used to receiving about half the total. Halving keeps the scale the same when all are on.
Idea · §6.1 · §7 · Figs. 3–4
What it learned, and why depth mattered
The paper looks inside. The first layer's 96 detectors (Fig. 3) turned out to be edge detectors at various angles and spacings, plus blobs of colour, much like what neuroscientists find early in animal vision. Curiously, the detectors on one graphics card were mostly colourless and those on the other mostly coloured, which happened on every run.
Near the top, photos that the network represents similarly are similar in meaning, even when the pixels are very different: elephants in different poses end up near each other (Fig. 4). And removing any one of the middle convolutional layers made results about 2% worse, which the authors took as evidence that depth itself matters.
TakeawayThe network learned sensible visual features on its own, from edges upward, and its success made deep learning the default way to do computer vision.
Abstract, restated
A large deep convolutional network was trained to classify the 1.2 million high-resolution images of the ILSVRC-2010 contest into 1,000 classes, reaching top-1 and top-5 test error of 37.5% and 17.0%, considerably better than the previous state of the art. The network has 60 million parameters and 650,000 neurons, with five convolutional layers (some followed by max-pooling) and three fully connected layers ending in a 1,000-way softmax. Training used non-saturating neurons and an efficient GPU implementation of convolution; overfitting in the fully connected layers was reduced with dropout. A variant entered in ILSVRC-2012 achieved a winning top-5 test error of 15.3%, against 26.2% for the second-best entry. NeurIPS 2012.
Section · §2 · §3.5 · Fig. 2
Data and architecture
Data (§2). ILSVRC uses roughly 1.2 M training, 50,000 validation and 150,000 test images in 1,000 classes. Images are rescaled so the shorter side is 256, centre-cropped to 256 × 256, and the mean training pixel is subtracted; there is no other preprocessing, so the net trains on (centred) raw RGB.
| Layer | Kernels | Notes |
|---|---|---|
| conv1 | 96 × 11×11×3, stride 4 | on the 224×224×3 input; ReLU, LRN, max-pool |
| conv2 | 256 × 5×5×48 | sees only its own GPU's 48 maps; ReLU, LRN, max-pool |
| conv3 | 384 × 3×3×256 | connected to all conv2 maps across both GPUs |
| conv4 | 384 × 3×3×192 | own GPU only |
| conv5 | 256 × 3×3×192 | own GPU only; max-pool |
| fc6, fc7 | 4,096 each | dropout 0.5 |
| fc8 | 1,000 | softmax |
Spatial sizes follow (W − F + 2P)/S + 1. Note that with the stated 224 × 224 input, conv1 gives (224 − 11)/4 + 1 = 54.25, not an integer; the 55 × 55 output drawn in Fig. 2 requires an effective input of 227 (or extra padding), which is what most reimplementations use. Check it below.
Section · §3.1 · §3.3 · §3.4
ReLU, local response normalisation, overlapping pooling
ReLU (§3.1). f(x) = max(0, x). Following Nair and Hinton, the paper calls these non-saturating; on CIFAR-10 a four-layer convnet with ReLUs reached 25% training error six times faster than with tanh (Fig. 1). They note that Jarrett et al. had used |tanh| on the small Caltech-101 dataset, where the concern was overfitting; on ImageNet-scale data the gain that matters is faster fitting.
Local response normalisation (§3.3). ReLUs need no input normalisation to avoid saturation, but the authors found a form of lateral inhibition across channels helped. With aix,y the ReLU output of kernel i at position (x, y) and N kernels in the layer:
with k = 2, n = 5, α = 10−4, β = 0.75, chosen on validation data. It is applied after ReLU in conv1 and conv2, and reduced top-1 and top-5 error by 1.4% and 1.2%. Explore what it does to one spatial position's channel vector:
Overlapping pooling (§3.4). Max-pooling with window z = 3 and stride s = 2 (so windows overlap) instead of z = s = 2 reduced top-1/top-5 error by 0.4%/0.3%, and the authors observed it made the model slightly harder to overfit.
Check yourself
- Why is LRN called "brightness normalisation" rather than contrast normalisation in the paper?
- It divides by local channel energy without subtracting a mean activity, so it rescales but does not re-centre.
Section · §4
Reducing overfitting
Translations and reflections (§4.1). Random 224 × 224 patches and their horizontal reflections from the 256 × 256 images increase the training set by a factor of 2,048 (though the samples are highly dependent). At test time the network averages its softmax over ten patches: the four corners and centre, and their reflections. Without this test-time averaging, ILSVRC-2010 error is 39.0% top-1 and 18.3% top-5.
PCA colour augmentation. PCA is computed on the RGB values of all training pixels. To each pixel of a training image is added
where pi and λi are the eigenvectors and eigenvalues of the 3 × 3 RGB covariance, with α drawn once per image per presentation. It encodes the idea that object identity is invariant to changes in illumination intensity and colour, and it reduced top-1 error by over 1%.
Dropout (§4.2). Each hidden unit in fc6 and fc7 outputs zero with probability 0.5 during training; at test time all units are used and outputs are multiplied by 0.5, a cheap approximation to the geometric mean of the exponentially many dropout networks' predictions. Without dropout the network overfits substantially; with it, convergence takes roughly twice as many iterations.
Section · §5 · §6 · Tables 1–2
Training and results
SGD with batch 128, momentum 0.9 and weight decay 0.0005, which the authors found reduces training error, not just regularises:
Weights start from 𝒩(0, 0.012); biases are 1 in conv2, conv4, conv5 and the fully connected hidden layers (giving ReLUs positive inputs early) and 0 elsewhere. The learning rate starts at 0.01 and is divided by 10 when validation error stops improving, three times in total. About 90 epochs over 1.2 M images took five to six days on two NVIDIA GTX 580 3 GB GPUs.
The GPU split itself also helped: compared with a one-GPU net with half as many kernels in most convolutional layers, the two-GPU net cut top-1 and top-5 error by 1.7% and 1.2% (§3.2). The one-GPU net kept the last convolutional and fully connected layers full size, so it is more than half as large, which the authors note biases the comparison in its favour. On Fall 2009 ImageNet (10,184 categories), top-1/top-5 error was 67.4%/40.9% against the best published 78.1%/60.9%.
Section · §7 · lineage
Limitations and lineage
- Depth evidence is anecdotal. Removing any middle convolutional layer costs about 2% top-1 (§7), but there is no systematic depth study.
- Architecture shaped by hardware. The two-GPU split, with restricted connectivity, is a memory workaround; its colour-agnostic/colour-specific specialisation (Fig. 3) is a by-product.
- Hand-tuned pieces. LRN constants, overlapping pooling, and the learning-rate schedule (divided by hand when validation stalled) are empirical choices with small gains.
- No unsupervised pre-training. The authors say they expected it to help with more compute; in the event, supervised scaling won.
Check yourself
- Where do most of AlexNet's 60 M parameters live?
- In the fully connected layers: fc6 alone connects a 6×6×256 = 9,216-dimensional input to 4,096 units, about 37.7 M weights.
- Why initialise some biases to 1?
- So the ReLUs start with positive inputs and pass gradient, which the authors found accelerates the early stages of learning.
Reference
Glossary
- ILSVRC
- ImageNet Large Scale Visual Recognition Challenge; 1,000-class classification.
- Top-k error
- Fraction of test images whose true label is not among the model's k highest-scoring classes.
- ReLU
- max(0, x): a non-saturating activation.
- LRN
- Local response normalisation across adjacent channels.
- Overlapping pooling
- Pooling windows larger than the stride (z = 3, s = 2).
- Dropout
- Randomly zeroing hidden units in training to prevent co-adaptation.
- Data augmentation
- Label-preserving transformations that enlarge the training set.
- Weight decay
- An L2 penalty on weights, implemented as a shrinkage term in the update.