Study guide · Computer vision · 2015

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · CVPR 2016

If deeper networks are more powerful, why does simply stacking more layers make them worse, even on the training data? This paper's answer is to let each block of layers learn only a correction to its input, carried forward by a shortcut. With that change, networks of 152 layers train easily and won the 2015 ImageNet competition, and the same shortcut now sits inside nearly every large model, transformers included.

The paper in four sentences

Making a network deeper should never hurt, because the extra layers could simply pass their input through unchanged; yet in practice a 56-layer network did worse than a 20-layer one, even on the examples it was trained on.

The fix is a shortcut around every couple of layers: the block's output is its input plus whatever the layers add, so the layers only have to learn a correction (a "residual").

If a block has nothing useful to add, it can learn to add almost nothing, which is far easier than learning to copy its input exactly.

With these shortcuts, accuracy kept improving up to 152 layers, and an ensemble got 3.57% top-5 error on ImageNet, winning the 2015 competition.

Idea · §1 · Fig. 1

The puzzle: deeper got worse

After AlexNet, the lesson seemed to be "deeper is better": more layers can build more complex features out of simpler ones. By 2015 the best networks had around 20 to 30 layers. But going further ran into a wall.

The authors trained a 20-layer and a 56-layer network on a small photo dataset (CIFAR-10). The deeper one made more mistakes, and not only on new photos: it also did worse on the very photos it was trained on (Fig. 1). That rules out the usual suspect, overfitting, where a model memorises its training data but fails on new data. Here the deeper model could not even fit its training data as well. The paper calls this the degradation problem.

This is strange, because the deeper network could in principle copy the shallow one: use the same 20 layers, and make the other 36 pass their input straight through. So a solution at least as good exists. The training method just doesn't find it.

Everyday pictureImagine passing a message along a line of 56 people, each allowed to improve the wording. In principle, anyone with nothing to add can repeat it exactly. In practice, everyone changes it a little, and by the end it is garbled. Twenty people do better than fifty-six.

TakeawayThe trouble with very deep networks was not memorising too much; it was that training could not find good settings at all.

Check yourself
How do we know the 56-layer network's problem is not overfitting?
Its error on the training data itself is higher than the 20-layer network's. Overfitting would show low training error but high test error.

Idea · §3.1–3.2 · Fig. 2

Learn the correction, not the whole thing

The fix is almost embarrassingly simple. Take a block of two layers. Normally its output is whatever the layers compute. In a residual block, a shortcut carries the block's input around the layers and adds it back on at the end:

output = input + (what the layers compute)

Now the layers only have to learn the difference between what comes in and what should go out, called the residual. If the best thing a block can do is pass its input along unchanged, the layers just need to output zeros, which is easy: small weights already give nearly zero. Without the shortcut, the layers would have to learn to reproduce their input exactly, which is surprisingly hard for a stack of layers with nonlinear switches in between.

The figure shows what happens to a simple input signal (a wave) as it passes through many layers of a network with random starting weights, with and without shortcuts.

Each panel plots one unit (of six) after the chosen number of layers, as the network's input slides from −1 to 1. Grey dashed: the same unit after the first layer. Note the readout: in the plain stack the variation usually collapses towards a near-constant output as layers are added, so information about the input is lost; with shortcuts the variation is kept (and reshaped) because every block adds to what came in. Untrained toy with tanh units; illustrative of the starting point, not a trained ResNet.
Everyday pictureEditing a document with "track changes" is easier than retyping it from memory. Each block of a ResNet gets the original handed to it and only writes in its edits.

TakeawayA shortcut makes "do nothing" the easy default, so extra layers can't make things much worse, and can make them better.

Check yourself
A block's ideal output is its input plus a small tweak. What do the layers in a residual block learn?
Only the small tweak. The shortcut supplies the input itself.
Does the shortcut add extra adjustable numbers?
No. When input and output have the same size it is a plain addition, with no weights at all.

Idea · §4.1 · Table 2 · Fig. 4

Shortcuts make depth pay off

The authors compared networks that were identical except for the shortcuts. Without them ("plain" networks), the 34-layer version was worse than the 18-layer one on ImageNet. With them, the 34-layer ResNet was clearly better than the 18-layer ResNet, and much better than plain-34: 25.0% versus 28.5% top-1 error (Table 2). The degradation problem had flipped around.

You can rerun a tiny version of this experiment below. Two networks of the same depth learn to fit a wiggly curve: one plain, one with shortcuts around every layer. Try a few depths.

Left: the target curve (dots) and each network's current fit. Right: training error over 900 steps (log scale). Both use six tanh units per layer, the same random start, momentum and step size, and gradient clipping to keep training stable; real ResNets use batch normalisation instead. At depth 2–3 the two are similar; at 10 and beyond the residual network fits far better while the plain one stalls. A toy re-run of the idea, not the paper's experiment.

TakeawayWith shortcuts, deeper networks finally behaved as theory promised: more layers, lower error.

Check yourself
Why is "same layers, with and without shortcuts" such a convincing comparison?
Everything else (number of weights, computation, training recipe) is held equal, so any difference must come from the shortcuts.

Idea · §3.3 · §4.1 · Table 1 · Fig. 5

Going to 152 layers

To go much deeper without the cost exploding, the deeper ResNets use a bottleneck block: first squeeze the number of channels down by four times with a cheap layer, do the expensive 3 by 3 processing on the slim version, then expand back. A 152-layer ResNet built this way needs fewer calculations per image (11.3 billion) than the 19-layer VGG network from the year before (19.6 billion).

Accuracy kept improving with depth: 50, 101, then 152 layers each did better. A combination of ResNets reached 3.57% top-5 error on the ImageNet test set and won the 2015 competition. Three years earlier, AlexNet had won with 15.3%.

basic block (ResNet-18/34) 3×3, 64 3×3, 64 +identity bottleneck (ResNet-50/101/152) 1×1, 64 3×3, 64 1×1, 256 +identity each layer: convolution, batch normalisation, ReLU (the last ReLU comes after the addition)
Redrawn after Fig. 2 and Fig. 5. The bottleneck block takes 256 channels in, works at 64, and returns 256.

TakeawayShortcuts plus cheap bottleneck blocks turned "very deep" from a liability into the winning design.

Idea · §4.3 · legacy

A shortcut that ended up everywhere

The same networks, used as the backbone for finding and outlining objects, gave a 28% relative improvement on the COCO object detection benchmark just by swapping in the deeper ResNet (§4.3). The team also won the 2015 ImageNet detection and localisation tasks and the COCO detection and segmentation tasks.

The deeper legacy is the shortcut itself. Almost every large network built since adds each layer's output back onto its input. The Transformer behind today's language models does exactly this around every attention and feed-forward layer, and so do Vision Transformers and the U-Net-style networks inside image generators.

Later refinementA follow-up by the same authors in 2016 ("Identity Mappings in Deep Residual Networks") moved the normalisation and activation inside the branch so the shortcut path is a completely clean addition from start to end. That "pre-activation" version trained a 1,001-layer network well. Transformers today use the same idea under the name "pre-norm".

Takeaway"Add the input back" is one of the most reused ideas in deep learning.

Follow the idea

Built on: AlexNet (2012)

Led to: ViT (2020)