The paper in four sentences
Making a network deeper should never hurt, because the extra layers could simply pass their input through unchanged; yet in practice a 56-layer network did worse than a 20-layer one, even on the examples it was trained on.
The fix is a shortcut around every couple of layers: the block's output is its input plus whatever the layers add, so the layers only have to learn a correction (a "residual").
If a block has nothing useful to add, it can learn to add almost nothing, which is far easier than learning to copy its input exactly.
With these shortcuts, accuracy kept improving up to 152 layers, and an ensemble got 3.57% top-5 error on ImageNet, winning the 2015 competition.
Idea · §1 · Fig. 1
The puzzle: deeper got worse
After AlexNet, the lesson seemed to be "deeper is better": more layers can build more complex features out of simpler ones. By 2015 the best networks had around 20 to 30 layers. But going further ran into a wall.
The authors trained a 20-layer and a 56-layer network on a small photo dataset (CIFAR-10). The deeper one made more mistakes, and not only on new photos: it also did worse on the very photos it was trained on (Fig. 1). That rules out the usual suspect, overfitting, where a model memorises its training data but fails on new data. Here the deeper model could not even fit its training data as well. The paper calls this the degradation problem.
This is strange, because the deeper network could in principle copy the shallow one: use the same 20 layers, and make the other 36 pass their input straight through. So a solution at least as good exists. The training method just doesn't find it.
TakeawayThe trouble with very deep networks was not memorising too much; it was that training could not find good settings at all.
Check yourself
- How do we know the 56-layer network's problem is not overfitting?
- Its error on the training data itself is higher than the 20-layer network's. Overfitting would show low training error but high test error.
Idea · §3.1–3.2 · Fig. 2
Learn the correction, not the whole thing
The fix is almost embarrassingly simple. Take a block of two layers. Normally its output is whatever the layers compute. In a residual block, a shortcut carries the block's input around the layers and adds it back on at the end:
output = input + (what the layers compute)
Now the layers only have to learn the difference between what comes in and what should go out, called the residual. If the best thing a block can do is pass its input along unchanged, the layers just need to output zeros, which is easy: small weights already give nearly zero. Without the shortcut, the layers would have to learn to reproduce their input exactly, which is surprisingly hard for a stack of layers with nonlinear switches in between.
The figure shows what happens to a simple input signal (a wave) as it passes through many layers of a network with random starting weights, with and without shortcuts.
TakeawayA shortcut makes "do nothing" the easy default, so extra layers can't make things much worse, and can make them better.
Check yourself
- A block's ideal output is its input plus a small tweak. What do the layers in a residual block learn?
- Only the small tweak. The shortcut supplies the input itself.
- Does the shortcut add extra adjustable numbers?
- No. When input and output have the same size it is a plain addition, with no weights at all.
Idea · §4.1 · Table 2 · Fig. 4
Shortcuts make depth pay off
The authors compared networks that were identical except for the shortcuts. Without them ("plain" networks), the 34-layer version was worse than the 18-layer one on ImageNet. With them, the 34-layer ResNet was clearly better than the 18-layer ResNet, and much better than plain-34: 25.0% versus 28.5% top-1 error (Table 2). The degradation problem had flipped around.
You can rerun a tiny version of this experiment below. Two networks of the same depth learn to fit a wiggly curve: one plain, one with shortcuts around every layer. Try a few depths.
TakeawayWith shortcuts, deeper networks finally behaved as theory promised: more layers, lower error.
Check yourself
- Why is "same layers, with and without shortcuts" such a convincing comparison?
- Everything else (number of weights, computation, training recipe) is held equal, so any difference must come from the shortcuts.
Idea · §3.3 · §4.1 · Table 1 · Fig. 5
Going to 152 layers
To go much deeper without the cost exploding, the deeper ResNets use a bottleneck block: first squeeze the number of channels down by four times with a cheap layer, do the expensive 3 by 3 processing on the slim version, then expand back. A 152-layer ResNet built this way needs fewer calculations per image (11.3 billion) than the 19-layer VGG network from the year before (19.6 billion).
Accuracy kept improving with depth: 50, 101, then 152 layers each did better. A combination of ResNets reached 3.57% top-5 error on the ImageNet test set and won the 2015 competition. Three years earlier, AlexNet had won with 15.3%.
TakeawayShortcuts plus cheap bottleneck blocks turned "very deep" from a liability into the winning design.
Idea · §4.3 · legacy
A shortcut that ended up everywhere
The same networks, used as the backbone for finding and outlining objects, gave a 28% relative improvement on the COCO object detection benchmark just by swapping in the deeper ResNet (§4.3). The team also won the 2015 ImageNet detection and localisation tasks and the COCO detection and segmentation tasks.
The deeper legacy is the shortcut itself. Almost every large network built since adds each layer's output back onto its input. The Transformer behind today's language models does exactly this around every attention and feed-forward layer, and so do Vision Transformers and the U-Net-style networks inside image generators.
Takeaway"Add the input back" is one of the most reused ideas in deep learning.
Abstract, restated
Deeper networks are harder to train. The paper reformulates layers as learning residual functions with reference to their inputs, instead of unreferenced functions, and gives empirical evidence that such residual networks are easier to optimise and gain accuracy from considerably increased depth. On ImageNet, residual nets of up to 152 layers (8× deeper than VGG, yet of lower complexity) are evaluated; an ensemble achieves 3.57% top-5 error on the test set and won ILSVRC 2015 classification. Analyses on CIFAR-10 cover 100- and 1,000-layer nets. The representations give a 28% relative improvement on COCO detection and underpinned first places in ImageNet detection and localisation and COCO detection and segmentation. CVPR 2016 (arXiv December 2015).
Section · §1 · §3.1–3.2 · Eqs. 1–2
Degradation and the residual reformulation
Degradation (§1). With depth, accuracy saturates and then degrades rapidly; this is "not caused by overfitting", since training error rises too (Fig. 1, Fig. 4 left). Vanishing/exploding gradients at initialisation are largely handled by normalised initialisation and batch normalisation, so the networks do start converging; the issue is what they converge to.
Construction argument. Given a shallower model, a deeper one built by adding identity layers has no higher training error. Hence degradation indicates that solvers have difficulty approximating identity mappings with stacks of nonlinear layers. If the desired mapping is H(x), let the stack fit F(x) := H(x) − x. When identity is (near-)optimal, pushing F to zero is easier than fitting identity; the paper's later response analysis supports that learned residuals are small.
For a two-layer block F = W2 σ(W1x) with σ = ReLU (biases omitted), and a second ReLU is applied after the addition. Eq. 1's shortcut adds neither parameters nor computation (apart from an element-wise add). Eq. 2's linear projection Ws is used only to match dimensions. F must span at least two layers: with one layer, Eq. 1 is just a linear layer y = W1x + x, for which the authors saw no advantage.
Check yourself
- Why does the paper insist F has at least two layers?
- With a single linear layer, Wx + x = (W + I)x is just a reparameterised linear layer, with no nonlinearity inside the residual branch.
Section · §3.3 · Table 1 · Fig. 3 · Fig. 5
Architectures and the bottleneck
Plain baseline. Inspired by VGG: mostly 3×3 convolutions; same output size implies same number of filters; halving the feature map doubles the filters; downsampling by stride-2 convolutions; global average pooling and a 1000-way fully connected layer. The 34-layer plain net has 3.6 billion FLOPs (multiply-adds), 18% of VGG-19's 19.6 billion.
Shortcuts across dimension changes. (A) identity with zero-padding for extra channels, parameter-free; (B) 1×1 projection shortcuts only where dimensions increase; (C) projections everywhere. On ImageNet B is slightly better than A and C marginally better than B (Table 3), small enough that projections are judged "not essential"; C is not used further to save memory and time.
Stages (Table 1). Basic blocks for 18/34 ([2,2,2,2] and [3,4,6,3]); bottleneck blocks for 50/101/152 ([3,4,6,3], [3,4,23,3], [3,8,36,3]) across conv2_x to conv5_x at widths 64, 128, 256, 512 (×4 output in bottlenecks). FLOPs: 1.8, 3.6, 3.8, 7.6 and 11.3 × 109. Parameter-free identity shortcuts matter most for bottlenecks: replacing them with projections would roughly double the cost, since the shortcut connects the two high-dimensional ends.
Section · §3.4 · §4.1 · Tables 2–5
ImageNet training and results
Recipe (§3.4). Shorter side resized to a random value in [256, 480] for scale augmentation; random 224 × 224 crop or its flip, per-pixel mean subtracted; AlexNet-style colour augmentation. Batch normalisation after each convolution and before activation; He initialisation; SGD, batch 256, learning rate 0.1 divided by 10 when error plateaus, up to 60 × 104 iterations, weight decay 0.0001, momentum 0.9, no dropout. Testing uses 10-crop for comparisons and fully convolutional multi-scale evaluation for the best results.
Why the plain net fails (§4.1). Because of BN, forward signals have non-zero variance and backward gradients have healthy norms, so the authors argue vanishing gradients are unlikely to be the cause. Plain-34 still reaches competitive accuracy, so the solver works to some degree; they conjecture deep plain nets may have exponentially low convergence rates, and leave the reason open.
Single-model ResNet-152 reaches 19.38% top-1 and 4.49% top-5 on validation (Table 4); an ensemble of six models of different depths (two of them 152-layer) gives 3.57% top-5 on the test set (Table 5).
Check yourself
- Plain-18 and ResNet-18 have nearly the same error (27.94 vs 27.88). What does ResNet-18 still gain?
- Faster convergence early in training (§4.1): when the net is not overly deep, SGD can find good solutions for the plain net too, but the residual one gets there sooner.
Section · §4.2 · Table 6 · Figs. 6–7
CIFAR-10 and what the layers learn
A deliberately simple CIFAR-10 architecture: a 3×3 conv, then three stages of 2n layers on 32×32, 16×16 and 8×8 maps with 16, 32, 64 filters, global average pooling and a 10-way classifier, for 6n + 2 layers; option A shortcuts. Plain nets degrade with depth (Fig. 6 left); ResNets improve up to 110 layers (6.43%).
Layer responses (Fig. 7). The standard deviation of each 3×3 layer's output (after BN, before the addition) is generally smaller for ResNets than for plain nets, and smaller still for deeper ResNets: individual blocks modify the signal less. This supports the premise that residual functions are generally closer to zero than non-residual ones.
1,202 layers. With n = 200 the network trains without optimisation difficulty (training error below 0.1%), but test error is 7.93%, worse than the 110-layer net. The authors attribute this to overfitting: 19.4 M parameters is large for a small dataset, and they did not use strong regularisers such as maxout or dropout.
Section · §4.3 · lineage
Detection, limitations and lineage
Detection (§4.3). Replacing VGG-16 with ResNet-101 in Faster R-CNN raises COCO mAP@[.5, .95] from 21.2% to 27.2%, the 6.0-point (28% relative) gain quoted in the abstract; [email protected] goes from 41.5% to 48.4%.
Limitations. The paper is empirical: it shows that residual nets optimise better but does not explain why deep plain nets fail, and the 1,202-layer result shows that depth alone, without regularisation, stops paying. Post-addition ReLU means the shortcut path is not a pure identity end to end.
Check yourself
- In a pre-activation ResNet, what is the gradient of the loss with respect to an early block's output?
- It contains a term passed straight through the identity path from the top, plus the sum of the branch contributions, so it cannot vanish through the shortcut path even if every branch's gradient is small.
Reference
Glossary
- Degradation
- Higher training error for deeper plain networks.
- Residual
- F(x) = H(x) − x, what a block adds to its input.
- Identity shortcut
- A parameter-free connection adding a block's input to its output.
- Projection shortcut
- A 1×1 convolution on the shortcut to match dimensions (Ws).
- Bottleneck
- 1×1 reduce, 3×3, 1×1 expand block used in ResNet-50/101/152.
- 10-crop
- Averaging predictions over four corners, centre and their flips.
- FLOPs
- Here, multiply-adds per forward pass of one image.
- Pre-activation
- Block ordering BN → ReLU → conv, leaving the shortcut path a pure identity.