The paper in four sentences
Segmentation means labelling every pixel of an image ("cell" or "background", say), which needs both the big picture (what is this?) and precise detail (exactly where is the edge?).
U-Net shrinks the image step by step to gather context, then grows it back step by step, and at each size it copies across the detailed features from the shrinking side so edges can be placed precisely.
Because labelled medical images are scarce, it trains on heavily warped copies of a few images, and it weights the loss so the thin gaps between touching cells count extra.
Trained on just a few dozen images, it beat the previous best method on a nerve-tissue challenge, won the 2015 cell tracking challenge by a wide margin, and segments a 512 by 512 image in under a second.
Idea · §1
Label every pixel, from very few examples
Networks like AlexNet answer one question per image: "is this a cat?". Biologists need something different. Given a microscope image, they want to know which pixels belong to which cell, so they can count cells, measure them, and track them over time. This is called segmentation: the output is a label for every single pixel.
Two things make it hard. First, there are very few labelled images, because outlining cells by hand takes experts hours. Second, the task needs two kinds of knowledge at once: context (is this blob a cell or a speck of dirt?) and precise location (exactly which pixel is the cell's edge?).
The best previous method slid a small window across the image and classified the centre pixel of each window, one by one. It worked, but it was slow (every pixel needs its own run of the network, and neighbouring windows repeat most of the work), and there was a trade-off: bigger windows gave more context but blurrier edges.
TakeawaySegmentation needs context and precision at the same time, and in biomedicine it has to be learned from a handful of examples.
Idea · §2 · Fig. 1
Squeeze, expand, and pass the details across
U-Net has two halves. The contracting path (left side of the U) works like an ordinary image-recognition network: detect patterns, then halve the image size, four times over. Each halving lets later layers see a wider area, so they understand context, but blurs where exactly things are.
The expanding path (right side) does the reverse: double the size, four times, back up to a detailed map. On its own, it could only produce a blurry, blocky outline, because the precise detail was thrown away on the way down. So at every size, U-Net copies the features from the matching level of the left side straight across and joins them on. These are the skip connections, the horizontal bars of the U.
The toy below makes the trade-off visible. A noisy image contains three cells. A purely "fine" detector looks at each pixel on its own: sharp but fooled by noise. A purely "coarse" one averages big blocks: robust but blocky. Combining them, as the skip connections let U-Net do, gets the best of both.
TakeawayThe bottom of the U knows what is in the image; the skip connections remember where. Together they give sharp, sensible outlines.
Check yourself
- What would the output look like without the skip connections?
- Roughly right but blurry or blocky around the edges, because the expanding path would have only the coarse, low-resolution features to work from.
- Why not skip the shrinking altogether and work at full size throughout?
- Each layer would then see only a tiny neighbourhood, so the network would lack context: it could not tell a cell from noise that looks locally similar.
Idea · §2 · Fig. 2
Big images, one tile at a time
U-Net only ever uses pixels it actually has: its layers never pad the image with made-up zeros at the edges. The price is that the output is smaller than the input: give it 572 by 572 pixels and it labels the central 388 by 388. Every output pixel was computed from real surrounding context.
To segment a large image, it is cut into overlapping tiles, each processed separately, and the labelled centres fit together seamlessly. At the outer edge of the whole image, where there is no real context to give, the missing area is filled with a mirror image of the nearby pixels. Because the network can work on any image size, it can also process a whole 512 by 512 image in under a second on a graphics card.
TakeawayTiles with overlapping borders let U-Net handle images of any size, limited only by memory, with no visible seams.
Idea · §3.1
Turn thirty images into thousands
With only about 30 labelled images, a network would quickly memorise them. The fix is data augmentation: show it altered copies, with the labels altered the same way. U-Net's key alteration is elastic deformation: the image is gently warped as if printed on a rubber sheet and stretched unevenly. Real tissue really does vary like this from sample to sample, so the warped copies look like plausible new images.
The warp is made by pushing the points of a coarse 3 by 3 grid in random directions and smoothly spreading those pushes across the whole image. Try a few:
TakeawayRealistic, label-preserving warps let a network learn from a tiny dataset without memorising it.
Check yourself
- Why must the same warp be applied to the label map?
- Otherwise the labels would no longer line up with the warped image, and the network would be taught wrong answers.
Idea · §3 · Fig. 3 · §4
Mind the gap between touching cells
Cells often touch. If the network labels the whole clump as "cell", you cannot count or measure them separately. The thin strip of background between two cells is only a pixel or two wide, so it hardly affects an ordinary error score. U-Net fixes this by giving every pixel a weight in the training error: background pixels squeezed between two nearby cells get a much larger weight, so mistakes there are punished heavily. (The In depth tab lets you play with the exact formula.)
Results. On a challenge to trace nerve-cell boundaries in electron-microscope images (30 training images), U-Net beat the previous best method, the sliding-window network. In the 2015 cell tracking challenge it scored 92% overlap with the true outlines on one type of light-microscope image, against 83% for the runner-up, and 77.5% against 46% on another. Training took about 10 hours on a single graphics card.
TakeawayWeighting the loss towards hard, thin boundaries is what lets U-Net separate cells that touch.
Abstract, restated
Successful training of deep networks is commonly thought to need many thousands of annotated samples. The paper presents a network and training strategy that rely on strong data augmentation to use the available annotated samples more efficiently. The architecture has a contracting path to capture context and a symmetric expanding path that enables precise localisation. It can be trained end to end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmenting neuronal structures in electron-microscopic stacks; trained on transmitted-light microscopy images (phase contrast and DIC), it won the ISBI cell tracking challenge 2015 in these categories by a large margin. Segmenting a 512 × 512 image takes less than a second on a recent GPU. MICCAI 2015.
Section · §1 · §2 · Fig. 1
Architecture
Starting point. The design builds on the fully convolutional network (FCN) of Long, Shelhamer and Darrell: supplement a contracting network with layers where pooling is replaced by upsampling, and combine high-resolution features from the contracting path with the upsampled output. U-Net's modification is a large number of feature channels in the upsampling part, so context can propagate to higher-resolution layers, making the expanding path roughly symmetric to the contracting one. There are no fully connected layers.
Contracting path. Repeated application of two 3 × 3 unpadded convolutions, each followed by ReLU, then 2 × 2 max-pooling with stride 2. Channels double at each downsampling: 64, 128, 256, 512, 1024. Expanding path. Each step: a 2 × 2 "up-convolution" that halves the channels, concatenation with the correspondingly cropped feature map from the contracting path (cropping is needed because unpadded convolutions lose border pixels), then two 3 × 3 convolutions with ReLU. A final 1 × 1 convolution maps the 64-component features to the classes. In total, 23 convolutional layers.
Valid convolutions and sizes. Each 3 × 3 unpadded convolution removes 2 pixels in each dimension, so the output map contains only pixels for which the full context is available. To tile seamlessly the input size must be chosen so that every 2 × 2 pooling sees an even-sized map. Check the arithmetic:
Check yourself
- Why are skip-connection features cropped before concatenation?
- With unpadded convolutions the contracting-path maps are larger than the upsampled maps at the same level (e.g. 64 vs 56 at the deepest skip for a 572 input), so their centres are cropped to match.
Section · §2 · Fig. 2
Overlap-tile strategy
Because the network uses only valid parts of each convolution, predicting the segmentation of a tile (yellow area in the paper's Fig. 2) requires the image data in a larger surrounding region (blue area). With input 572 and output 388, the extra context is 92 pixels on each side. For pixels at the border of the whole image, the missing context is extrapolated by mirroring the input image. Tiling is important for applying the network to large images, since otherwise the resolution would be limited by GPU memory.
This is also why §3 favours large input tiles over a large batch: the batch is reduced to a single image, and a high momentum (0.99) is used so that a large number of previously seen training samples determine each update.
Section · §3 · Eqs. 1–2 · Fig. 3
Loss, weight map and initialisation
The energy is a pixel-wise softmax over the final feature map combined with a weighted cross-entropy. With ak(x) the activation in channel k at pixel x, pk(x) = exp(ak(x)) / Σk′ exp(ak′(x)), and ℓ(x) the true label:
(As printed, Eq. 1 is a log-likelihood to be maximised; equivalently, the loss minimised is its negative.) The weight map is precomputed from each ground-truth segmentation:
Here wc balances class frequencies, d1 is the distance to the border of the nearest cell and d2 to the border of the second nearest. The paper sets w0 = 10 and σ ≈ 5 pixels. The morphological separation of touching cells is computed beforehand, so the network is forced to learn the small separation borders.
Initialisation. Weights are drawn from a Gaussian with standard deviation √(2/N), N the number of incoming nodes of one neuron (e.g. 9 · 64 = 576 for a 3 × 3 convolution with 64 input channels), so that each feature map has approximately unit variance; important when alternating convolution and ReLU in deep networks.
Check yourself
- Why use d1 + d2 rather than only d1?
- A pixel near a single isolated cell has small d1 but large d2. Only pixels close to two different cells, i.e. in a separating gap, have a small sum.
Section · §3.1 · §4 · Tables 1–2
Augmentation and experiments
Augmentation (§3.1). Shift and rotation invariance plus robustness to deformations and grey-value variations are the main needs with microscopy. Random elastic deformations are described as the key concept for training with very few annotated images: displacement vectors are sampled on a coarse 3 × 3 grid from a Gaussian with 10-pixel standard deviation, and per-pixel displacements are computed by bicubic interpolation. Drop-out layers at the end of the contracting path perform further implicit augmentation.
EM segmentation (ISBI 2012). 30 training images of 512 × 512 (serial-section transmission EM of the Drosophila first-instar larva ventral nerve cord), with full annotation of cells and membranes; evaluation by thresholding the predicted membrane probability at 10 levels and computing warping, Rand and pixel errors. Averaged over 7 rotated versions of the input and with no further pre- or post-processing, U-Net achieves warping error 0.000353 (a new best at the time) and Rand error 0.0382, against 0.000420 and 0.0504 for the sliding-window network of Cireşan et al. (Table 1). Some challenge entries ranked higher on Rand error, using highly data-set-specific post-processing applied to the probability map of Cireşan et al.
| ISBI cell tracking 2015 (Table 2) | PhC-U373 | DIC-HeLa |
|---|---|---|
| U-Net (IOU) | 0.9203 | 0.7756 |
| Second best | 0.83 | 0.46 |
PhC-U373 is glioblastoma-astrocytoma cells under phase-contrast microscopy, DIC-HeLa is HeLa cells under differential interference contrast; both come with small, partially annotated training sets. Training time is about 10 hours on an NVIDIA Titan GPU (6 GB), in Caffe.
Section · §5 · lineage
Limitations and lineage
- Narrow evaluation. Three 2D datasets, all transmitted-light or EM microscopy; no ablation isolating skip connections, weight map or augmentation.
- Engineering choices of the day. Valid convolutions and overlap-tiling were motivated by GPU memory; most later implementations use padded convolutions so input and output sizes match, with batch normalisation added.
- Instance separation via loss weighting. Touching objects are separated only through a semantic boundary class and weighting; true instance segmentation came later with dedicated heads (e.g. Mask R-CNN) or learned embeddings.
Reference
Glossary
- Segmentation
- Assigning a class label to every pixel.
- Contracting path
- Convolutions and pooling that reduce resolution and increase channels.
- Expanding path
- Up-convolutions and convolutions that restore resolution.
- Up-convolution
- A learned 2 × 2 upsampling (transposed convolution) that halves channels.
- Valid convolution
- Unpadded convolution; output shrinks by kernel size − 1.
- Overlap-tile
- Predicting large images tile by tile with context borders, mirroring at image edges.
- Weight map
- Per-pixel loss weights emphasising separating borders (Eq. 2).
- IOU
- Intersection over union of predicted and true regions.
- Warping error
- A topology-aware segmentation metric counting splits and mergers.