Study guide · Computer vision · 2015

U-Net: Convolutional Networks for Biomedical Image Segmentation

Olaf Ronneberger, Philipp Fischer, Thomas Brox · MICCAI 2015

How do you teach a network to outline every cell in a microscope image, pixel by pixel, when you only have a few dozen hand-labelled examples? This paper's answer is a U-shaped network that first squeezes the image down to understand what is there, then expands it back to full size, with direct links that hand over the fine detail lost on the way down. Its shape became one of the most widely used designs in imaging, from medical scans to the image generators of today.

The paper in four sentences

Segmentation means labelling every pixel of an image ("cell" or "background", say), which needs both the big picture (what is this?) and precise detail (exactly where is the edge?).

U-Net shrinks the image step by step to gather context, then grows it back step by step, and at each size it copies across the detailed features from the shrinking side so edges can be placed precisely.

Because labelled medical images are scarce, it trains on heavily warped copies of a few images, and it weights the loss so the thin gaps between touching cells count extra.

Trained on just a few dozen images, it beat the previous best method on a nerve-tissue challenge, won the 2015 cell tracking challenge by a wide margin, and segments a 512 by 512 image in under a second.

Idea · §1

Label every pixel, from very few examples

Networks like AlexNet answer one question per image: "is this a cat?". Biologists need something different. Given a microscope image, they want to know which pixels belong to which cell, so they can count cells, measure them, and track them over time. This is called segmentation: the output is a label for every single pixel.

Two things make it hard. First, there are very few labelled images, because outlining cells by hand takes experts hours. Second, the task needs two kinds of knowledge at once: context (is this blob a cell or a speck of dirt?) and precise location (exactly which pixel is the cell's edge?).

The best previous method slid a small window across the image and classified the centre pixel of each window, one by one. It worked, but it was slow (every pixel needs its own run of the network, and neighbouring windows repeat most of the work), and there was a trade-off: bigger windows gave more context but blurrier edges.

Everyday pictureColouring in a map by hand, you glance at the whole region to know it is a lake, then look closely at the shoreline to colour exactly up to the edge. A segmentation network needs both glances.

TakeawaySegmentation needs context and precision at the same time, and in biomedicine it has to be learned from a handful of examples.

Idea · §2 · Fig. 1

Squeeze, expand, and pass the details across

U-Net has two halves. The contracting path (left side of the U) works like an ordinary image-recognition network: detect patterns, then halve the image size, four times over. Each halving lets later layers see a wider area, so they understand context, but blurs where exactly things are.

The expanding path (right side) does the reverse: double the size, four times, back up to a detailed map. On its own, it could only produce a blurry, blocky outline, because the precise detail was thrown away on the way down. So at every size, U-Net copies the features from the matching level of the left side straight across and joins them on. These are the skip connections, the horizontal bars of the U.

The toy below makes the trade-off visible. A noisy image contains three cells. A purely "fine" detector looks at each pixel on its own: sharp but fooled by noise. A purely "coarse" one averages big blocks: robust but blocky. Combining them, as the skip connections let U-Net do, gets the best of both.

Each map is 48 × 48 pixels. Fine: threshold each pixel on its own. Coarse: average over big blocks, threshold, then blow back up. Combined: average the evidence from each pixel's 3 × 3 neighbourhood with the smoothly enlarged coarse evidence, then threshold. Overlap with the true cells is measured as IoU (1.00 is perfect). A hand-built illustration of why the U needs both paths; U-Net learns its own combination.
64 128 256 512 1024 512 256 128 64 copy and join (skip connections) down: detect,then halve up: double,then detect
Redrawn after Fig. 1. Numbers are feature channels at each level: they double each time the image is halved, and halve again on the way up.

TakeawayThe bottom of the U knows what is in the image; the skip connections remember where. Together they give sharp, sensible outlines.

Check yourself
What would the output look like without the skip connections?
Roughly right but blurry or blocky around the edges, because the expanding path would have only the coarse, low-resolution features to work from.
Why not skip the shrinking altogether and work at full size throughout?
Each layer would then see only a tiny neighbourhood, so the network would lack context: it could not tell a cell from noise that looks locally similar.

Idea · §2 · Fig. 2

Big images, one tile at a time

U-Net only ever uses pixels it actually has: its layers never pad the image with made-up zeros at the edges. The price is that the output is smaller than the input: give it 572 by 572 pixels and it labels the central 388 by 388. Every output pixel was computed from real surrounding context.

To segment a large image, it is cut into overlapping tiles, each processed separately, and the labelled centres fit together seamlessly. At the outer edge of the whole image, where there is no real context to give, the missing area is filled with a mirror image of the nearby pixels. Because the network can work on any image size, it can also process a whole 512 by 512 image in under a second on a graphics card.

TakeawayTiles with overlapping borders let U-Net handle images of any size, limited only by memory, with no visible seams.

Idea · §3.1

Turn thirty images into thousands

With only about 30 labelled images, a network would quickly memorise them. The fix is data augmentation: show it altered copies, with the labels altered the same way. U-Net's key alteration is elastic deformation: the image is gently warped as if printed on a rubber sheet and stretched unevenly. Real tissue really does vary like this from sample to sample, so the warped copies look like plausible new images.

The warp is made by pushing the points of a coarse 3 by 3 grid in random directions and smoothly spreading those pushes across the whole image. Try a few:

Left: original. Right: warped copy. Amber dots are the 3 × 3 control grid; each is pushed by a random amount whose typical size is the slider value (standard deviation, in pixels of a 512-pixel image; the paper uses 10). The pushes are spread smoothly over every pixel (bilinear here; the paper uses bicubic interpolation). The same warp is applied to the label map, so labels still match.
Everyday pictureTeaching someone to recognise handwriting, you would show the same words written by many people, slightly squashed, slanted and stretched. Elastic deformation fakes that variety from a single example.

TakeawayRealistic, label-preserving warps let a network learn from a tiny dataset without memorising it.

Check yourself
Why must the same warp be applied to the label map?
Otherwise the labels would no longer line up with the warped image, and the network would be taught wrong answers.

Idea · §3 · Fig. 3 · §4

Mind the gap between touching cells

Cells often touch. If the network labels the whole clump as "cell", you cannot count or measure them separately. The thin strip of background between two cells is only a pixel or two wide, so it hardly affects an ordinary error score. U-Net fixes this by giving every pixel a weight in the training error: background pixels squeezed between two nearby cells get a much larger weight, so mistakes there are punished heavily. (The In depth tab lets you play with the exact formula.)

Results. On a challenge to trace nerve-cell boundaries in electron-microscope images (30 training images), U-Net beat the previous best method, the sliding-window network. In the 2015 cell tracking challenge it scored 92% overlap with the true outlines on one type of light-microscope image, against 83% for the runner-up, and 77.5% against 46% on another. Training took about 10 hours on a single graphics card.

Later refinementU-Net's shape spread far beyond microscopy. Medical imaging adopted it as a default, later with 3D versions and the self-configuring "nnU-Net". Most surprisingly, the image generators of the 2020s use a U-Net at their core: diffusion models train a U-Net to remove noise from images step by step, and Stable Diffusion runs one on a compressed version of the image. More recent systems such as Segment Anything use a transformer instead, trained on vastly more data.

TakeawayWeighting the loss towards hard, thin boundaries is what lets U-Net separate cells that touch.

Follow the idea

Led to: DDPM (2020), Segment Anything (2023)