Study guide · Safety & security · 2013

Intriguing properties of neural networks

Christian Szegedy et al. · ICLR 2014

Are image classifiers that score near-human accuracy actually seeing what we see? This paper found that for every test image, there is a nearby image, changed by amounts too small for a person to notice, that the network labels completely wrongly, and that the same altered image often fools other networks trained separately. It named these "adversarial examples" and started a field.

The paper in four sentences

The authors searched, with an optimisation method, for the smallest change to an image that would make a neural network give a chosen wrong answer.

For every image they tried, on every network they tried, such a change existed and was tiny: the altered picture looks identical to a person, yet the network now says "ostrich" with confidence.

Random noise of the same size barely affects the network; only carefully aimed changes do. And an image altered to fool one network often fools other networks with different designs or trained on different data.

A second finding: single neurons inside a network are no more meaningful than random combinations of neurons, so the "meaning" lives in the whole space, not in individual units.

Idea · §4 · Figs. 5, 7

An invisible change, a confident wrong answer

A classifier is a program that looks at an image and outputs a label, such as "school bus" or "dog". In 2013 deep neural network classifiers were rapidly approaching human accuracy on photographs, and it was natural to assume they "saw" roughly as we do.

The authors asked a sharper question: what is the smallest change to this image that makes the network say something else? They let a computer search for it. The answer was startling. For images from ImageNet, the changes were so small that the altered images look identical to the originals, yet the network labelled every one of them "ostrich". They called these altered images adversarial examples.

Just as important: a random speckle of the same size does almost nothing. The change has to be aimed.

A toy classifier deciding "7" versus "1" on a 16 × 16 image (pixel brightness 0 to 1). The middle panel shows the change, magnified so you can see it; the right panel is what the classifier actually receives. The aimed change nudges every pixel by the same small amount in whichever direction lowers the score for "7". This aiming rule is the later "fast gradient sign" method, used here because it is simple; the paper itself used a slower optimiser to find the very smallest change. Illustrative, not one of the paper's networks.

TakeawayFor a trained network there are tiny, aimed changes that flip its answer, while random changes of the same size do not.

Check yourself
Why is the random change so much weaker than the aimed one at the same size?
Random nudges push the score up for some pixels and down for others, so they mostly cancel. Aimed nudges all push the same way, so their effects add up.
Does the network make this mistake because the picture became ambiguous?
No. To a person the altered picture is just as clear as the original. The error reveals something about how the network computes, not about the image.

Idea · §4.3, §5 · later work

Why tiny changes can add up

The paper offered a geometric picture: adversarial examples are rare (you will almost never stumble on one by chance) but they are everywhere, close to every ordinary image, "much like the rational numbers" are close to every number on a line. It also measured how much each layer of a network can stretch a small input change, and found the stretch could be large even in the first layer.

A year later, Goodfellow, Shlens and Szegedy gave a simpler explanation. Images have many pixels. If each pixel is changed by a tiny amount in the direction the network is most sensitive to, the effects add up across all pixels. Double the number of pixels and the total effect doubles, while the change per pixel stays just as invisible. Random changes only grow with the square root of the number of pixels, because they partly cancel.

The slider sets the number of pixels as a power of ten (from 10 to a million). Each pixel changes by 1% of its range; each of the classifier's weights has a typical size of 1. Amber: total shift in the score from an aimed change (grows with the number of pixels). Blue: typical shift from a random change (grows with its square root). The dashed line is an illustrative score margin of 10 that must be overcome to flip the answer. This is the linear explanation from Goodfellow et al. (2014), not from this paper.
Later refinementThe "rational numbers" picture in this paper was largely replaced by the linear explanation above, which also produced a fast attack (the fast gradient sign method) and made adversarial training practical. Later still, Ilyas et al. (2019) argued that adversarial examples exploit genuinely predictive but non-human-like "features" in the data.

TakeawayIn high dimensions, many invisible nudges in the right direction add up to a big change in the network's output.

Idea · §4.2 · Tables 2–4

A trick that fools one network fools others

If adversarial examples were quirks of one particular network, they would be a curiosity. The paper found something worse: images altered to fool one network often fool others that were built differently or trained on a different half of the data.

On handwritten digits (the MNIST dataset), changes made against one network caused errors in a separately trained network of a different design on anywhere from a few percent to over 80% of images (examples made on the deeper networks transferred at up to about 40%), while random noise of a similar strength caused almost none. Changes made against a network trained on one half of the training set fooled a network trained on the other half 5.9% of the time, and amplifying them a little raised that to 43% (Table 4).

Altered for……then shown toError rate
FC200-200-10 (two hidden layers)FC10 (no hidden layer)38.2%
FC200-200-10FC100-100-1020.3%
AE400-10 (autoencoder features)FC10, heavy weight decay24.8%
Random noise, std 0.1FC100-100-100%

Selected cells from the paper's Table 2. A network named FCa-b-10 is fully connected with hidden layers of a and b units and 10 outputs.

TakeawayAdversarial examples transfer between models, so an attacker may not need access to the exact model they want to fool.

Check yourself
Why does transfer make adversarial examples a security problem rather than just a curiosity?
An attacker can train their own substitute model, craft examples against it, and use them on a system whose internals they cannot see.

Idea · §3 · Figs. 1–4

The other finding: single neurons aren't special

Researchers often tried to understand a network by finding the images that most strongly activate a single neuron, and noticing that they share a theme ("this neuron detects faces"). The authors did the same for random combinations of neurons, and found the top images shared themes just as clearly.

Their conclusion: it is the overall space of activity, not the individual neurons, that carries meaning. Later interpretability research refined this. Many neurons respond to several unrelated things at once (called polysemantic neurons), and some meaningful directions do align with neurons while many do not.

Everyday pictureDescribing colours by "how red, how green, how blue" works, but so does "how bright, how warm, how saturated". Neither set of directions is the "true" one; the colour space itself is what matters.

TakeawayInspecting single neurons can mislead; meaningful directions in a network need not line up with individual units.

Idea · §4.2, §5 · Legacy

Defences, arms races and language models

The paper also tried a first defence: add adversarial examples to the training data. On digit recognition, this brought test error below 1.2%, better than the same network with weight decay or dropout, so adversarial examples were useful training data. That idea, adversarial training, is still the most reliable defence, though it is expensive and only protects against the kinds of change it was trained on.

A decade later, the same pattern showed up in language models: a string of odd-looking words, found by search, can make a chatbot ignore its safety training, and it transfers to other chatbots (GCG, 2023).

Later refinementMadry et al. (2018) framed adversarial training as a min-max problem solved with projected gradient descent, the standard recipe today. Carlini and Wagner (2017) showed many proposed defences failed against stronger attacks. Adversarial examples were also shown to work in the physical world, for example printed photos and stickers on road signs.

TakeawayThis paper revealed a weakness that every later generation of models, including language models, has shared.

Follow the idea

Led to: Universal jailbreaks (GCG) (2023)