The paper in four sentences
The authors searched, with an optimisation method, for the smallest change to an image that would make a neural network give a chosen wrong answer.
For every image they tried, on every network they tried, such a change existed and was tiny: the altered picture looks identical to a person, yet the network now says "ostrich" with confidence.
Random noise of the same size barely affects the network; only carefully aimed changes do. And an image altered to fool one network often fools other networks with different designs or trained on different data.
A second finding: single neurons inside a network are no more meaningful than random combinations of neurons, so the "meaning" lives in the whole space, not in individual units.
Idea · §4 · Figs. 5, 7
An invisible change, a confident wrong answer
A classifier is a program that looks at an image and outputs a label, such as "school bus" or "dog". In 2013 deep neural network classifiers were rapidly approaching human accuracy on photographs, and it was natural to assume they "saw" roughly as we do.
The authors asked a sharper question: what is the smallest change to this image that makes the network say something else? They let a computer search for it. The answer was startling. For images from ImageNet, the changes were so small that the altered images look identical to the originals, yet the network labelled every one of them "ostrich". They called these altered images adversarial examples.
Just as important: a random speckle of the same size does almost nothing. The change has to be aimed.
TakeawayFor a trained network there are tiny, aimed changes that flip its answer, while random changes of the same size do not.
Check yourself
- Why is the random change so much weaker than the aimed one at the same size?
- Random nudges push the score up for some pixels and down for others, so they mostly cancel. Aimed nudges all push the same way, so their effects add up.
- Does the network make this mistake because the picture became ambiguous?
- No. To a person the altered picture is just as clear as the original. The error reveals something about how the network computes, not about the image.
Idea · §4.3, §5 · later work
Why tiny changes can add up
The paper offered a geometric picture: adversarial examples are rare (you will almost never stumble on one by chance) but they are everywhere, close to every ordinary image, "much like the rational numbers" are close to every number on a line. It also measured how much each layer of a network can stretch a small input change, and found the stretch could be large even in the first layer.
A year later, Goodfellow, Shlens and Szegedy gave a simpler explanation. Images have many pixels. If each pixel is changed by a tiny amount in the direction the network is most sensitive to, the effects add up across all pixels. Double the number of pixels and the total effect doubles, while the change per pixel stays just as invisible. Random changes only grow with the square root of the number of pixels, because they partly cancel.
TakeawayIn high dimensions, many invisible nudges in the right direction add up to a big change in the network's output.
Idea · §4.2 · Tables 2–4
A trick that fools one network fools others
If adversarial examples were quirks of one particular network, they would be a curiosity. The paper found something worse: images altered to fool one network often fool others that were built differently or trained on a different half of the data.
On handwritten digits (the MNIST dataset), changes made against one network caused errors in a separately trained network of a different design on anywhere from a few percent to over 80% of images (examples made on the deeper networks transferred at up to about 40%), while random noise of a similar strength caused almost none. Changes made against a network trained on one half of the training set fooled a network trained on the other half 5.9% of the time, and amplifying them a little raised that to 43% (Table 4).
| Altered for… | …then shown to | Error rate |
|---|---|---|
| FC200-200-10 (two hidden layers) | FC10 (no hidden layer) | 38.2% |
| FC200-200-10 | FC100-100-10 | 20.3% |
| AE400-10 (autoencoder features) | FC10, heavy weight decay | 24.8% |
| Random noise, std 0.1 | FC100-100-10 | 0% |
Selected cells from the paper's Table 2. A network named FCa-b-10 is fully connected with hidden layers of a and b units and 10 outputs.
TakeawayAdversarial examples transfer between models, so an attacker may not need access to the exact model they want to fool.
Check yourself
- Why does transfer make adversarial examples a security problem rather than just a curiosity?
- An attacker can train their own substitute model, craft examples against it, and use them on a system whose internals they cannot see.
Idea · §3 · Figs. 1–4
The other finding: single neurons aren't special
Researchers often tried to understand a network by finding the images that most strongly activate a single neuron, and noticing that they share a theme ("this neuron detects faces"). The authors did the same for random combinations of neurons, and found the top images shared themes just as clearly.
Their conclusion: it is the overall space of activity, not the individual neurons, that carries meaning. Later interpretability research refined this. Many neurons respond to several unrelated things at once (called polysemantic neurons), and some meaningful directions do align with neurons while many do not.
TakeawayInspecting single neurons can mislead; meaningful directions in a network need not line up with individual units.
Idea · §4.2, §5 · Legacy
Defences, arms races and language models
The paper also tried a first defence: add adversarial examples to the training data. On digit recognition, this brought test error below 1.2%, better than the same network with weight decay or dropout, so adversarial examples were useful training data. That idea, adversarial training, is still the most reliable defence, though it is expensive and only protects against the kinds of change it was trained on.
A decade later, the same pattern showed up in language models: a string of odd-looking words, found by search, can make a chatbot ignore its safety training, and it transfers to other chatbots (GCG, 2023).
TakeawayThis paper revealed a weakness that every later generation of models, including language models, has shared.
Abstract, restated
Deep networks are highly expressive, which yields good performance but also uninterpretable solutions with counter-intuitive properties. Two are reported. First, by a unit-analysis method, there is no distinction between individual high-level units and random linear combinations of them, suggesting the space rather than individual units holds semantic information. Second, deep networks learn input–output mappings that are discontinuous to a significant extent: an imperceptible, optimised perturbation can cause misclassification, and the same perturbation can cause a different network trained on a different subset of the data to misclassify. Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, Fergus; arXiv 1312.6199 (Dec 2013), ICLR 2014.
Section · §2–3 · Figs. 1–4
Setup, and units versus random directions
Models (§2). MNIST: fully connected nets with one or more hidden layers and softmax ("FC"), and a classifier on autoencoder features ("AE"). ImageNet: the Krizhevsky et al. convolutional network ("AlexNet"). YouTube: "QuocNet", an unsupervised network trained on about 10M YouTube images, with a linear classifier on top.
Unit analysis (§3). Let φ(x) be a layer's activation vector and ei the natural basis vector of unit i. Prior work interpreted units by the held-out images maximising the unit:
The paper repeats this with a random direction v ∈ ℝn in place of ei, and finds the retrieved images are similarly semantically coherent on both MNIST (Figs. 1–2) and ImageNet (Figs. 3–4). The evidence is qualitative (visual inspection). It undercuts the assumption that the coordinate basis is privileged.
Section · §4.1
Finding minimal adversarial perturbations
Let f : ℝm → {1, …, k} be a classifier with continuous loss lossf. For an image x ∈ [0, 1]m and target label l, the minimal perturbation solves
Its minimiser defines D(x, l), and ‖D(x, l) − x‖ is the distortion. The problem is hard, so the paper approximates it with box-constrained L-BFGS on a penalty formulation:
with a line search over c > 0 for the value at which the minimiser satisfies f(x + r) = l. The paper words this as finding the minimum such c; since c weights the size penalty, the smallest successful perturbation corresponds to the boundary where increasing c further would make the attack fail, and this boundary is what the search locates. For convex losses the penalty method is exact; for networks it is an approximation. Try it on a toy logistic classifier:
Section · §4.2 · Tables 1–4
Experiments
Table 1 (MNIST): every model admits adversarial examples with small average minimum distortion (the paper's distortion measure, an RMS-style per-pixel quantity).
| Model | Train err. | Test err. | Av. min. distortion |
|---|---|---|---|
| FC10(10−4) | 6.7% | 7.4% | 0.062 |
| FC10(10−2) | 10% | 9.4% | 0.1 |
| FC10(1) | 21.2% | 20% | 0.14 |
| FC100-100-10 | 0% | 1.64% | 0.058 |
| FC200-200-10 | 0% | 1.54% | 0.065 |
| AE400-10 | 0.57% | 1.9% | 0.086 |
FC10(λ) is softmax regression with weight decay λ; heavier regularisation requires larger distortion but costs accuracy. Fig. 7 notes that randomly distorted images with standard deviation 1 are still classified correctly 51% of the time, whereas adversarial distortions near 0.06 succeed always.
Cross-model (Table 2). Adversarial examples crafted on one model and fed to others: examples made for FC200-200-10 cause 38.2% error on FC10(10−4), those for FC100-100-10 cause 28.9% on the same model, and those for AE400-10 cause 24.8% on FC10(1); Gaussian noise of std 0.1 gives 0% error and std 0.3 gives 5% and 4.3% on the two deep FC nets. Cross-training-set (Tables 3–4). MNIST split into disjoint halves P1, P2 of 30,000 each; examples for FC100-100-10 on P1 cause 26.2% error on FC123-456-10 (P1) and 5.9% on FC100-100-10′ (P2); magnified to std 0.1, 98% and 43%.
ImageNet (Fig. 5). For AlexNet, all perturbed images are classified "ostrich, Struthio camelus", with average distortion 0.006508 over 64 examples. QuocNet (Fig. 6) shows the same for a binary car classifier.
Adversarial training. Training a 100-100-10 network on a pool of adversarial examples refreshed during training reached test error below 1.2%, versus 1.6% with weight decay alone and about 1.3% with dropout. The paper notes examples generated for higher layers seemed more useful than input-level ones, and calls this preliminary.
Section · §4.3 · Eq. 1 · Table 5
Spectral analysis of instability
Write the network as φ(x) = φK(φK−1(… φ1(x; W1) …)). If each layer is Lk-Lipschitz, ‖φk(x) − φk(x + r)‖ ≤ Lk ‖r‖, then the whole network satisfies
For a ReLU layer φk(x) = max(0, Wkx + bk), Lk ≤ ‖Wk‖ (operator norm). Max-pooling is contractive; contrast normalisation is contractive for γ ∈ [0.5, 1]. For convolutions, the operator norm is computed in the Fourier domain via Parseval: ‖W‖ = supξ ‖A(ξ)‖ (Eq. 1), where A(ξ) collects the filters' Fourier coefficients at frequency ξ. Table 5 reports these bounds for the ImageNet network.
The authors conclude instability can appear as early as the first convolutional layer, and suggest penalising these bounds as a regulariser (not tested in the paper). Later work did exactly this: Parseval networks (Cissé et al., 2017) and spectral normalisation (Miyato et al., 2018) constrain layer operator norms.
Section · §5 · Lineage
Limitations and lineage
- The explanation offered (adversarial "pockets" that are low-probability but dense, like the rationals) is a metaphor, not a theory; the authors state they lack a deep understanding of how often adversarial negatives appear.
- L-BFGS is slow and only approximates the minimal perturbation for non-convex models.
- Transfer is quantified only on MNIST; ImageNet transfer is not measured.
- Adversarial training results are preliminary and small-scale.
- A minor inconsistency: §3 uses a convolutional MNIST network, while §4.2 says convolutional MNIST results were not yet available.
Reference
Glossary
- Adversarial example
- An input deliberately perturbed, usually imperceptibly, to cause misclassification.
- Distortion
- Size of the perturbation; the paper reports an average minimum per model.
- Box constraint
- Keeping pixel values in [0, 1].
- L-BFGS
- Limited-memory quasi-Newton optimiser; the "box-constrained" variant handles bounds.
- Transferability
- Adversarial examples crafted for one model fooling another.
- Lipschitz constant
- Maximum ratio of output change to input change.
- Operator norm
- Largest singular value of a linear map; bounds how much it can stretch a vector.
- Adversarial training
- Training on adversarial examples to increase robustness.