The paper in four sentences
A perceptron can only adjust its final layer, so it is stuck with whatever features it is given. Networks with "hidden" layers in the middle could do much more, if only we knew how to train the middle.
The fix: run an example forward, measure how wrong the answer is, then pass that error backwards through the network, layer by layer, so every connection learns how much it contributed.
Nudge every connection a little in the direction that reduces the error, and repeat over many examples.
The hidden units end up inventing useful features nobody programmed, such as "which generation is this person in?" in a family-tree task.
Idea · p. 533
The problem: who trains the middle?
A neural network is layers of simple units passing numbers along weighted connections. The input layer receives the data and the output layer gives the answer. In between are hidden units: nobody tells them what they should be doing.
For the output layer, learning is easy: you know the right answer, so you know which way to push. For hidden units there is no "right answer" to compare against. The perceptron dodged this by keeping its middle layer fixed and random, and Minsky and Papert showed how much that limits it. The real prize was a way to train the middle.
TakeawayHidden layers make networks powerful, but they need a way to learn without being told what each hidden unit should do.
Idea · p. 534
Pass the blame backwards
Each training example takes two trips. On the forward pass, the input flows through the network and produces an answer. Compare it with the correct answer and you get an error.
On the backward pass, the error flows the other way. Each output unit knows how wrong it was. Each hidden unit then collects blame from the units it feeds, in proportion to how strongly it is connected to them. Every connection learns how much a small change in it would change the error, and moves a small step the helpful way.
Below, a tiny network learns XOR ("one or the other, but not both"), the very kind of problem a single-layer perceptron cannot solve. The coloured background shows what it currently answers at each point.
TakeawayRun forward to get the error, run backward to share out the blame, nudge every weight. Repeat.
Check yourself
- What travels backwards in backpropagation?
- The error signal: for each unit, how much the total error would change if that unit's input changed.
- Why can this network solve XOR when a perceptron can't?
- Its hidden units learn intermediate features (roughly "at least one is on" and "both are on") that make the final decision a simple straight-line cut.
Idea · p. 535
Walking downhill, with momentum
Think of the error as a landscape: every possible setting of the weights is a location, and the height is how wrong the network is. Training means walking downhill. The backward pass tells you which way is down; the learning rate decides how big a step to take.
The paper also adds momentum: each step keeps part of the previous step's direction, like a ball rolling downhill that doesn't stop dead in every small dip. Try the sliders: too large a step overshoots, and momentum can carry the ball through a shallow dip to a deeper valley.
TakeawayLearning is downhill walking on an error landscape. Step size and momentum decide whether you get stuck, overshoot, or arrive.
Idea · pp. 534–535 · Figs. 1–4
Hidden units invent their own features
The paper's title says "learning representations", and this is the real headline. In one test, a network had to say whether a row of six on/off lights was a mirror image of itself. It had only two hidden units. Training found an elegant trick nobody told it: each light's weight is matched by an equal and opposite weight on its mirror partner, and the three sizes are in the ratio 1 : 2 : 4. A mirror-symmetric input cancels out exactly, while any asymmetry leaves a non-zero total, and the 1 : 2 : 4 sizes make sure different asymmetries can never cancel each other by accident.
In another test, a network learned two family trees, one English and one Italian, from statements like "Colin's mother is Victoria". Its hidden units came to stand for things like nationality, generation, and which branch of the family a person was in. Nobody named those features; they emerged because they made the task easier.
TakeawayBackprop doesn't just fit answers; the hidden layers discover meaningful features of the data on their own.
Check yourself
- In the symmetry task, why do mirror partners get equal and opposite weights?
- So that a symmetric input contributes zero total to each hidden unit, while any asymmetry pushes the total away from zero.
Idea · Context
Not the first, but the one that convinced people
The mathematics was older. Seppo Linnainmaa described the general backward method for computing derivatives in 1970, and Paul Werbos proposed using it to train neural networks in 1974. What this paper added was clear evidence of what it learns: interesting internal representations, in a short paper in a widely read journal, plus a book chapter with many more experiments.
The authors also flagged the weak spots: the downhill walk can get stuck in a poor valley (they found this rare in practice), and they said plainly that this is not a realistic model of how brains learn.
TakeawayBackprop's idea predates 1986; this paper showed it could make hidden layers learn meaningful features, which is what restarted the field.
Abstract, restated
A procedure for networks of neuron-like units repeatedly adjusts weights to minimise a measure of the difference between actual and desired output vectors. As a result, internal hidden units come to represent important features of the task domain, and regularities are captured by the interactions of these units. The ability to create useful new features distinguishes back-propagation from earlier procedures such as the perceptron-convergence procedure. Nature 323 (1986), 533–536.
Section · p. 533
Units and the error measure
Layered feed-forward net; units in a layer receive from lower layers only. Biases are weights from a unit with constant output 1. For unit j:
Here c indexes input–output cases and d the desired outputs. Any differentiable input–output function with bounded derivative would do; the logistic is used for convenience.
Section · p. 534
The backward pass
For one case, starting at the output units:
Equation 7 is the step that makes it work: the error derivative for a hidden unit is a weighted sum of the derivatives of the units it feeds, so the computation proceeds layer by layer from top to bottom at the same cost as the forward pass. This is reverse-mode automatic differentiation applied to the network.
Step through it below on a 2-2-1 network with fixed initial weights, input (1, 0) and target 1.
Section · p. 535
Weight changes and acceleration
The simplest rule is plain gradient descent, Δw = −ε ∂E/∂w, with ∂E/∂w accumulated over all cases before changing the weights (a full "sweep"). The paper uses an accelerated version in which the current gradient modifies the velocity of the point in weight space rather than its position:
with α ∈ [0, 1) an exponential decay factor: what is now called momentum. Weights start at small random values; if they started equal, hidden units with identical connections would receive identical gradients and could never differentiate (symmetry breaking).
Section · pp. 534–535 · Figs. 1–4
Experiments: symmetry and family trees
Mirror symmetry (Fig. 1). Six binary inputs, two hidden units, one output that should be on iff the input is symmetric about its centre. Trained on all 64 vectors with ε = 0.1, α = 0.9, the paper reports a solution after 1,425 sweeps. In the solution, each hidden unit's weights are mirror-antisymmetric with magnitudes in ratio 1 : 2 : 4 (in the paper's Fig. 1 the signs are mixed within one side, e.g. 14.2, −3.6, 7.2, so the ratio holds in some order), so symmetric inputs give each hidden unit zero net input from the array and the shared negative bias keeps both off, turning the output on. The figure below re-runs this experiment with the same equations in your browser.
Family trees (Figs. 2–4). Two isomorphic trees of 12 people each (English and Italian) and 12 relationship types give 104 triples (person 1, relationship, person 2). The network has 24 local input units for person 1 and 12 for the relationship, each group feeding its own 6-unit distributed encoding, a central layer of 12 units, a 6-unit layer and 24 output units for person 2. Trained on 100 triples, it generalised correctly to the 4 held out. Inspection of the 6 units encoding person 1 found units for nationality, generation and branch of the family, features never given explicitly.
Recurrent nets (Fig. 5). A recurrent network run for T steps is equivalent to a layered net with T layers and tied weights; backprop applies after summing the gradients of corresponding weights. This is the origin of back-propagation through time, used later by LSTM.
Section · p. 536 and history
Limitations and lineage
- Local minima. The error surface may have local minima, so gradient descent is not guaranteed to find a global one. The authors report that in many tasks the network very rarely gets stuck in poor local minima, and that extra connections provide paths around them.
- Biology. The authors state that the procedure in its current form is not a plausible model of learning in brains, but that it shows what gradient descent in weight space can discover.
- Priority. Reverse-mode differentiation: Linnainmaa (1970). Application to neural networks: Werbos (1974 thesis; 1982). Independent rediscoveries: Parker (1985), LeCun (1985). The fuller treatment is Rumelhart, Hinton & Williams, "Learning internal representations by error propagation", in Parallel Distributed Processing (1986).
Check yourself
- Why is backprop's cost comparable to a forward pass?
- Each weight is used once forward and once backward (Eq. 7 reuses the already computed ∂E/∂x of the layer above), so the cost is linear in the number of weights rather than requiring a separate perturbation per weight.
- What would happen if all weights started at zero?
- All hidden units in a layer would compute the same function and receive the same gradient, so they would stay identical forever.
Reference
Glossary
- Hidden unit
- A unit that is neither input nor output; its desired state is not specified.
- Forward pass
- Computing all activities from input to output.
- Backward pass
- Computing ∂E/∂x and ∂E/∂w from output to input.
- Sweep
- One pass through all training cases (an epoch).
- ε
- Learning rate.
- α
- Momentum (acceleration) coefficient.
- Local minimum
- A point where every small weight change increases error, though better points exist elsewhere.