The paper in four sentences
A recurrent network reads a sequence one step at a time, passing a memory forward. To learn, it sends error signals backwards through those steps.
Each step multiplies the signal by some factor. Below 1, it shrinks to nothing within a few dozen steps; above 1, it blows up. So long-range lessons can't be learned.
LSTM adds a memory cell that copies its value forward unchanged (factor exactly 1), so the signal neither fades nor explodes.
Learned "gates" decide when to write into the cell and when to read from it, so the cell isn't disturbed by irrelevant inputs. The "forget" gate every modern LSTM has was added three years later.
Idea · §1–3
Reading a sequence with a memory
Some data comes in order: words in a sentence, notes in a song, readings from a sensor. A recurrent network handles this by reading one item at a time and carrying a small internal memory from each step to the next, like a person reading a sentence word by word.
To learn, it uses backpropagation, unrolled through time: the error at the end is passed back step by step to the moments that caused it. The trouble starts when the cause is far back, say the subject of a long sentence that decides a verb many words later.
TakeawayRecurrent networks learn from sequences by passing error back through time, which gets hard when cause and effect are far apart.
Idea · §3
Why the signal fades or explodes
Every step back in time, the error signal gets multiplied by roughly the same number (a weight times the slope of the unit). Multiply by 0.9 a hundred times and you get 0.00003: the signal has vanished. Multiply by 1.1 a hundred times and you get 13,780: it has exploded. Only a factor of exactly 1 keeps it steady.
Hochreiter had worked this out in detail in his 1991 diploma thesis. It means ordinary recurrent networks effectively can't learn links longer than about 5–10 steps.
TakeawayRepeated multiplication makes learning signals vanish or explode over long time gaps. A factor of exactly 1 is the only stable choice.
Check yourself
- Roughly how much signal is left after 50 steps at factor 0.8?
- 0.850 ≈ 0.000014, essentially nothing.
Idea · §4
A memory cell with gates
LSTM's core is a memory cell that connects to itself with a factor of exactly 1. Left alone, it keeps its value forever, and error flowing back through it stays the same size. The authors call this the constant error carousel.
But a cell that everything writes into gets scrambled, and a cell that always broadcasts disturbs everything else. So the cell gets two learned gates: an input gate that decides when new information may be written in, and an output gate that decides when the stored value is let out. Each gate is a number between 0 (closed) and 1 (open), computed from the current situation.
Below, open the input gate at the moment you want to store something, keep it closed while noise goes by, and open the output gate when you want to read it.
TakeawayThe cell remembers by copying itself forward unchanged; the gates control what gets written in and what gets read out.
Check yourself
- What would happen without an input gate?
- Every input would be added to the cell, so irrelevant inputs would overwrite the information it is meant to keep.
- What can't the 1997 cell do that the forget gate fixes?
- It cannot clear itself. Its contents only accumulate, which becomes a problem on long, continuous streams.
Idea · §5
Tests built to need long memory
The authors design tasks where the answer depends on something seen long before, with nothing but noise in between. In the "adding problem", the network sees a long stream of random numbers, two of which are marked, and at the end must output their sum. In others, it must remember one symbol from the start of a sequence over 100 or even 1,000 steps.
Other recurrent methods of the time failed or learned very slowly once the gap went beyond about ten steps. LSTM solved these tasks with gaps of over 1,000 steps, and learned faster.
TakeawayOn tasks designed to require long memory, LSTM succeeded where earlier recurrent networks failed.
Idea · What came next
From 1997 cell to modern LSTM
The LSTM most people use differs from this paper. In 2000, Gers, Schmidhuber and Cummins added the forget gate, letting the cell gradually let go of old information or reset. Later came "peephole" connections and full training through time. For about two decades LSTMs powered speech recognition, machine translation and phone keyboards, until transformers took over by attending to every earlier step directly.
TakeawayLSTM's gated, self-copying memory solved the long-memory problem for recurrent networks and dominated sequence learning until transformers.
Abstract, restated
Learning to store information over extended time intervals by recurrent backpropagation takes very long, mostly because of insufficient, decaying error backflow. LSTM is a gradient-based method that truncates the gradient where this does no harm and enforces constant error flow through "constant error carousels" within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time, with computational complexity O(1) per time step and weight, and bridges lags exceeding 1,000 discrete steps even with noisy, incompressible inputs. Neural Computation 9(8), 1735–1780 (1997).
Section · §3.1
Exponential error decay
For a fully recurrent net with unit activations yi(t) = fi(neti(t)), the error signal ϑ propagated back q steps from unit v to unit u scales as
If |f′(net)·w| > 1 for all terms the product grows exponentially (explodes; weights oscillate); if < 1 it decays exponentially. With logistic units, f′ ≤ 0.25, so |w| < 4 guarantees decay. The analysis follows Hochreiter's 1991 diploma thesis; Bengio, Simard and Frasconi (1994) reached similar conclusions independently.
Section · §3.2
The constant error carousel and two conflicts
For a single unit j with a self-connection, constant error flow requires f′j(netj(t)) wjj = 1.0, which integrates to fj linear and wjj = 1: the constant error carousel (CEC).
A naive CEC has two problems. Input weight conflict: the same incoming weight must sometimes store an input and at other times protect the stored value from irrelevant inputs. Output weight conflict: the same outgoing weight must sometimes retrieve the content and at other times keep it from perturbing other units. LSTM resolves both by moving the decision to multiplicative, context-sensitive gates.
Section · §4 · Figs. 1–2
The memory cell
Cell cj has input gate inj and output gate outj, both logistic. With g a centred logistic squashing to [−2, 2] and h one squashing to [−1, 1]:
Cells sharing the same gates form a memory cell block of size S. All net inputs may come from input units, gates, cells and conventional hidden units. Note what is absent relative to the modern cell: there is no forget gate (the internal state's self-weight is fixed at 1) and no peephole connections from the state to the gates.
Learning. A variant of real-time recurrent learning in which errors arriving at net inputs of cells and gates are truncated: they do not propagate further back in time, except within the CEC, where they flow unchanged indefinitely. This gives O(W) cost per time step, with W the number of weights, and is local in space and time.
The paper is aware of this internal state drift (§4.8, "Internal state drift and remedies") and suggests biasing input gates negatively at initialisation. A separate issue, §4.7 "Abuse problem and solutions" (cells being used early in training as mere bias units), is handled with sequential network construction and a negative initial output-gate bias. The principled fix, a learned forget gate, came in 2000.
Section · §5
Experiments
| Exp. | Task | What it tests |
|---|---|---|
| 1 | Embedded Reber grammar | A standard benchmark with short lags; LSTM learns faster and more reliably than RTRL, Elman nets, recurrent cascade-correlation. |
| 2 | Noise-free and noisy sequences with long lags | Remember the first symbol of a sequence; lags up to 1,000 steps, including with many distractor symbols. |
| 3 | Noise and signal on the same channel | Classify a sequence whose relevant information is buried in continuous-valued noise. |
| 4 | Adding problem | Output the sum of two marked real values in a long sequence; lags of 100–1,000. |
| 5 | Multiplication problem | As 4, with a product; tests non-integrative computation. |
| 6 | Temporal order | Classify by the order of two or three widely separated symbols. |
Comparisons were with real-time recurrent learning (RTRL), backpropagation through time (BPTT), Elman nets, recurrent cascade-correlation and neural sequence chunking. The authors also report that random weight guessing solves several earlier "long-lag" benchmarks from the literature, motivating their harder tasks.
Check yourself
- Why does truncating gradients outside the CEC not hurt long-lag learning?
- The long-range credit is carried entirely by the CEC, where error flows unchanged; the truncated paths are the ones that would vanish anyway.
- Why does g map to [−2, 2] but h to [−1, 1]?
- These are the paper's choices of squashing ranges for cell input and output; modern cells use tanh for both.
Section · Lineage
The modern cell
- Gers, Schmidhuber & Cummins (2000): forget gate ft on the self-connection.
- Gers & Schmidhuber (2000), Gers et al. (2002): peephole connections from the state to the gates.
- Graves & Schmidhuber (2005): full BPTT instead of truncated gradients; bidirectional LSTM.
- Cho et al. (2014): the GRU, a simplified gated unit. Greff et al. (2017, "LSTM: A Search Space Odyssey") found the forget gate and output activation the most critical components.
Reference
Glossary
- RNN
- Recurrent neural network; units feed back into themselves across time steps.
- BPTT
- Backpropagation through time: backprop on the network unrolled over the sequence.
- RTRL
- Real-time recurrent learning: forward-mode gradient computation online.
- Vanishing gradient
- Exponential decay of error signals propagated back over many steps.
- CEC
- Constant error carousel: a linear unit with self-weight 1.
- Gate
- A logistic unit whose output multiplies a signal, controlling its flow.
- Memory block
- Cells sharing one input gate and one output gate.
- State drift
- Unbounded growth of the cell state on long input streams.