Study guide · Neural networks · 1997

Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation

Networks that read sequences, like sentences or sound, kept forgetting anything more than a few steps back, because the learning signal faded as it travelled back in time. Hochreiter and Schmidhuber fix this with a memory cell whose contents are protected by gates, so information and error can survive for over a thousand steps.

The paper in four sentences

A recurrent network reads a sequence one step at a time, passing a memory forward. To learn, it sends error signals backwards through those steps.

Each step multiplies the signal by some factor. Below 1, it shrinks to nothing within a few dozen steps; above 1, it blows up. So long-range lessons can't be learned.

LSTM adds a memory cell that copies its value forward unchanged (factor exactly 1), so the signal neither fades nor explodes.

Learned "gates" decide when to write into the cell and when to read from it, so the cell isn't disturbed by irrelevant inputs. The "forget" gate every modern LSTM has was added three years later.

Idea · §1–3

Reading a sequence with a memory

Some data comes in order: words in a sentence, notes in a song, readings from a sensor. A recurrent network handles this by reading one item at a time and carrying a small internal memory from each step to the next, like a person reading a sentence word by word.

To learn, it uses backpropagation, unrolled through time: the error at the end is passed back step by step to the moments that caused it. The trouble starts when the cause is far back, say the subject of a long sentence that decides a verb many words later.

Everyday pictureReading a mystery novel where the clue is on page 3 and the reveal on page 300. You only learn to spot clues if the "aha" at the end can reach back to page 3.

TakeawayRecurrent networks learn from sequences by passing error back through time, which gets hard when cause and effect are far apart.

Idea · §3

Why the signal fades or explodes

Every step back in time, the error signal gets multiplied by roughly the same number (a weight times the slope of the unit). Multiply by 0.9 a hundred times and you get 0.00003: the signal has vanished. Multiply by 1.1 a hundred times and you get 13,780: it has exploded. Only a factor of exactly 1 keeps it steady.

Hochreiter had worked this out in detail in his 1991 diploma thesis. It means ordinary recurrent networks effectively can't learn links longer than about 5–10 steps.

Height of each bar = how much of the error signal reaches that many steps back (log scale; the dashed line is 1, no loss). Below 1 the bars drain away; above 1 they shoot off the chart.

TakeawayRepeated multiplication makes learning signals vanish or explode over long time gaps. A factor of exactly 1 is the only stable choice.

Check yourself
Roughly how much signal is left after 50 steps at factor 0.8?
0.850 ≈ 0.000014, essentially nothing.

Idea · §4

A memory cell with gates

LSTM's core is a memory cell that connects to itself with a factor of exactly 1. Left alone, it keeps its value forever, and error flowing back through it stays the same size. The authors call this the constant error carousel.

But a cell that everything writes into gets scrambled, and a cell that always broadcasts disturbs everything else. So the cell gets two learned gates: an input gate that decides when new information may be written in, and an output gate that decides when the stored value is let out. Each gate is a number between 0 (closed) and 1 (open), computed from the current situation.

Below, open the input gate at the moment you want to store something, keep it closed while noise goes by, and open the output gate when you want to read it.

Click the gate squares to open (amber) or close them. The cell adds input × input gate at each step and never loses anything in the 1997 design. With the forget gate on, a third row appears: closing it at a step wipes the cell.

TakeawayThe cell remembers by copying itself forward unchanged; the gates control what gets written in and what gets read out.

Check yourself
What would happen without an input gate?
Every input would be added to the cell, so irrelevant inputs would overwrite the information it is meant to keep.
What can't the 1997 cell do that the forget gate fixes?
It cannot clear itself. Its contents only accumulate, which becomes a problem on long, continuous streams.

Idea · §5

Tests built to need long memory

The authors design tasks where the answer depends on something seen long before, with nothing but noise in between. In the "adding problem", the network sees a long stream of random numbers, two of which are marked, and at the end must output their sum. In others, it must remember one symbol from the start of a sequence over 100 or even 1,000 steps.

Other recurrent methods of the time failed or learned very slowly once the gap went beyond about ten steps. LSTM solved these tasks with gaps of over 1,000 steps, and learned faster.

TakeawayOn tasks designed to require long memory, LSTM succeeded where earlier recurrent networks failed.

Idea · What came next

From 1997 cell to modern LSTM

The LSTM most people use differs from this paper. In 2000, Gers, Schmidhuber and Cummins added the forget gate, letting the cell gradually let go of old information or reset. Later came "peephole" connections and full training through time. For about two decades LSTMs powered speech recognition, machine translation and phone keyboards, until transformers took over by attending to every earlier step directly.

Later refinementThe forget gate (2000) is so standard that "LSTM" today means the version with it. The 1997 design here has only input and output gates.

TakeawayLSTM's gated, self-copying memory solved the long-memory problem for recurrent networks and dominated sequence learning until transformers.

Follow the idea

Built on: Backpropagation (1986)

Led to: Forget gate (2000), Attention (2014), Transformer (2017)