Study guide · Neural networks · 2000

Learning to Forget: Continual Prediction with LSTM

Felix Gers, Jürgen Schmidhuber, Fred Cummins · Neural Computation

The original LSTM could hold a memory for a thousand steps but could never let go of it. Fed one long, unbroken stream, its memory cells filled up and jammed. This paper adds a third, learned gate that lets each cell fade or wipe its contents, which turned LSTM into the version used for the next twenty years.

The paper in four sentences

The 1997 LSTM memory cell only ever adds to what it holds. That works when someone resets the network between examples, but real data (speech, text, sensor feeds) arrives as one endless stream.

On an endless stream the cell's contents keep growing until it saturates and stops learning.

The fix is a forget gate: a learned dial between 0 and 1 that multiplies the cell's old contents at every step. Near 1 it remembers; near 0 it wipes.

With forget gates, LSTM learns by itself to reset at the right moments, and solves continual tasks that the original LSTM failed.

Idea · §1–2

A memory that can't forget fills up

An LSTM cell is a running total. Each step, its input gate decides how much new information to add, and the cell keeps everything it has added so far. In the 1997 experiments this was fine, because each training sequence was short and the network was reset to zero before the next one.

Now imagine listening to a radio station that never stops. There is no "start of the next example" for anyone to press reset. The cell keeps adding, its value creeps up and up, and eventually it saturates: its output is stuck at the maximum, and the learning signal through it dies. The authors call the general version of this problem the inability to deal with continual input streams.

A stream of "sentences" (grey bands) with no resets between them. Each sentence starts with one fact worth remembering. Without a forget gate, every sentence's fact piles on top of the previous ones. With a forget gate that has learned to close at each sentence boundary, the cell holds only the current fact.

TakeawayA memory that only accumulates works for short, separated examples, but jams on a continuous stream.

Idea · §2

The forget gate is a fading dial

The new gate sits on the cell's loop back to itself. Each step, the old contents are multiplied by the forget gate's value before anything new is added. At 1, nothing is lost (the old LSTM). At 0, the slate is wiped. In between, memory fades smoothly, like a sound echoing away.

Crucially, the gate is learned and changes with the input. The network can keep the gate at nearly 1 while a sentence is in progress, then drop it to near 0 exactly when the sentence ends.

A value of 1 is stored at step 0 and then left alone. Each bar is what remains after that many steps if the forget gate stays at the chosen value.
Everyday pictureA whiteboard in a meeting. The 1997 LSTM can only add notes. The forget gate is the eraser: use it a little to tidy up, or wipe the board when the topic changes.

TakeawayThe forget gate multiplies the old memory by a learned number between 0 and 1 at every step, so the cell can keep, fade or wipe its contents.

Check yourself
Which forget gate value reproduces the original 1997 LSTM?
Exactly 1, at every step.
With the gate at 0.5, how much of a stored value is left after 3 steps?
0.5 × 0.5 × 0.5 = 0.125, one eighth.

Idea · §3–4

It learns when to reset, on its own

The main test is a "grammar" game. Strings of letters follow fixed rules, and the second letter of each string must be remembered until near its end, where it has to be predicted again. In the continual version, the strings are glued together into one endless stream with no resets.

The original LSTM fails this continual task. The LSTM with forget gates solves it, and when the authors looked inside they found what they hoped: the forget gates drop toward zero right at the end of each string, wiping the cell just when the old information stops being useful. Nobody told the network where strings end; it worked that out because resetting there made the predictions better.

TakeawayGiven a forget gate, the network discovers by itself where one piece of the stream ends and resets its memory there.

Idea · Legacy

"LSTM" now means this version

Almost every LSTM used since includes the forget gate, so much so that textbooks often present it as part of the original design. Speech recognition, machine translation, handwriting recognition and phone keyboards ran on this cell through the 2000s and 2010s, until transformers took over.

One practical lesson stuck: start the forget gate close to "remember" (open) when training begins, so the network doesn't wipe its memory before it has learned what is worth keeping.

TakeawayThe forget gate is the piece that made LSTM practical on real, unsegmented data.

Follow the idea

Built on: LSTM (1997)