The paper in four sentences
The 1997 LSTM memory cell only ever adds to what it holds. That works when someone resets the network between examples, but real data (speech, text, sensor feeds) arrives as one endless stream.
On an endless stream the cell's contents keep growing until it saturates and stops learning.
The fix is a forget gate: a learned dial between 0 and 1 that multiplies the cell's old contents at every step. Near 1 it remembers; near 0 it wipes.
With forget gates, LSTM learns by itself to reset at the right moments, and solves continual tasks that the original LSTM failed.
Idea · §1–2
A memory that can't forget fills up
An LSTM cell is a running total. Each step, its input gate decides how much new information to add, and the cell keeps everything it has added so far. In the 1997 experiments this was fine, because each training sequence was short and the network was reset to zero before the next one.
Now imagine listening to a radio station that never stops. There is no "start of the next example" for anyone to press reset. The cell keeps adding, its value creeps up and up, and eventually it saturates: its output is stuck at the maximum, and the learning signal through it dies. The authors call the general version of this problem the inability to deal with continual input streams.
TakeawayA memory that only accumulates works for short, separated examples, but jams on a continuous stream.
Idea · §2
The forget gate is a fading dial
The new gate sits on the cell's loop back to itself. Each step, the old contents are multiplied by the forget gate's value before anything new is added. At 1, nothing is lost (the old LSTM). At 0, the slate is wiped. In between, memory fades smoothly, like a sound echoing away.
Crucially, the gate is learned and changes with the input. The network can keep the gate at nearly 1 while a sentence is in progress, then drop it to near 0 exactly when the sentence ends.
TakeawayThe forget gate multiplies the old memory by a learned number between 0 and 1 at every step, so the cell can keep, fade or wipe its contents.
Check yourself
- Which forget gate value reproduces the original 1997 LSTM?
- Exactly 1, at every step.
- With the gate at 0.5, how much of a stored value is left after 3 steps?
- 0.5 × 0.5 × 0.5 = 0.125, one eighth.
Idea · §3–4
It learns when to reset, on its own
The main test is a "grammar" game. Strings of letters follow fixed rules, and the second letter of each string must be remembered until near its end, where it has to be predicted again. In the continual version, the strings are glued together into one endless stream with no resets.
The original LSTM fails this continual task. The LSTM with forget gates solves it, and when the authors looked inside they found what they hoped: the forget gates drop toward zero right at the end of each string, wiping the cell just when the old information stops being useful. Nobody told the network where strings end; it worked that out because resetting there made the predictions better.
TakeawayGiven a forget gate, the network discovers by itself where one piece of the stream ends and resets its memory there.
Idea · Legacy
"LSTM" now means this version
Almost every LSTM used since includes the forget gate, so much so that textbooks often present it as part of the original design. Speech recognition, machine translation, handwriting recognition and phone keyboards ran on this cell through the 2000s and 2010s, until transformers took over.
One practical lesson stuck: start the forget gate close to "remember" (open) when training begins, so the network doesn't wipe its memory before it has learned what is worth keeping.
TakeawayThe forget gate is the piece that made LSTM practical on real, unsegmented data.
Abstract, restated
LSTM can solve many tasks not solvable by previous learning algorithms for recurrent networks. But when processing continual input streams without explicitly marked sequence ends, its internal state values may grow indefinitely and eventually cause the network to break down. The remedy is an adaptive forget gate that enables an LSTM cell to learn to reset itself at appropriate times, releasing internal resources. Standard LSTM fails on continual versions of benchmark problems that extended LSTM solves. Gers, Schmidhuber & Cummins, Neural Computation 12(10), 2451–2471 (2000); first presented at ICANN 1999.
Section · §2
Extended LSTM equations
Keep the 1997 notation: memory block with cells cjv, input gate inj, output gate outj, cell-input squashing g ∈ [−2, 2], cell-output squashing h ∈ [−1, 1]. Add a forget gate φj per block, a logistic unit with its own weighted inputs:
The output equation is unchanged: yc(t) = youtj(t) h(sc(t)). The constant error carousel becomes a gated carousel: ∂sc(t)/∂sc(t−1) = yφ(t) (along the direct path), so error flows unattenuated exactly when the gate is open and is cut when the network chooses to forget. The authors keep the 1997 truncated gradient, extended with the partial derivatives for forget-gate weights, preserving O(1) cost per weight and time step.
The forget gates are meant to start open (remembering), otherwise the cell would forget everything before learning what matters; a positive initial bias does this. Jozefowicz, Zaremba & Sutskever (2015) later found a forget-gate bias of 1 to be an important default in a large architecture search.
Check yourself
- Doesn't a gate below 1 bring back the vanishing gradient?
- Only by choice. The gradient over a span is the product of the gate values along it, so if the network holds the gate near 1 while information matters, the signal survives; it decays only where the network has decided to forget.
Section · §3–4
Benchmarks: continual Reber and noisy sequences
The embedded Reber grammar (ERG) wraps an inner Reber string between an outer B·T…T·E or B·P…P·E; predicting the second-to-last symbol requires remembering whether the second symbol was T or P across the inner string. In the continual ERG (CERG) strings are concatenated into one stream with no resets, and the network must predict every next symbol. A trial counts as a success only if the trained network then processes a long test stream without prediction errors. Generate examples below.
A second experiment uses a continual version of one of the 1997 noisy long-lag tasks. In both, standard LSTM fails on the unbroken stream; its state grows until the cell output saturates. The paper also compares standard LSTM with external resets supplied by hand at string ends (which work, but need boundaries that a continual stream does not provide). Among methods given no boundary information, the learned, input-dependent forget gate is what makes the continual tasks solvable, and learning-rate decay improves its success rate further.
Analysis. Plots of cell states and forget-gate activations in trained networks show the forget gates dropping to near zero at string boundaries, implementing learned resets, while some blocks also learn slower, partial decay.
Section · Lineage
Where the forget gate sits in the LSTM family
- Peepholes (Gers & Schmidhuber 2000; Gers, Schraudolph & Schmidhuber 2002): gates also see the cell state, enabling precise timing tasks.
- Full BPTT (Graves & Schmidhuber 2005) replaced truncated gradients, giving the "vanilla LSTM" of textbooks.
- GRU (Cho et al. 2014) couples forget and input gates (z and 1 − z) and drops the separate cell state.
- Ablations: Greff et al. (2017) found removing the forget gate one of the most damaging changes across speech, handwriting and music tasks.
Reference
Glossary
- Continual stream
- Input with no externally marked sequence boundaries or resets.
- Forget gate φ
- Logistic gate multiplying the previous cell state.
- State drift
- Unbounded growth of sc leading to saturated h.
- ERG / CERG
- Embedded Reber grammar; its continual (concatenated) version.
- Peephole
- Connection from cell state to the gates (a later addition).