LSTM

The full recurrence#

Hochreiter & Schmidhuber (1997) introduced the memory cell; Gers, Schmidhuber & Cummins (2000) added the forget gate, giving the form universally used today:

i_t = σ( W_i x_t  +  U_i h_{t−1}  +  b_i )          input gate
f_t = σ( W_f x_t  +  U_f h_{t−1}  +  b_f )          forget gate
o_t = σ( W_o x_t  +  U_o h_{t−1}  +  b_o )          output gate
g_t = tanh( W_g x_t  +  U_g h_{t−1}  +  b_g )       candidate update

c_t = f_t ⊙ c_{t−1}  +  i_t ⊙ g_t                   cell state
h_t = o_t ⊙ tanh( c_t )                             hidden state

• x_t ∈ R^d — input at step t; d is the input dimension
• h_t, c_t ∈ R^m — hidden and cell state; m is the hidden size
• W_• ∈ R^{m×d}, U_• ∈ R^{m×m}, b_• ∈ R^m — per-gate parameters
• σ — logistic sigmoid, mapping to (0,1), so each gate is a soft per-coordinate switch
• ⊙ — elementwise (Hadamard) product

The gates are vectors, not scalars. Each of the m memory slots is independently written, held and read.

Implementation note. PyTorch concatenates the four gates into single matrices weight_ih_l0 ∈ R^{4m×d} and weight_hh_l0 ∈ R^{4m×m}, ordered [i, f, g, o], and carries two bias vectors b_ih and b_hh — a redundancy inherited from cuDNN. Parameter count per layer is therefore

4 m (d + m + 2)          PyTorch, with the duplicated bias
4 m (d + m + 1)          the mathematical minimum

For d = m = 512: 2.10 M parameters per layer. Roughly four times an Elman RNN of the same width, which is the price of the gates.

Why the gradient survives: the constant error carousel#

Differentiate the cell recurrence with respect to the previous cell state:

∂c_t / ∂c_{t−1}  =  diag(f_t)  +  ( terms via h_{t−1} inside i_t, f_t, g_t )

The dominant term is diag(f_t). Composing across a span from s to t:

∂c_t / ∂c_s  ≈  Π_{k=s+1..t}  diag(f_k)

Compare the Elman case, where each factor is diag(φ'(a_k)) · W_hh. Three differences matter.

1. No weight matrix in the path. The product involves no repeated multiplication by W_hh, so its spectral radius does not govern decay. Bengio, Simard & Frasconi's dichotomy — stable storage requires contraction, gradient flow forbids it — does not bind on this path.
2. No saturating derivative in the path. tanh' has supremum 1 and σ' supremum 0.25; neither appears here.
3. The decay rate is learned and data-dependent. With f_k → 1 the product approaches the identity, giving the constant error carousel of the original paper.

The measurement in the Developer section is this expression made numeric. At f = σ(5) ≈ 0.9933, the product over 199 steps is 0.9933^199 ≈ 0.263, matching the observed drop from 0.4835 to 0.1305.

The corollary is the exercise at the end of that section: with b_f = 0, f ≈ 0.5 and 0.5^199 underflows. An LSTM whose forget gate sits near 0.5 has no more long-range gradient than a vanilla RNN. The architecture supplies a path; the gate values decide whether it is open.

Forget-gate bias initialisation#

b_f = 1 (sometimes 2) is standard and the reason is exactly the above. Jozefowicz, Zaremba & Sutskever (2015) ran an architecture search over thousands of RNN variants and reported that adding a forget bias of 1 was worth more than most architectural changes they tested — including, on several tasks, the difference between LSTM and GRU.

nn.LSTM initialises all parameters from U(−1/√m, 1/√m), so the default forget bias is near zero. Correcting it takes one line, applied to the [m : 2m] slice of bias_ih_l0:

with torch.no_grad():
    lstm.bias_ih_l0[m:2*m].fill_(1.0)

There is no output block because there is nothing to print: fill_ mutates the parameter in place and returns the tensor it wrote into.

TensorFlow's LSTM layer has unit_forget_bias=True by default. PyTorch does not, which is a live source of unexplained training failures.

Variants#

Peephole connections (Gers & Schmidhuber, 2000) let the gates see c_{t−1} directly: i_t = σ(W_i x_t + U_i h_{t−1} + p_i ⊙ c_{t−1} + b_i). Needed for tasks requiring precise interval timing, where the output gate would otherwise be blind to the cell it controls. Rarely used now; not exposed by nn.LSTM.

Coupled input and forget gates set i_t = 1 − f_t, saving m(d + m + 1) parameters at negligible cost in quality.

GRU (Cho et al., 2014) merges the cell and hidden states and uses two gates:

z_t = σ( W_z x_t + U_z h_{t−1} + b_z )                     update gate
r_t = σ( W_r x_t + U_r h_{t−1} + b_r )                     reset gate
n_t = tanh( W_n x_t + r_t ⊙ (U_n h_{t−1}) + b_n )
h_t = (1 − z_t) ⊙ n_t  +  z_t ⊙ h_{t−1}

3m(d + m + 1) parameters, 25 percent fewer than an LSTM. Chung et al. (2014) found the two comparable on the tasks they tested. Greff et al. (2017), LSTM: A Search Space Odyssey, ablated eight variants over 5,400 runs and found none reliably better than the standard LSTM, with the forget gate and the output activation the two components whose removal hurt most.

Layer-normalised LSTM (Ba, Kiros & Hinton, 2016) normalises the pre-activations per step, stabilising training at the cost of losing the fused cuDNN kernel — typically a 2–4× slowdown.

ConvLSTM (Shi et al., 2015) replaces the matrix products with convolutions, giving a recurrent model over spatial grids for precipitation nowcasting and video.

Cost, against a transformer#

                       sequential steps   train time / layer    inference state
LSTM                   O(T)               O(T · m²)             O(m)          constant
Transformer            O(1)               O(T² d + T d²)        O(T · d)      KV cache

The first column is the decisive one. Vaswani et al. (2017) cite the number of sequential operations explicitly as a design driver. An LSTM's step t cannot begin before t−1 completes, so wall-clock training time scales with T irrespective of how many accelerators you have. This is what ended LSTM dominance in language, and it is a statement about hardware utilisation rather than about representational power.

The third column is why recurrence survives. An LSTM's decoding state is O(m) regardless of how much it has read; a transformer's KV cache grows linearly in context. For a model streaming for hours, or running inside a hearing aid, that is the binding constraint.

Maximum path length between two positions is O(1) under attention and O(T) under recurrence, which removes the long-range gradient argument entirely rather than mitigating it. That is the second, quieter reason transformers won.

Where LSTMs remain in production#

Streaming ASR. The RNN-Transducer (Graves, 2012) pairs a recurrent encoder over audio with a prediction network over emitted labels, and emits tokens without seeing future audio. He et al. (2019) describe a fully on-device LSTM RNN-T shipped for mobile speech recognition. Conformer encoders have replaced LSTMs in many places, but chunked-streaming variants still carry recurrent state, and RNN-T remains the dominant streaming loss.

Probabilistic forecasting. DeepAR (Salinas et al., 2020) is an LSTM emitting the parameters of a likelihood per step, trained across many related series. It remains a live baseline in forecasting libraries, and the datasets in this domain are frequently far too small to train a transformer well.

Reinforcement learning with partial observability. R2D2 (Kapturowski et al., 2019) uses an LSTM over the agent's observation history with stored recurrent states in the replay buffer. Constant-size state per timestep is the reason.

Embedded and low-power inference. Keyword spotting, gesture recognition and sensor anomaly detection on microcontrollers, where the KV cache is not affordable.

The revival#

xLSTM (Beck et al., 2024) revisits the cell directly. Two changes: exponential rather than sigmoid gating with a normaliser state, which permits revising a stored value the original could not overwrite; and mLSTM, which replaces the vector cell with a matrix memory and a covariance update rule, removing the sequential dependency in the recurrence so training parallelises. Reported competitive with transformers and Mamba at billion-parameter scale.

This sits inside the broader linear-recurrence line — S4, Mamba, RWKV — surveyed in the researcher block of the RNN lesson. The shared observation is that the fixed-size state which forced LSTMs out of language is also the property that makes constant-memory inference possible, and that the sequential training constraint, not the recurrence itself, was the fatal part.

Papers#

• Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8).
• Gers, F., Schmidhuber, J. & Cummins, F. (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation 12(10).
• Gers, F. & Schmidhuber, J. (2000). Recurrent Nets that Time and Count. IJCNN.
• Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. — arxiv.org/abs/1211.3711
• Cho, K. et al. (2014). Learning Phrase Representations using RNN Encoder-Decoder. — arxiv.org/abs/1406.1078
• Chung, J. et al. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. — arxiv.org/abs/1412.3555
• Jozefowicz, R., Zaremba, W. & Sutskever, I. (2015). An Empirical Exploration of Recurrent Network Architectures. ICML.
• Shi, X. et al. (2015). Convolutional LSTM Network. NeurIPS — arxiv.org/abs/1506.04214
• Greff, K. et al. (2017). LSTM: A Search Space Odyssey. IEEE TNNLS — arxiv.org/abs/1503.04069
• Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS — arxiv.org/abs/1706.03762
• Kapturowski, S. et al. (2019). Recurrent Experience Replay in Distributed Reinforcement Learning. ICLR.
• Salinas, D. et al. (2020). DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. IJF 36(3).
• Beck, M. et al. (2024). xLSTM: Extended Long Short-Term Memory. — arxiv.org/abs/2405.04517

What to learn next#

• Transformers — the architecture that replaced this one for language, and why.
• Recurrent neural networks — the simpler cell, and the failure this lesson repairs.
• PyTorch basics — the training loop these examples assume.