Long short-term memory

This article may be too technical for most readers to understand. Please help improve it to make it understandable to non-experts, without removing the technical details. (March 2022) (Learn how and when to remove this template message)

Long short-term memory (LSTM)

A common LSTM unit is composed of a cell, an input gate, an output gate[14] and a forget gate.^[15] The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell. Forget gates decide what information to discard from a previous state by assigning a previous state, compared to a current input, a value between 0 and 1. A (rounded) value of 1 means to keep the information, and a value of 0 means to discard it. Input gates decide which pieces of new information to store in the current state, using the same system as forget gates. Output gates control which pieces of information in the current state to output by assigning a value from 0 to 1 to the information, considering the previous and current states. Selectively outputting relevant information from the current state allows the LSTM network to maintain useful, long-term dependencies to make predictions, both in current and future time-steps.

Motivation

In theory, classic

The intuition behind the LSTM architecture is to create an additional module in a neural network that learns when to remember and when to forget pertinent information.[15] In other words, the network effectively learns which information might be needed later on in a sequence and when that information is no longer needed. For instance, in the context of natural language processing, the network can learn grammatical dependencies.^[17] An LSTM might process the sentence "Dave, as a result of his controversial claims, is now a pariah" by remembering the (statistically likely) grammatical gender and number of the subject Dave, note that this information is pertinent for the pronoun his and note that this information is no longer important after the verb is.

Variants

In the equations below, the lowercase variables represent vectors. Matrices ${\displaystyle W_{q}}$ and ${\displaystyle U_{q}}$ contain, respectively, the weights of the input and recurrent connections, where the subscript ${\displaystyle _{q}}$ can either be the input gate ${\displaystyle i}$, output gate ${\displaystyle o}$, the forget gate ${\displaystyle f}$ or the memory cell ${\displaystyle c}$, depending on the activation being calculated. In this section, we are thus using a "vector notation". So, for example, ${\displaystyle c_{t}\in \mathbb {R} ^{h}}$ is not just one unit of one LSTM cell, but contains ${\displaystyle h}$ LSTM cell's units.

LSTM with a forget gate

The compact forms of the equations for the forward pass of an LSTM cell with a forget gate are:^[1][15]

${\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\{\tilde {c}}_{t}&=\sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\c_{t}&=f_{t}\odot c_{t-1}+i_{t}\odot {\tilde {c}}_{t}\\h_{t}&=o_{t}\odot \sigma _{h}(c_{t})\end{aligned}}}$

where the initial values are ${\displaystyle c_{0}=0}$ and ${\displaystyle h_{0}=0}$ and the operator ${\displaystyle \odot }$ denotes the Hadamard product (element-wise product). The subscript ${\displaystyle t}$ indexes the time step.

Variables

Letting the superscripts ${\displaystyle d}$ and ${\displaystyle h}$ refer to the number of input features and number of hidden units, respectively:

• ${\displaystyle x_{t}\in \mathbb {R} ^{d}}$: input vector to the LSTM unit
• ${\displaystyle f_{t}\in {(0,1)}^{h}}$: forget gate's activation vector
• ${\displaystyle i_{t}\in {(0,1)}^{h}}$: input/update gate's activation vector
• ${\displaystyle o_{t}\in {(0,1)}^{h}}$: output gate's activation vector
• ${\displaystyle h_{t}\in {(-1,1)}^{h}}$: hidden state vector also known as output vector of the LSTM unit
• ${\displaystyle {\tilde {c}}_{t}\in {(-1,1)}^{h}}$: cell input activation vector
• ${\displaystyle c_{t}\in \mathbb {R} ^{h}}$: cell state vector
• ${\displaystyle W\in \mathbb {R} ^{h\times d}}$, ${\displaystyle U\in \mathbb {R} ^{h\times h}}$ and ${\displaystyle b\in \mathbb {R} ^{h}}$: weight matrices and bias vector parameters which need to be learned during training

Activation functions

• ${\displaystyle \sigma _{g}}$: sigmoid function.
• ${\displaystyle \sigma _{c}}$:
  hyperbolic tangent
  function.
• ${\displaystyle \sigma _{h}}$: hyperbolic tangent function or, as the peephole LSTM paper^[18][19] suggests, ${\displaystyle \sigma _{h}(x)=x}$.

Peephole LSTM

The figure on the right is a graphical representation of an LSTM unit with peephole connections (i.e. a peephole LSTM).^[18][19] Peephole connections allow the gates to access the constant error carousel (CEC), whose activation is the cell state.^[18] ${\displaystyle h_{t-1}}$ is not used, ${\displaystyle c_{t-1}}$ is used instead in most places.

${\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}c_{t-1}+b_{o})\\c_{t}&=f_{t}\odot c_{t-1}+i_{t}\odot \sigma _{c}(W_{c}x_{t}+b_{c})\\h_{t}&=o_{t}\odot \sigma _{h}(c_{t})\end{aligned}}}$

Each of the gates can be thought as a "standard" neuron in a feed-forward (or multi-layer) neural network: that is, they compute an activation (using an activation function) of a weighted sum. ${\displaystyle i_{t},o_{t}}$ and ${\displaystyle f_{t}}$ represent the activations of respectively the input, output and forget gates, at time step ${\displaystyle t}$.

The 3 exit arrows from the memory cell ${\displaystyle c}$ to the 3 gates ${\displaystyle i,o}$ and ${\displaystyle f}$ represent the peephole connections. These peephole connections actually denote the contributions of the activation of the memory cell ${\displaystyle c}$ at time step ${\displaystyle t-1}$, i.e. the contribution of ${\displaystyle c_{t-1}}$ (and not ${\displaystyle c_{t}}$, as the picture may suggest). In other words, the gates ${\displaystyle i,o}$ and ${\displaystyle f}$ calculate their activations at time step ${\displaystyle t}$ (i.e., respectively, ${\displaystyle i_{t},o_{t}}$ and ${\displaystyle f_{t}}$) also considering the activation of the memory cell ${\displaystyle c}$ at time step ${\displaystyle t-1}$, i.e. ${\displaystyle c_{t-1}}$.

The single left-to-right arrow exiting the memory cell is not a peephole connection and denotes ${\displaystyle c_{t}}$.

The little circles containing a ${\displaystyle \times }$ symbol represent an element-wise multiplication between its inputs. The big circles containing an S-like curve represent the application of a differentiable function (like the sigmoid function) to a weighted sum.

Peephole convolutional LSTM

Peephole convolutional LSTM.^[20] The ${\displaystyle *}$ denotes the convolution operator.

${\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}*x_{t}+U_{f}*h_{t-1}+V_{f}\odot c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}*x_{t}+U_{i}*h_{t-1}+V_{i}\odot c_{t-1}+b_{i})\\c_{t}&=f_{t}\odot c_{t-1}+i_{t}\odot \sigma _{c}(W_{c}*x_{t}+U_{c}*h_{t-1}+b_{c})\\o_{t}&=\sigma _{g}(W_{o}*x_{t}+U_{o}*h_{t-1}+V_{o}\odot c_{t}+b_{o})\\h_{t}&=o_{t}\odot \sigma _{h}(c_{t})\end{aligned}}}$

Training

An RNN using LSTM units can be trained in a supervised fashion on a set of training sequences, using an optimization algorithm like gradient descent combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight.

A problem with using gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to ${\displaystyle \lim _{n\to \infty }W^{n}=0}$ if the spectral radius of ${\displaystyle W}$ is smaller than 1.^[2][21]

However, with LSTM units, when error values are back-propagated from the output layer, the error remains in the LSTM unit's cell. This "error carousel" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value.

CTC score function

Many applications use stacks of LSTM RNNs

Sometimes, it can be advantageous to train (parts of) an LSTM by neuroevolution^[24] or by policy gradient methods, especially when there is no "teacher" (that is, training labels).

Success

There have been several successful stories of training, in a non-supervised fashion, RNNs with LSTM units.

In 2018, Bill Gates called it a "huge milestone in advancing artificial intelligence" when bots developed by OpenAI were able to beat humans in the game of Dota 2.^[11] OpenAI Five consists of five independent but coordinated neural networks. Each network is trained by a policy gradient method without supervising teacher and contains a single-layer, 1024-unit Long-Short-Term-Memory that sees the current game state and emits actions through several possible action heads.^[11]

In 2018, OpenAI also trained a similar LSTM by policy gradients to control a human-like robot hand that manipulates physical objects with unprecedented dexterity.^[10]

In 2019,

Timeline of development

1989: Mike Mozer's work on focused back-propagation^[49] will later be cited by the main LSTM paper.^[1] Mozer's equation (3.1) anticipates aspects of LSTM cells: c_i(t+1) = d_i c_i(t) + f(x(t)), where c_i(t) is the activation of the i-th self-connected "context unit" at time step t, x(t) is the current input, f is a non-linear function, and d_i is a real-valued "decay weight" that can be learned. The residual connection in the "constant error carousel" of an LSTM cell simplifies this by setting d_i = 1.0: c_i(t+1) = c_i(t) + f(x(t)). The LSTM paper^[1] calls this "LSTM’s central feature," and states: "Note the similarity to Mozer’s fixed time constant system (1992) -- a time constant of 1.0 is appropriate for potentially infinite time lags."

1991:

1995: "Long Short-Term Memory (LSTM)" is published in a technical report by Sepp Hochreiter and Jürgen Schmidhuber.^[51]

1996: LSTM is published at NIPS'1996, a peer-reviewed conference.^[14]

1997: The main LSTM paper is published in the journal Neural Computation.^[1] By introducing Constant Error Carousel (CEC) units, LSTM deals with the vanishing gradient problem. The initial version of LSTM block included cells, input and output gates.^[52]

1999: Felix Gers, Jürgen Schmidhuber, and Fred Cummins introduced the forget gate (also called "keep gate") into the LSTM architecture,^[53] enabling the LSTM to reset its own state.^[52]

2000: Gers, Schmidhuber, and Cummins added peephole connections (connections from the cell to the gates) into the architecture.^[18][19] Additionally, the output activation function was omitted.^[52]

2001: Gers and Schmidhuber trained LSTM to learn languages unlearnable by traditional models such as Hidden Markov Models.^[18][54]

Hochreiter et al. used LSTM for meta-learning (i.e. learning a learning algorithm).^[55]

2004: First successful application of LSTM to speech Alex Graves et al.^[56][54]

2005: First publication (Graves and Schmidhuber) of LSTM with full backpropagation through time and of bi-directional LSTM.^[25][54]

2005: Daan Wierstra, Faustino Gomez, and Schmidhuber trained LSTM by neuroevolution without a teacher.^[24]

2006: Graves, Fernandez, Gomez, and Schmidhuber introduce a new error function for LSTM:

Mayer et al. trained LSTM to control robots.^[9]

2007: Wierstra, Foerster, Peters, and Schmidhuber trained LSTM by policy gradients for reinforcement learning without a teacher.^[60]

Hochreiter, Heuesel, and Obermayr applied LSTM to protein homology detection the field of biology.^[36]

2009: An LSTM trained by CTC won the

2009: Justin Bayer et al. introduced neural architecture search for LSTM.^[62][54]

2013: Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton used LSTM networks as a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset.^[27]

2014: Kyunghyun Cho et al. put forward a simplified variant of the forget gate LSTM^[53] called Gated recurrent unit (GRU).^[63]

2015: Google started using an LSTM trained by CTC for speech recognition on Google Voice.^[57][58] According to the official blog post, the new model cut transcription errors by 49%.^[64]

2015: Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber used LSTM principles^[53] to create the Highway network, a feedforward neural network with hundreds of layers, much deeper than previous networks.^[65][66][67] 7 months later, Kaiming He, Xiangyu Zhang; Shaoqing Ren, and Jian Sun won the ImageNet 2015 competition with an open-gated or gateless Highway network variant called Residual neural network.^[68] This has become the most cited neural network of the 21st century.^[67]

2016: Google started using an LSTM to suggest messages in the Allo conversation app.^[69] In the same year, Google released the Google Neural Machine Translation system for Google Translate which used LSTMs to reduce translation errors by 60%.^[6][70][71]

Apple announced in its Worldwide Developers Conference that it would start using the LSTM for quicktype^[72][73][74] in the iPhone and for Siri.^[75][76]

Amazon released Polly, which generates the voices behind Alexa, using a bidirectional LSTM for the text-to-speech technology.^[77]

2017: Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks.^[7]

Researchers from Michigan State University, IBM Research, and Cornell University published a study in the Knowledge Discovery and Data Mining (KDD) conference.^[78][79][80] Their Time-Aware LSTM (T-LSTM) performs better on certain data sets than standard LSTM.

Microsoft reported reaching 94.9% recognition accuracy on the Switchboard corpus, incorporating a vocabulary of 165,000 words. The approach used "dialog session-based long-short-term memory".^[59]

2018: OpenAI used LSTM trained by policy gradients to beat humans in the complex video game of Dota 2,^[11] and to control a human-like robot hand that manipulates physical objects with unprecedented dexterity.^[10][54]

2019:

2021: According to Google Scholar, in 2021, LSTM was cited over 16,000 times within a single year. This reflects applications of LSTM in many different fields including healthcare.^[13]

See also

References