Recurrent neural network
Part of a series on |
Machine learning and data mining |
---|
A recurrent neural network (RNN) is one of the two broad types of
Additional stored states and the storage under direct control by the network can be added to both
History
The Ising model (1925) by Wilhelm Lenz[9] and Ernst Ising[10][11] was the first RNN architecture that did not learn. Shun'ichi Amari made it adaptive in 1972.[12][13] This was also called the Hopfield network (1982). See also David Rumelhart's work in 1986.[14] In 1993, a neural history compressor system solved a "Very Deep Learning" task that required more than 1000 subsequent layers in an RNN unfolded in time.[15]
LSTM
Long short-term memory (LSTM) networks were invented by Hochreiter and Schmidhuber in 1997 and set accuracy records in multiple applications domains.[16]
Around 2007, LSTM started to revolutionize
LSTM also improved large-vocabulary speech recognition
LSTM broke records for improved
Architectures
RNNs come in many variants.
Fully recurrent
Fully recurrent neural networks (FRNN) connect the outputs of all neurons to the inputs of all neurons. This is the most general neural network topology because all other topologies can be represented by setting some connection weights to zero to simulate the lack of connections between those neurons. The illustration to the right may be misleading to many because practical neural network topologies are frequently organized in "layers" and the drawing gives that appearance. However, what appears to be layers are, in fact, different steps in time of the same fully recurrent neural network. The left-most item in the illustration shows the recurrent connections as the arc labeled 'v'. It is "unfolded" in time to produce the appearance of layers.
Elman networks and Jordan networks
An
Jordan networks are similar to Elman networks. The context units are fed from the output layer instead of the hidden layer. The context units in a Jordan network are also called the state layer. They have a recurrent connection to themselves.[29]
Elman and Jordan networks are also known as "Simple recurrent networks" (SRN).
Variables and functions
- : input vector
- : hidden layer vector
- : output vector
- , and : parameter matrices and vector
- and : Activation functions
Hopfield
The
Bidirectional associative memory
Introduced by Bart Kosko,[32] a bidirectional associative memory (BAM) network is a variant of a Hopfield network that stores associative data as a vector. The bi-directionality comes from passing information through a matrix and its transpose. Typically, bipolar encoding is preferred to binary encoding of the associative pairs. Recently, stochastic BAM models using Markov stepping were optimized for increased network stability and relevance to real-world applications.[33]
A BAM network has two layers, either of which can be driven as an input to recall an association and produce an output on the other layer.[34]
Echo state
Echo state networks (ESN) have a sparsely connected random hidden layer. The weights of output neurons are the only part of the network that can change (be trained). ESNs are good at reproducing certain time series.[35] A variant for spiking neurons is known as a liquid state machine.[36]
Independently RNN (IndRNN)
The independently recurrent neural network (IndRNN)[37] addresses the gradient vanishing and exploding problems in the traditional fully connected RNN. Each neuron in one layer only receives its own past state as context information (instead of full connectivity to all other neurons in this layer) and thus neurons are independent of each other's history. The gradient backpropagation can be regulated to avoid gradient vanishing and exploding in order to keep long or short-term memory. The cross-neuron information is explored in the next layers. IndRNN can be robustly trained with non-saturated nonlinear functions such as ReLU. Deep networks can be trained using skip connections.
Recursive
A
Neural history compressor
The neural history compressor is an unsupervised stack of RNNs.[43] At the input level, it learns to predict its next input from the previous inputs. Only unpredictable inputs of some RNN in the hierarchy become inputs to the next higher level RNN, which therefore recomputes its internal state only rarely. Each higher level RNN thus studies a compressed representation of the information in the RNN below. This is done such that the input sequence can be precisely reconstructed from the representation at the highest level.
The system effectively minimizes the description length or the negative logarithm of the probability of the data.[44] Given a lot of learnable predictability in the incoming data sequence, the highest level RNN can use supervised learning to easily classify even deep sequences with long intervals between important events.
It is possible to distill the RNN hierarchy into two RNNs: the "conscious" chunker (higher level) and the "subconscious" automatizer (lower level).[43] Once the chunker has learned to predict and compress inputs that are unpredictable by the automatizer, then the automatizer can be forced in the next learning phase to predict or imitate through additional units the hidden units of the more slowly changing chunker. This makes it easy for the automatizer to learn appropriate, rarely changing memories across long intervals. In turn, this helps the automatizer to make many of its once unpredictable inputs predictable, such that the chunker can focus on the remaining unpredictable events.[43]
A generative model partially overcame the vanishing gradient problem[45] of automatic differentiation or backpropagation in neural networks in 1992. In 1993, such a system solved a "Very Deep Learning" task that required more than 1000 subsequent layers in an RNN unfolded in time.[15]
Second order RNNs
Second-order RNNs use higher order weights instead of the standard weights, and states can be a product. This allows a direct mapping to a finite-state machine both in training, stability, and representation.[46][47] Long short-term memory is an example of this but has no such formal mappings or proof of stability.
Long short-term memory
Long short-term memory (LSTM) is a deep learning system that avoids the vanishing gradient problem. LSTM is normally augmented by recurrent gates called "forget gates".[48] LSTM prevents backpropagated errors from vanishing or exploding.[45] Instead, errors can flow backward through unlimited numbers of virtual layers unfolded in space. That is, LSTM can learn tasks[18] that require memories of events that happened thousands or even millions of discrete time steps earlier. Problem-specific LSTM-like topologies can be evolved.[49] LSTM works even given long delays between significant events and can handle signals that mix low and high-frequency components.
Many applications use stacks of LSTM RNNs[50] and train them by connectionist temporal classification (CTC)[51] to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.
LSTM can learn to recognize
Gated recurrent unit
Gated recurrent units (GRUs) are a gating mechanism in recurrent neural networks introduced in 2014. They are used in the full form and several simplified variants.[53][54] Their performance on polyphonic music modeling and speech signal modeling was found to be similar to that of long short-term memory.[55] They have fewer parameters than LSTM, as they lack an output gate.[56]
Bi-directional
Bi-directional RNNs use a finite sequence to predict or label each element of the sequence based on the element's past and future contexts. This is done by concatenating the outputs of two RNNs, one processing the sequence from left to right, and the other one from right to left. The combined outputs are the predictions of the teacher-given target signals. This technique has been proven to be especially useful when combined with LSTM RNNs.[57][58]
Continuous-time
A continuous-time recurrent neural network (CTRNN) uses a system of
For a neuron in the network with activation , the rate of change of activation is given by:
Where:
- : Time constant of postsynaptic node
- : Activation of postsynaptic node
- : Rate of change of activation of postsynaptic node
- : Weight of connection from pre to postsynaptic node
- : Sigmoid of x e.g. .
- : Activation of presynaptic node
- : Bias of presynaptic node
- : Input (if any) to node
CTRNNs have been applied to evolutionary robotics where they have been used to address vision,[59] co-operation,[60] and minimal cognitive behaviour.[61]
Note that, by the
Hierarchical recurrent neural network
This section needs expansion. You can help by adding to it. (August 2019) |
Hierarchical recurrent neural networks (HRNN) connect their neurons in various ways to decompose hierarchical behavior into useful subprograms.[43][63] Such hierarchical structures of cognition are present in theories of memory presented by philosopher Henri Bergson, whose philosophical views have inspired hierarchical models.[64]
Hierarchical recurrent neural networks are useful in forecasting, helping to predict disaggregated inflation components of the consumer price index (CPI). The HRNN model leverages information from higher levels in the CPI hierarchy to enhance lower-level predictions. Evaluation of a substantial dataset from the US CPI-U index demonstrates the superior performance of the HRNN model compared to various established inflation prediction methods.[65]
Recurrent multilayer perceptron network
Generally, a recurrent multilayer perceptron network (RMLP network) consists of cascaded subnetworks, each containing multiple layers of nodes. Each subnetwork is feed-forward except for the last layer, which can have feedback connections. Each of these subnets is connected only by feed-forward connections.[66]
Multiple timescales model
A multiple timescales recurrent neural network (MTRNN) is a neural-based computational model that can simulate the functional hierarchy of the brain through self-organization depending on the spatial connection between neurons and on distinct types of neuron activities, each with distinct time properties.[67][68] With such varied neuronal activities, continuous sequences of any set of behaviors are segmented into reusable primitives, which in turn are flexibly integrated into diverse sequential behaviors. The biological approval of such a type of hierarchy was discussed in the memory-prediction theory of brain function by Hawkins in his book On Intelligence.[citation needed] Such a hierarchy also agrees with theories of memory posited by philosopher Henri Bergson, which have been incorporated into an MTRNN model.[64][69]
Neural Turing machines
Neural Turing machines (NTMs) are a method of extending recurrent neural networks by coupling them to external memory resources which they can interact with by attentional processes. The combined system is analogous to a Turing machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent.[70]
Differentiable neural computer
Differentiable neural computers (DNCs) are an extension of Neural Turing machines, allowing for the usage of fuzzy amounts of each memory address and a record of chronology.
Neural network pushdown automata
Neural network pushdown automata (NNPDA) are similar to NTMs, but tapes are replaced by analog stacks that are differentiable and trained. In this way, they are similar in complexity to recognizers of
Memristive networks
Greg Snider of
Pseudocode
Given a time series x
of length sequence_length
.
In the recurrent neural network, there is a loop that processes all entries of the time series x
through the layers neural_network
one after another. These have as return value in each time step i
both the prediction y_pred[i]
and an updated hidden state hidden
, which has the length hidden_size
. As a result, after the loop, the collection of all predictions y_pred
is returned.
The following pseudocode (based on the programming language Python) illustrates the functionality of a recurrent neural network.[74]
def RNN_forward(x, sequence_length, neural_network, hidden_size):
hidden = zeros(size=hidden_size) # initialize with zeros for each independent time series separately
y_pred = zeros(size=sequence_length)
for i in range(sequence_length):
y_pred[i], hidden = neural_network(x[i], hidden) # update hidden state
return y_pred
Modern libraries provide runtime-optimized implementations of the above functionality or allow to speed up the slow loop by just-in-time compilation.
Training
Gradient descent
Gradient descent is a
The standard method is called "
In this context, local in space means that a unit's weight vector can be updated using only information stored in the connected units and the unit itself such that update complexity of a single unit is linear in the dimensionality of the weight vector. Local in time means that the updates take place continually (on-line) and depend only on the most recent time step rather than on multiple time steps within a given time horizon as in BPTT. Biological neural networks appear to be local with respect to both time and space.[79][80]
For recursively computing the partial derivatives, RTRL has a time-complexity of O(number of hidden x number of weights) per time step for computing the
A major problem with gradient descent for standard RNN architectures is that error gradients vanish exponentially quickly with the size of the time lag between important events.[45][85] LSTM combined with a BPTT/RTRL hybrid learning method attempts to overcome these problems.[16] This problem is also solved in the independently recurrent neural network (IndRNN)[37] by reducing the context of a neuron to its own past state and the cross-neuron information can then be explored in the following layers. Memories of different ranges including long-term memory can be learned without the gradient vanishing and exploding problem.
The on-line algorithm called causal recursive backpropagation (CRBP), implements and combines BPTT and RTRL paradigms for locally recurrent networks.[86] It works with the most general locally recurrent networks. The CRBP algorithm can minimize the global error term. This fact improves the stability of the algorithm, providing a unifying view of gradient calculation techniques for recurrent networks with local feedback.
One approach to gradient information computation in RNNs with arbitrary architectures is based on signal-flow graphs diagrammatic derivation.[87] It uses the BPTT batch algorithm, based on Lee's theorem for network sensitivity calculations.[88] It was proposed by Wan and Beaufays, while its fast online version was proposed by Campolucci, Uncini and Piazza.[88]
Global optimization methods
Training the weights in a neural network can be modeled as a non-linear global optimization problem. A target function can be formed to evaluate the fitness or error of a particular weight vector as follows: First, the weights in the network are set according to the weight vector. Next, the network is evaluated against the training sequence. Typically, the sum-squared difference between the predictions and the target values specified in the training sequence is used to represent the error of the current weight vector. Arbitrary global optimization techniques may then be used to minimize this target function.
The most common global optimization method for training RNNs is genetic algorithms, especially in unstructured networks.[89][90][91]
Initially, the genetic algorithm is encoded with the neural network weights in a predefined manner where one gene in the chromosome represents one weight link. The whole network is represented as a single chromosome. The fitness function is evaluated as follows:
- Each weight encoded in the chromosome is assigned to the respective weight link of the network.
- The training set is presented to the network which propagates the input signals forward.
- The mean-squared error is returned to the fitness function.
- This function drives the genetic selection process.
Many chromosomes make up the population; therefore, many different neural networks are evolved until a stopping criterion is satisfied. A common stopping scheme is:
- When the neural network has learned a certain percentage of the training data or
- When the minimum value of the mean-squared-error is satisfied or
- When the maximum number of training generations has been reached.
The fitness function evaluates the stopping criterion as it receives the mean-squared error reciprocal from each network during training. Therefore, the goal of the genetic algorithm is to maximize the fitness function, reducing the mean-squared error.
Other global (and/or evolutionary) optimization techniques may be used to seek a good set of weights, such as simulated annealing or particle swarm optimization.
Related fields and models
RNNs may behave chaotically. In such cases, dynamical systems theory may be used for analysis.
They are in fact recursive neural networks with a particular structure: that of a linear chain. Whereas recursive neural networks operate on any hierarchical structure, combining child representations into parent representations, recurrent neural networks operate on the linear progression of time, combining the previous time step and a hidden representation into the representation for the current time step.
In particular, RNNs can appear as nonlinear versions of finite impulse response and infinite impulse response filters and also as a nonlinear autoregressive exogenous model (NARX).[92]
The effect of memory-based learning for the recognition of sequences can also be implemented by a more biological-based model which uses the silencing mechanism exhibited in neurons with a relatively high frequency spiking activity.[93]
Libraries
- Apache Singa
- Caffe: Created by the Berkeley Vision and Learning Center (BVLC). It supports both CPU and GPU. Developed in C++, and has Python and MATLAB wrappers.
- Chainer: Fully in Python, production support for CPU, GPU, distributed training.
- Deeplearning4j: Deep learning in Java and Scala on multi-GPU-enabled Spark.
- Flux: includes interfaces for RNNs, including GRUs and LSTMs, written in Julia.
- Keras: High-level API, providing a wrapper to many other deep learning libraries.
- Microsoft Cognitive Toolkit
- MXNet: an open-source deep learning framework used to train and deploy deep neural networks.
- PyTorch: Tensors and Dynamic neural networks in Python with GPU acceleration.
- TPU,[94]mobile
- Theano: A deep-learning library for Python with an API largely compatible with the NumPy library.
- Torch: A scientific computing framework with support for machine learning algorithms, written in C and Lua.
Applications
Applications of recurrent neural networks include:
- Machine translation[25]
- Robot control[95]
- Speech recognition[99][17][100]
- Speech synthesis[101]
- Brain–computer interfaces[102]
- Time series anomaly detection[103]
- Text-to-Video model[104]
- Rhythm learning[105]
- Music composition[106]
- Grammar learning[107][52][108]
- Handwriting recognition[109][110]
- Human action recognition[111]
- Protein homology detection[112]
- Predicting subcellular localization of proteins[58]
- Several prediction tasks in the area of business process management[113]
- Prediction in medical care pathways[114]
- Predictions of fusion plasma disruptions in reactors (Fusion Recurrent Neural Network (FRNN) code) [115]
References
- ^ Dupond, Samuel (2019). "A thorough review on the current advance of neural network structures". Annual Reviews in Control. 14: 200–230.
- PMID 30519653.
- ISSN 2314-7288.
- S2CID 14635907.
- ^ a b Sak, Haşim; Senior, Andrew; Beaufays, Françoise (2014). "Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling" (PDF). Google Research.
- ^ ].
- ^ Miljanovic, Milos (Feb–Mar 2012). "Comparative analysis of Recurrent and Finite Impulse Response Neural Networks in Time Series Prediction" (PDF). Indian Journal of Computer and Engineering. 3 (1).
- ^ Hyötyniemi, Heikki (1996). "Turing machines are recurrent neural networks". Proceedings of STeP '96/Publications of the Finnish Artificial Intelligence Society: 13–24.
- ^ Lenz, W. (1920), "Beiträge zum Verständnis der magnetischen Eigenschaften in festen Körpern", Physikalische Zeitschrift, 21: 613–615.
- S2CID 122157319
- .
- S2CID 3708480.
- arXiv:2212.11279 [cs.NE].
- S2CID 205001834.
- ^ a b Schmidhuber, Jürgen (1993). Habilitation thesis: System modeling and optimization (PDF). Page 150 ff demonstrates credit assignment across the equivalent of 1,200 layers in an unfolded RNN.
- ^ S2CID 1915014.
- ^ ISBN 978-3-540-74693-5.
- ^ S2CID 11715509.
- ^ Graves, Alex; Schmidhuber, Jürgen (2009). "Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks". In Koller, D.; Schuurmans, D.; Bengio, Y.; Bottou, L. (eds.). Advances in Neural Information Processing Systems. Vol. 21. Neural Information Processing Systems (NIPS) Foundation. pp. 545–552.
- ^ "2000 HUB5 English Evaluation Speech - Linguistic Data Consortium". catalog.ldc.upenn.edu.
- ].
- ISBN 978-1-4673-6997-8.
- ISBN 978-1-4673-6997-8.
- ^ Sak, Haşim; Senior, Andrew; Rao, Kanishka; Beaufays, Françoise; Schalkwyk, Johan (September 2015). "Google voice search: faster and more accurate".
- ^ Bibcode:2014arXiv1409.3215S.
- arXiv:1602.02410 [cs.CL].
- arXiv:1512.00103 [cs.CL].
- ].
- ^ a b Cruse, Holk; Neural Networks as Cybernetic Systems, 2nd and revised edition
- .
- S2CID 15375627.
- S2CID 59875735.
- .
- ISBN 978-3-540-60505-8.
- S2CID 2184251.
- S2CID 1045112.
- ^ arXiv:1803.04831 [cs.CV].
- S2CID 6536466.
- ^ Linnainmaa, Seppo (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors (MSc) (in Finnish). University of Helsinki.
- ISBN 978-0-89871-776-1.
- ^ Socher, Richard; Lin, Cliff; Ng, Andrew Y.; Manning, Christopher D., "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" (PDF), 28th International Conference on Machine Learning (ICML 2011)
- ^ Socher, Richard; Perelygin, Alex; Wu, Jean Y.; Chuang, Jason; Manning, Christopher D.; Ng, Andrew Y.; Potts, Christopher. "Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank" (PDF). Emnlp 2013.
- ^ S2CID 18271205.
- .
- ^ a b c Hochreiter, Sepp (1991). Untersuchungen zu dynamischen neuronalen Netzen (PDF) (Diploma). Institut f. Informatik, Technische University Munich.
- S2CID 19666035.
- S2CID 228941.
- ^ Gers, Felix A.; Schraudolph, Nicol N.; Schmidhuber, Jürgen (2002). "Learning Precise Timing with LSTM Recurrent Networks" (PDF). Journal of Machine Learning Research. 3: 115–143. Retrieved 2017-06-13.
- ISBN 978-3-642-04276-8.
- CiteSeerX 10.1.1.79.1887.
- ISBN 1-59593-383-2.
- ^ S2CID 10192330.
- arXiv:1701.03452 [cs.NE].
- arXiv:1701.05923 [cs.NE].
- ].
- ^ Britz, Denny (October 27, 2015). "Recurrent Neural Network Tutorial, Part 4 – Implementing a GRU/LSTM RNN with Python and Theano – WildML". Wildml.com. Retrieved May 18, 2016.
- S2CID 1856462.
- ^ S2CID 11787259.
- ^ Harvey, Inman; Husbands, Phil; Cliff, Dave (1994), "Seeing the light: Artificial evolution, real vision", 3rd international conference on Simulation of adaptive behavior: from animals to animats 3, pp. 392–401
- ISBN 978-3-540-42567-0.
- .
- ^ Sherstinsky, Alex (2018-12-07). Bloem-Reddy, Benjamin; Paige, Brooks; Kusner, Matt; Caruana, Rich; Rainforth, Tom; Teh, Yee Whye (eds.). Deriving the Recurrent Neural Network Definition and RNN Unrolling Using Signal Processing. Critiquing and Correcting Trends in Machine Learning Workshop at NeurIPS-2018.
- S2CID 9932565.
- ^ a b "Burns, Benureau, Tani (2018) A Bergson-Inspired Adaptive Time Constant for the Multiple Timescales Recurrent Neural Network Model. JNNS".
- .
- CiteSeerX 10.1.1.45.3527.
- PMID 18989398.
- PMID 23423881.
- ^ "Proceedings of the 28th Annual Conference of the Japanese Neural Network Society (October, 2018)" (PDF).
- ].
- ISBN 978-3-540-64341-8.
- ^ Snider, Greg (2008), "Cortical computing with memristive nanodevices", Sci-DAC Review, 10: 58–65
- S2CID 6758362.
- ISBN 978-1-63835-078-1.
- S2CID 205001834.
- ^ Rumelhart, David E. (1985). Learning Internal Representations by Error Propagation. San Diego (CA): Institute for Cognitive Science, University of California.
- ^ Robinson, Anthony J.; Fallside, Frank (1987). The Utility Driven Dynamic Error Propagation Network. Technical Report CUED/F-INFENG/TR.1. Department of Engineering, University of Cambridge.
- ISBN 978-1-134-77581-1.
- S2CID 18721007.
- ISBN 978-0-471-35167-2.
- arXiv:1507.07680 [cs.NE].
- S2CID 11761172.
- ^ Williams, Ronald J. (1989). Complexity of exact gradient computation algorithms for recurrent neural networks (Report). Technical Report NU-CCS-89-27. Boston (MA): Northeastern University, College of Computer Science. Archived from the original on 2017-10-20. Retrieved 2017-07-02.
- S2CID 16813485.
- ISBN 978-0-7803-5369-5.
- PMID 18252525.
- S2CID 15512077.
- ^ S2CID 15090951.
- ^ Gomez, Faustino J.; Miikkulainen, Risto (1999), "Solving non-Markovian control tasks with neuroevolution" (PDF), IJCAI 99, Morgan Kaufmann, retrieved 5 August 2017
- ^ Syed, Omar (May 1995). Applying Genetic Algorithms to Recurrent Neural Networks for Learning Network Parameters and Architecture (MSc). Department of Electrical Engineering, Case Western Reserve University.
- ^ Gomez, Faustino J.; Schmidhuber, Jürgen; Miikkulainen, Risto (June 2008). "Accelerated Neural Evolution Through Cooperatively Coevolved Synapses" (PDF). Journal of Machine Learning Research. 9: 937–965.
- PMID 18255858.
- PMID 36175466.
- ^ Metz, Cade (May 18, 2016). "Google Built Its Very Own Chips to Power Its AI Bots". Wired.
- S2CID 12284900.
- OCLC 62330637.
- arXiv:1901.00069 [cs.LG].
- S2CID 202540863.
- S2CID 1856462.
- S2CID 206741496.
- S2CID 129946122.
- PMID 34260835.
- ISBN 978-2-87587-015-5.
- ^ "Papers with Code - DeepHS-HDRVideo: Deep High Speed High Dynamic Range Video Reconstruction". paperswithcode.com. Retrieved 2022-10-13.
- ^ Gers, Felix A.; Schraudolph, Nicol N.; Schmidhuber, Jürgen (2002). "Learning precise timing with LSTM recurrent networks" (PDF). Journal of Machine Learning Research. 3: 115–143.
- ISBN 978-3-540-46084-8.
- S2CID 30459046.
- PMID 12628609.
- ^ Graves, Alex; Schmidhuber, Jürgen (2009). "Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks" (PDF). Advances in Neural Information Processing Systems. Vol. 22, NIPS'22. MIT Press. pp. 545–552.
- ISBN 978-1-60560-352-0.
- ISBN 978-3-642-25445-1.
- PMID 17488755.
- S2CID 2192354.
- PMID 28286600.
- ^ "Artificial intelligence helps accelerate progress toward efficient fusion reactions". Princeton University. Retrieved 2023-06-12.
Further reading
- Mandic, Danilo P.; Chambers, Jonathon A. (2001). Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. Wiley. ISBN 978-0-471-49517-8.
External links
- Recurrent Neural Networks with over 60 RNN papers by Jürgen Schmidhuber's group at Dalle Molle Institute for Artificial Intelligence Research
- Elman Neural Network implementation for WEKA