History of artificial neural networks

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages) This article relies excessively on references to primary sources. Please improve this article by adding secondary or tertiary sources. Find sources: "History of artificial neural networks" – news · newspapers · books · scholar · JSTOR (August 2022) (Learn how and when to remove this message) This article needs to be updated. Please help update this article to reflect recent events or newly available information. (September 2021) (Learn how and when to remove this message)

Part of a series on
Machine learning and data mining
Paradigms Supervised learning Unsupervised learning Online learning Batch learning Meta-learning Semi-supervised learning Self-supervised learning Reinforcement learning Curriculum learning Rule-based learning Quantum machine learning
Problems Classification Generative modeling Regression Clustering Dimensionality reduction Density estimation Anomaly detection Data cleaning AutoML Association rules Semantic analysis Structured prediction Feature engineering Feature learning Learning to rank Grammar induction Ontology learning Multimodal learning
Supervised learning (classification • regression) Apprenticeship learning Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Fuzzy Expectation–maximization (EM) DBSCAN OPTICS Mean shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE SDL
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection RANSAC k-NN Local outlier factor Isolation forest
Artificial neural network Autoencoder Cognitive computing Deep learning DeepDream Feedforward neural network Recurrent neural network LSTM GRU ESN reservoir computing Restricted Boltzmann machine GAN Diffusion model SOM Convolutional neural network U-Net Transformer Vision Mamba Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
Reinforcement learning Q-learning SARSA Temporal difference (TD) Multi-agent Self-play
Learning with humans Active learning Crowdsourcing Human-in-the-loop RLHF
Model diagnostics Coefficient of determination Confusion matrix Learning curve ROC curve
Mathematical foundations Kernel machines Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
Machine-learning venues ECML PKDD NeurIPS ICML ICLR IJCAI ML JMLR
Related articles Glossary of artificial intelligence List of datasets for machine-learning research List of datasets in computer vision and image processing Outline of machine learning
v t e

Later, advances in hardware and the development of the

]

Linear neural network

The simplest kind of

Perceptrons and other early neural networks

In the early 1940s,

Some say that research stagnated following

First deep learning

The first

, where useless units in hidden layers are pruned with the help of a validation set.

The first deep learning MLP trained by stochastic gradient descent^[22] was published in 1967 by Shun'ichi Amari.^[23][9] In computer experiments conducted by Amari's student Saito, a five layer MLP with two modifiable layers learned useful

Backpropagation

The backpropagation algorithm is an efficient application of the Leibniz chain rule (1673)^[24] to networks of differentiable nodes.^[9] It is also known as the reverse mode of

Recurrent network architectures

Wilhelm Lenz and Ernst Ising created and analyzed the Ising model (1925)^[33] which is essentially a non-learning artificial recurrent neural network (RNN) consisting of neuron-like threshold elements.^[9] In 1972, Shun'ichi Amari made this architecture adaptive.^[34][9] His learning RNN was popularised by John Hopfield in 1982.^[35]

Self-organizing maps

.

SOMs create internal representations reminiscent of the cortical homunculus,^[39] a distorted representation of the human body, based on a neurological "map" of the areas and proportions of the human brain dedicated to processing sensory functions, for different parts of the body.

Convolutional neural networks (CNNs)

The origin of the CNN architecture is the "neocognitron"^[40] introduced by Kunihiko Fukushima in 1980.^[41][42] It was inspired by work of Hubel and Wiesel in the 1950s and 1960s which showed that cat visual cortices contain neurons that individually respond to small regions of the visual field. The neocognitron introduced the two basic types of layers in CNNs: convolutional layers, and downsampling layers. A convolutional layer contains units whose receptive fields cover a patch of the previous layer. The weight vector (the set of adaptive parameters) of such a unit is often called a filter. Units can share filters. Downsampling layers contain units whose receptive fields cover patches of previous convolutional layers. Such a unit typically computes the average of the activations of the units in its patch. This downsampling helps to correctly classify objects in visual scenes even when the objects are shifted.

In 1969, Kunihiko Fukushima also introduced the ReLU (rectified linear unit) activation function.^[43][9] The rectifier has become the most popular activation function for CNNs and deep neural networks in general.^[44]

The time delay neural network (TDNN) was introduced in 1987 by Alex Waibel and was one of the first CNNs, as it achieved shift invariance.^[45] It did so by utilizing weight sharing in combination with backpropagation training.^[46] Thus, while also using a pyramidal structure as in the neocognitron, it performed a global optimization of the weights instead of a local one.^[45]

In 1988, Wei Zhang et al. applied backpropagation to a CNN (a simplified Neocognitron with convolutional interconnections between the image feature layers and the last fully connected layer) for alphabet recognition. They also proposed an implementation of the CNN with an optical computing system.^[47][48]

In 1989, Yann LeCun et al. trained a CNN with the purpose of recognizing handwritten ZIP codes on mail. While the algorithm worked, training required 3 days.^[49] Learning was fully automatic, performed better than manual coefficient design, and was suited to a broader range of image recognition problems and image types. Subsequently, Wei Zhang, et al. modified their model by removing the last fully connected layer and applied it for medical image object segmentation in 1991^[50] and breast cancer detection in mammograms in 1994.^[51]

In 1990 Yamaguchi et al. introduced max-pooling, a fixed filtering operation that calculates and propagates the maximum value of a given region. They combined TDNNs with max-pooling in order to realize a speaker independent isolated word recognition system.^[52] In a variant of the neocognitron called the cresceptron, instead of using Fukushima's spatial averaging, J. Weng et al. also used max-pooling where a downsampling unit computes the maximum of the activations of the units in its patch.^{[53][54][55][56]} Max-pooling is often used in modern CNNs.^[57]

LeNet-5, a 7-level CNN by

: cheques) digitized in 32x32 pixel images. The ability to process higher-resolution images requires larger and more layers of CNNs, so this technique is constrained by the availability of computing resources.

In 2010, Backpropagation training through max-pooling was accelerated by GPUs and shown to perform better than other pooling variants.^[59] Behnke (2003) relied only on the sign of the gradient (

In 2011, a deep GPU-based CNN called "DanNet" by Dan Ciresan, Ueli Meier, and

ANNs were able to guarantee shift invariance to deal with small and large natural objects in large cluttered scenes, only when invariance extended beyond shift, to all ANN-learned concepts, such as location, type (object class label), scale, lighting and others. This was realized in Developmental Networks (DNs)[65] whose embodiments are Where-What Networks, WWN-1 (2008)^[66] through WWN-7 (2013).^[67]

Artificial curiosity and generative adversarial networks

In 1991,

In 2014, this adversarial principle was used in a generative adversarial network (GAN) by Ian Goodfellow et al.^[71] Here the environmental reaction is 1 or 0 depending on whether the first network's output is in a given set. This can be used to create realistic deepfakes.^[72]

In 1992, Schmidhuber also published another type of gradient-based adversarial neural networks where the goal of the zero-sum game is to create disentangled representations of input patterns. This was called predictability minimization.^[73][74]

Nvidia's StyleGAN (2018)^[75] is based on the Progressive GAN by Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.^[76] Here the GAN generator is grown from small to large scale in a pyramidal fashion. StyleGANs improve consistency between fine and coarse details in the generator network.

Transformers and their variants

Many modern large language models such as

Transformers have increasingly become the model of choice for natural language processing problems,^[78] replacing recurrent neural networks (RNNs) such as long short-term memory (LSTM).^[79]

Basic ideas for this go back a long way: in 1992,

which is also called the "linear Transformer."

Transformers are also increasingly being used in computer vision.^[84]

Deep learning with unsupervised or self-supervised pre-training

In the 1980s, backpropagation did not work well for deep FNNs and RNNs. Here the word "deep" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial credit assignment path (CAP) depth.^[85] The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For an FNN, the depth of the CAPs is that of the network and is the number of hidden layers plus one (as the output layer is also parameterized). For RNNs, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited.

To overcome this problem,

The deep architecture may be used to reproduce the original data from the top level feature activations.[86] The RNN hierarchy can be "collapsed" into a single RNN, by "distilling" a higher level "chunker" network into a lower level "automatizer" network.^[86][9] In 1993, a chunker solved a deep learning task whose CAP depth exceeded 1000.^[87] Such history compressors can substantially facilitate downstream supervised deep learning.^[9]

The vanishing gradient problem and its solutions

In 2015, Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber used

In 2011, Xavier Glorot, Antoine Bordes and Yoshua Bengio found that the ReLU^[43] of Kunihiko Fukushima also helps to overcome the vanishing gradient problem,^[99] compared to widely used activation functions prior to 2011.

Hardware-based designs

The development of

Computational devices were created in

in Schmidhuber's group showed that despite the vanishing gradient problem, GPUs make backpropagation feasible for many-layered feedforward neural networks.

Contests

Between 2009 and 2012, recurrent neural networks and deep feedforward neural networks developed in Schmidhuber's research group won eight international competitions in pattern recognition and machine learning.^[104][105] For example, the bi-directional and multi-dimensional long short-term memory (LSTM)^{[106][107][108][109]} of Graves et al. won three competitions in connected handwriting recognition at the 2009 International Conference on Document Analysis and Recognition (ICDAR), without any prior knowledge about the three languages to be learned.^[108][107]

Ciresan and colleagues won pattern recognition contests, including the IJCNN 2011 Traffic Sign Recognition Competition,^[110] the ISBI 2012 Segmentation of Neuronal Structures in Electron Microscopy Stacks challenge^[111] and others. Their neural networks were the first pattern recognizers to achieve human-competitive/superhuman performance^[62] on benchmarks such as traffic sign recognition (IJCNN 2012), or the MNIST handwritten digits problem.

Researchers demonstrated (2010) that deep neural networks interfaced to a hidden Markov model with context-dependent states that define the neural network output layer can drastically reduce errors in large-vocabulary speech recognition tasks such as voice search.^{[citation needed]}

GPU-based implementations

and others.

Deep, highly nonlinear neural architectures similar to the neocognitron^[113] and the "standard architecture of vision",^[114] inspired by simple and complex cells, were pre-trained with unsupervised methods by Hinton.^[90][89] A team from his lab won a 2012 contest sponsored by Merck to design software to help find molecules that might identify new drugs.^[115]

Notes

References

External links

• "Lecun 2019-7-11 ACM Tech Talk". Google Docs. Retrieved 2020-02-13.