Spiking neural network
This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages)
|
Spiking neural networks (SNNs) are
Although it was previously believed that the brain encoded information through spike rates, which can be considered as the analogue variable output of a traditional ANN,[3] research in the field of neurobiology has indicated that high speed processing cannot solely be performed through a rate based scheme. For example humans can perform an image recognition task at rate requiring no more than 10ms of processing time per neuron through the successive layers (going from the retina to the temporal lobe). This time window is too short for a rate based encoding. The precise spike timings in a small set of spiking neurons also has a higher information coding capacity compared with a rate based approach.[4]
The most prominent spiking neuron model is the leaky integrate-and-fire model.[5] In the integrate-and-fire model, the momentary activation level (modeled as a differential equation) is normally considered to be the neuron's state, with incoming spikes pushing this value higher or lower, until the state eventually either decays or—if the firing threshold is reached—the neuron fires. After firing, the state variable is reset to a lower value.
Various decoding methods exist for interpreting the outgoing
History
This section needs additional citations for verification. (December 2018) |
Many multi-layer artificial neural networks are fully connected, receiving input from every neuron in the previous layer and signalling every neuron in the subsequent layer. Although these networks have achieved breakthroughs in many fields, they are biologically inaccurate and do not mimic the operation mechanism of neurons in the brain of a living thing.[6]
The biologically inspired
Underpinnings
This article needs additional citations for verification. (November 2021) |
Information in the brain is represented as action potentials (neuron spikes), which may be grouped into spike trains or even coordinated waves of brain activity. A fundamental question of neuroscience is to determine whether neurons communicate by a
An SNN computes in the continuous rather than the discrete domain. The idea is that neurons may not test for activation in every iteration of propagation (as is the case in a typical multilayer perceptron network), but only when their membrane potentials reach a certain value. When a neuron is activated, it produces a signal that is passed to connected neurons, raising or lowering their membrane potential.
In a spiking neural network, a neuron's current state is defined as its membrane potential (possibly modeled as a differential equation). An input pulse causes the membrane potential to rise for a period of time and then gradually decline. Encoding schemes have been constructed to interpret these pulse sequences as a number, taking into account both pulse frequency and pulse interval. A neural network model based on pulse generation time can be established. Using the exact time of pulse occurrence, a neural network can employ more information and offer better computing properties.
The SNN approach produces a continuous output instead of the binary output of traditional
SNNs are theoretically more powerful than so called "second-generation networks" defined in[10] as "[ANNs] based on computational units that apply activation function with a continuous set of possible output values to a weighted sum (or polynomial) of the inputs; however, SNN training issues and hardware requirements limit their use. Although unsupervised biologically inspired learning methods are available such as Hebbian learning and STDP, no effective supervised training method is suitable for SNNs that can provide better performance than second-generation networks.[10] Spike-based activation of SNNs is not differentiable thus making it hard to develop gradient descent based training methods to perform error backpropagation.
SNNs have much larger computational costs for simulating realistic neural models than traditional ANNs.[11]
Pulse-coupled neural networks (PCNN) are often confused with SNNs. A PCNN can be seen as a kind of SNN.
Currently there are a few challenges when using SNNs that researchers are actively working on. The first challenge concerns the nondifferentiability of the spiking nonlinearity. The expressions for both the forward- and backward-learning methods contain the derivative of the neural activation function which is non-differentiable because neuron's output is either 1 when it spikes, and 0 otherwise. This all-or-nothing behavior of the binary spiking nonlinearity stops gradients from “flowing” and makes LIF neurons unsuitable for gradient-based optimization. The second challenge concerns the implementation of the optimization algorithm itself. Standard BP can be expensive in terms of computation, memory, and communication and may be poorly suited to the constraints dictated by the hardware that implements it (e.g., a computer, brain, or neuromorphic device).[12] Regarding the first challenge there are several approached in order to overcome it. A few of them are:
- resorting to entirely biologically inspired local learning rules for the hidden units
- translating conventionally trained “rate-based” NNs to SNNs
- smoothing the network model to be continuously differentiable
- defining an SG (Surogate Gradient) as a continuous relaxation of the real gradients
In the development of SNNs, incorporating additional neuron dynamics like Spike Frequency Adaptation (SFA) into neuron models marks a notable advance, enhancing both efficiency and computational power.[5][13] These neurons stand in between biological complexity and compuational complexity.[14] Originating from biological insights, SFA offers significant computational benefits by reducing power usage through efficient coding,[15] especially in cases of repetitive or intense stimuli. This adaptation improves signal clarity against background noise and introduces an elementary short-term memory at the neuron level, which in turn, refines the accuracy and efficiency of information processing.[16] Recently, This phenomenon is achieved mostly achieved using Compartmental neuron models. The simpler versions are of neuron models with adaptive thresholds, indirect way of achieving SFA, equips SNNs with improved learning capabilities, even with constrained synaptic plasticity, and elevates computational efficiency.[17][18] This feature lessens the demand on network layers by decreasing the need for spike processing, thus cutting down on computational load and memory access time—essential aspects of neural computation. Moreover, SNNs utilizing neurons capable of SFA achieve levels of accuracy that rival those of conventional artificial neural networks, including those based on long short-term memory models,[19][20] while also requiring fewer neurons for comparable computational tasks. This efficiency not only streamlines the computational workflow but also conserves space and energy, offering a pragmatic step forward in the practical application of SNNs for complex computing tasks, all while maintaining a commitment to technical integrity.
Applications
This section needs additional citations for verification. (December 2018) |
SNNs can in principle apply to the same applications as traditional ANNs.
As of 2019 SNNs lag behind ANNs in terms of accuracy, but the gap is decreasing, and has vanished on some tasks.[23]
When using SNNs for image based data we need to convert static images into binary spike trains coding.[24] Types of encodings:[25]
- Temporal coding generates one spike per neuron in which spike latency is inversely proportional to the pixel intensity.
- Rate coding converts pixel intensity into a spike train where the number of spikes is proportional to the pixel intensity.
- Direct coding uses a trainable layer to generate float value for each time-step. We have a learnable layer which converts each pixel at certain time step in float number and then threshold is used on the generated floating numbers to see if they will be 1 or 0.
- Phase coding encodes temporal information into spike patterns based on a global oscillator.
- Burst coding transmits the burst of spikes in a small-time duration, increasing the reliability of synaptic communication between neurons.
Software
This section needs additional citations for verification. (December 2018) |
A diverse range of application software can simulate SNNs. This software can be classified according to its uses:
SNN simulation
These simulate complex neural models with a high level of detail and accuracy. Large networks usually require lengthy processing. Candidates include:[26]
- École Normale Supérieure;
- Caltech;
- NEST – developed by the NEST Initiative;
- NEURON – mainly developed by Michael Hines, John W. Moore and Ted Carnevale in Yale University and Duke University;
- RAVSim (Runtime Tool) [28] – mainly developed by Sanaullah in Bielefeld University of Applied Sciences and Arts;
Hardware
This section needs additional citations for verification. (December 2018) |
Future neuromorphic architectures[29] will comprise billions of such nanosynapses, which require a clear understanding of the physical mechanisms responsible for plasticity. Experimental systems based on ferroelectric tunnel junctions have been used to show that STDP can be harnessed from heterogeneous polarization switching. Through combined scanning probe imaging, electrical transport and atomic-scale molecular dynamics, conductance variations can be modelled by nucleation-dominated reversal of domains. Simulations show that arrays of ferroelectric nanosynapses can autonomously learn to recognize patterns in a predictable way, opening the path towards unsupervised learning.[30]
- Akida is a completely digital event-based neural processing device with 1.2 million artificial neurons and 10 billion artificial synapses developed by BrainChip. Utilizing event-based possessing, it analyzes essential inputs at specific points. Results are stored in the on-chip memory units.
- Neurogrid is a board that can simulate spiking neural networks directly in hardware. (Stanford University)
- neuromorphic computing. Its primary purpose is pattern recognition. While critics say the chip isn't powerful enough, its supporters point out that this is only the first generation, and the capabilities of improved iterations will become clear. (IBM)[34]
Benchmarks
Classification capabilities of spiking networks trained according to unsupervised learning methods[35] have been tested on the common benchmark datasets, such as, Iris, Wisconsin Breast Cancer or Statlog Landsat dataset.[36][37] Various approaches to information encoding and network design have been used. For example, a 2-layer feedforward network for data clustering and classification. Based on the idea proposed in Hopfield (1995) the authors implemented models of local receptive fields combining the properties of radial basis functions (RBF) and spiking neurons to convert input signals (classified data) having a floating-point representation into a spiking representation.[38][39]
See also
References
- ^ ISSN 0893-6080.
- OCLC 57417395.
- S2CID 212638634.
- S2CID 207904985.
- ^ ISSN 2731-3395.
- ^ "Spiking Neural Networks, the Next Generation of Machine Learning". 16 July 2019.
- S2CID 50778421.
- ISBN 978-0-262-63221-8.
- ^ Van Wezel M (2020). A robust modular spiking neural networks training methodology for time-series datasets: With a focus on gesture control (Master of Science thesis). Delft University of Technology.
- ^ .
- PMID 27529195.
- .
- PMID 34310281.
- S2CID 7354646. Retrieved 2024-02-14.
- ^ Adibi, M., McDonald, J. S., Clifford, C. W. & Arabzadeh, E. Adaptation improves neural coding efficiency despite increasing correlations in variability. J. Neurosci. 33, 2108–2120 (2013)
- PMID 7303823.
- S2CID 14416573. Retrieved 2024-02-14.
- PMID 35884670.
- PMID 34244491.
- arXiv:1803.09574
- S2CID 16577867.
- S2CID 13992150.
- S2CID 5039751.
- PMID 35884670.
- arXiv:2202.03133 [cs.NE].
- S2CID 2048100.
- PMID 18249797.
- S2CID 259445644.
- ^ Sutton RS, Barto AG (2002) Reinforcement Learning: An Introduction. Bradford Books, MIT Press, Cambridge, MA.
- PMID 28368007.
- S2CID 2103654.
- ^ "Neuromorphic Computing". Human Brain Project.
- ^ "Hardware: Available Systems". Human Brain Project. Retrieved 2020-05-10.
- ^ Markoff J (8 August 2014). "A new chip functions like a brain, IBM says". The New York Times. p. B1.
- S2CID 12572538.
- ^ Newman D, Hettich S, Blake C, Merz C (1998). "UCI repository of machine learning databases".
- .
- S2CID 6379045.
- PMID 18244443.