Connectionism
Connectionism (coined by Edward Thorndike in the 1930s[citation needed]) is the name of an approach to the study of human mental processes and cognition that utilizes mathematical models known as connectionist networks or artificial neural networks.[1] Connectionism has had many 'waves' since its beginnings.
The first wave appeared 1943 with Warren Sturgis McCulloch and Walter Pitts both focusing on comprehending neural circuitry through a formal and mathematical approach,[2] and Frank Rosenblatt who published the 1958 book “The Perceptron: A Probabilistic Model For Information Storage and Organization in the Brain” in Psychological Review, while working at the Cornell Aeronautical Laboratory.[3] The first wave ended with the 1969 book about the limitations of the original perceptron idea, written by
The second wave blossomed in the late 1980s, following the 1987 book about Parallel Distributed Processing by
The current (third) wave has been marked by advances in
Basic principle
The central connectionist principle is that mental phenomena can be described by interconnected networks of simple and often uniform units. The form of the connections and the units can vary from model to model. For example, units in the network could represent
Activation function
Internal states of any network change over time due to neurons sending a signal to a succeeding layer of neurons in the case of a feedforward network, or to a previous layer in the case of a recurrent network. Discovery of non-linear activation functions has enabled the second wave of connectionism.
Memory and learning
Neural networks follow two basic principles:
- Any mental state can be described as an (N)-dimensional vectorof numeric activation values over neural units in a network.
- Memory and learning are created by modifying the 'weights' of the connections between neural units, generally represented as an N×M matrix. The weights are adjusted according to some learning rule or algorithm, such as Hebbian learning.[9]
Most of the variety among the models comes from:
- Interpretation of units: Units can be interpreted as neurons or groups of neurons.
- Definition of activation: Activation can be defined in a variety of ways. For example, in a Boltzmann machine, the activation is interpreted as the probability of generating an action potential spike, and is determined via a logistic function on the sum of the inputs to a unit.
- Learning algorithm: Different networks modify their connections differently. In general, any mathematically defined change in connection weights over time is referred to as the "learning algorithm".
Biological realism
Connectionist work in general does not need to be biologically realistic.
Precursors
Precursors of the connectionist principles can be traced to early work in psychology, such as that of William James.[20] Psychological theories based on knowledge about the human brain were fashionable in the late 19th century. As early as 1869, the neurologist John Hughlings Jackson argued for multi-level, distributed systems. Following from this lead, Herbert Spencer's Principles of Psychology, 3rd edition (1872), and Sigmund Freud's Project for a Scientific Psychology (composed 1895) propounded connectionist or proto-connectionist theories. These tended to be speculative theories. But by the early 20th century, Edward Thorndike was experimenting on learning that posited a connectionist type network.
Hopfield networks had precursors in the Ising model due to Wilhelm Lenz (1920) and Ernst Ising (1925), though the Ising model conceived by them did not involve time. Monte Carlo simulations of Ising model required the advent of computers in the 1950s.[21]
The first wave
The first wave begun in 1943 with Warren Sturgis McCulloch and Walter Pitts both focusing on comprehending neural circuitry through a formal and mathematical approach.[3] McCulloch and Pitts showed how neural systems could implement first-order logic: Their classic paper "A Logical Calculus of Ideas Immanent in Nervous Activity" (1943) is important in this development here. They were influenced by the work of Nicolas Rashevsky in the 1930s.
The Perceptron machines were proposed and built by Frank Rosenblatt, who published the 1958 paper “The Perceptron: A Probabilistic Model For Information Storage and Organization in the Brain” in Psychological Review, while working at the Cornell Aeronautical Laboratory. He cited Hebb, Hayek, Uttley, and Ashby as main influences.
Another form of connectionist model was the
The research group led by Widrow empirically searched for methods to train two-layered ADALINE networks, with limited success.[25][26]
A method to train multilayered perceptrons with arbitrary levels of trainable weights was published by
The first multilayered perceptrons trained by
The second wave
The second wave begun in late 1980s, following the 1987 two-volume book about the Parallel Distributed Processing (PDP) by
Connectionism vs. computationalism debate
As connectionism became increasingly popular in the late 1980s, some researchers (including
Connectionism and computationalism need not be at odds, but the debate in the late 1980s and early 1990s led to opposition between the two approaches. Throughout the debate, some researchers have argued that connectionism and computationalism are fully compatible, though full consensus on this issue has not been reached. Differences between the two approaches include the following:
- Computationalists posit symbolic models that are structurally similar to underlying brain structure, whereas connectionists engage in "low-level" modeling, trying to ensure that their models resemble neurological structures.
- Computationalists in general focus on the structure of explicit symbols (syntacticalrules for their internal manipulation, whereas connectionists focus on learning from environmental stimuli and storing this information in a form of connections between neurons.
- Computationalists believe that internal mental activity consists of manipulation of explicit symbols, whereas connectionists believe that the manipulation of explicit symbols provides a poor model of mental activity.
- Computationalists often posit domain specific symbolic sub-systems designed to support learning in specific areas of cognition (e.g., language, intentionality, number), whereas connectionists posit one or a small set of very general learning-mechanisms.
Despite these differences, some theorists have proposed that the connectionist architecture is simply the manner in which organic brains happen to implement the symbol-manipulation system. This is logically possible, as it is well known that connectionist models can implement symbol-manipulation systems of the kind used in computationalist models,[33] as indeed they must be able if they are to explain the human ability to perform symbol-manipulation tasks. Several cognitive models combining both symbol-manipulative and connectionist architectures have been proposed. Among them are Paul Smolensky's Integrated Connectionist/Symbolic Cognitive Architecture (ICS).[8][34] and Ron Sun's CLARION (cognitive architecture). But the debate rests on whether this symbol manipulation forms the foundation of cognition in general, so this is not a potential vindication of computationalism. Nonetheless, computational descriptions may be helpful high-level descriptions of cognition of logic, for example.
The debate was largely centred on logical arguments about whether connectionist networks could produce the syntactic structure observed in this sort of reasoning. This was later achieved although using fast-variable binding abilities outside of those standardly assumed in connectionist models.[33][35]
Part of the appeal of computational descriptions is that they are relatively easy to interpret, and thus may be seen as contributing to our understanding of particular mental processes, whereas connectionist models are in general more opaque, to the extent that they may be describable only in very general terms (such as specifying the learning algorithm, the number of units, etc.), or in unhelpfully low-level terms. In this sense, connectionist models may instantiate, and thereby provide evidence for, a broad theory of cognition (i.e., connectionism), without representing a helpful theory of the particular process that is being modelled. In this sense, the debate might be considered as to some extent reflecting a mere difference in the level of analysis in which particular theories are framed. Some researchers suggest that the analysis gap is the consequence of connectionist mechanisms giving rise to emergent phenomena that may be describable in computational terms.[36]
In the 2000s, the popularity of
In 2014,
Symbolism vs. connectionism debate
Smolensky's Subsymbolic Paradigm[40][41] has to meet the Fodor-Pylyshyn challenge[42][43][44][45] formulated by classical symbol theory for a convincing theory of cognition in modern connectionism. In order to be an adequate alternative theory of cognition, Smolensky's Subsymbolic Paradigm would have to explain the existence of systematicity or systematic relations in language cognition without the assumption that cognitive processes are causally sensitive to the classical constituent structure of mental representations. The subsymbolic paradigm, or connectionism in general, would thus have to explain the existence of systematicity and compositionality without relying on the mere implementation of a classical cognitive architecture. This challenge implies a dilemma: If the Subsymbolic Paradigm could contribute nothing to the systematicity and compositionality of mental representations, it would be insufficient as a basis for an alternative theory of cognition. However, if the Subsymbolic Paradigm's contribution to systematicity requires mental processes grounded in the classical constituent structure of mental representations, the theory of cognition it develops would be, at best, an implementation architecture of the classical model of symbol theory and thus not a genuine alternative (connectionist) theory of cognition.[46] The classical model of symbolism is characterized by (1) a combinatorial syntax and semantics of mental representations and (2) mental operations as structure-sensitive processes, based on the fundamental principle of syntactic and semantic constituent structure of mental representations as used in Fodor's "Language of Thought (LOT)".[47][48] This can be used to explain the following closely related properties of human cognition, namely its (1) productivity, (2) systematicity, (3) compositionality, and (4) inferential coherence.[49]
This challenge has been met in modern connectionism, for example, not only by Smolensky's "Integrated Connectionist/Symbolic (ICS) Cognitive Architecture",[50][51] but also by Werning and Maye's "Oscillatory Networks".[52][53][54] An overview of this is given for example by Bechtel & Abrahamsen,[55] Marcus[56] and Maurer.[57]
See also
Notes
- ^ "Internet Encyclopedia of Philosophy". iep.utm.edu. Retrieved 2023-08-19.
- ISSN 1522-9602.
- ^ S2CID 201061823.
- ISBN 978-0-262-63268-3.
- S2CID 201061823.
- ISBN 978-0-262-63268-3.
- .
- ^ a b c Garson, James (27 November 2018). Zalta, Edward N. (ed.). The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University – via Stanford Encyclopedia of Philosophy.
- ISSN 1134-3478.
- ^ "Encephalos Journal". www.encephalos.gr. Retrieved 2018-02-20.
- ISBN 978-1-317-95876-5.
- S2CID 15349751. Retrieved 29 December 2023.
- PMID 23970869.
- ^ Tieszen, R. (2011). "Analytic and Continental Philosophy, Science, and Global Philosophy". Comparative Philosophy. 2 (2): 4–22. Retrieved 29 December 2023.
- ISBN 978-0-7503-0455-9.
- ISBN 978-0-444-59876-9.
- S2CID 5892527.
- S2CID 205001834.
- S2CID 85501792.
- ISBN 978-0-262-51048-6.
- .
- ^ Hayek, Friedrich A. [1920] 1991. Beiträge zur Theorie der Entwicklung des Bewusstseins [Contributions to a theory of how consciousness develops]. Manuscript, translated by Grete Heinz.
- S2CID 144437624.
- ^ Hayek, F. A. (2012-09-15). The Sensory Order: An Inquiry into the Foundations of Theoretical Psychology (1st ed.). The University of Chicago Press.
- ^ pp 124-129, Olazaran Rodriguez, Jose Miguel. A historical sociology of neural network research. PhD Dissertation. University of Edinburgh, 1991.
- ^ Widrow, B. (1962) Generalization and information storage in networks of ADALINE "neurons". In M. C. Yovits, G. T. Jacobi, & G. D. Goldstein (Ed.), Self-Organizing Svstems-1962 (pp. 435-461). Washington, DC: Spartan Books.
- ^ arXiv:2212.11279 [cs.NE].
- ^ Ivakhnenko, A. G. (1973). Cybernetic Predicting Devices. CCM Information Corporation.
- ^ Ivakhnenko, A. G.; Grigorʹevich Lapa, Valentin (1967). Cybernetics and forecasting techniques. American Elsevier Pub. Co.
- ^ Amari, Shun-Ichi (1972). "Learning patterns and pattern sequences by self-organizing nets of threshold elements". IEEE Transactions. C (21): 1197–1206.
- .
- ^ Amari, Shun'ichi (1967). "A theory of adaptive pattern classifier". IEEE Transactions. EC (16): 279–307.
- ^ ISSN 1551-6709.
- .
- S2CID 14973656.
- .
- PMID 10097022, retrieved 28 May 2022
- S2CID 16515284.
- ].
- ^ P. Smolensky: On the proper treatment of connectionism. In: Behavioral and Brain Sciences. Band 11, 1988, S. 1-74.
- ^ P. Smolensky: The constituent structure of connectionist mental states: a reply to Fodor and Pylyshyn. In: T. Horgan, J. Tienson (Hrsg.): Spindel Conference 1987: Connectionism and the Philosophy of Mind. The Southern Journal of Philosophy. Special Issue on Connectionism and the Foundations of Cognitive Science. Supplement. Band 26, 1988, S. 137-161.
- ^ J.A. Fodor, Z.W. Pylyshyn: Connectionism and cognitive architecture: a critical analysis. Cognition. Band 28, 1988, S. 12-13, 33-50.
- ^ J.A. Fodor, B. McLaughlin: Connectionism and the problem of systematicity: why Smolensky's solution doesn't work. Cognition. Band 35, 1990, S. 183-184.
- ^ B. McLaughlin: The connectionism/classicism battle to win souls. Philosophical Studies, Band 71, 1993, S. 171-172.
- ^ B. McLaughlin: Can an ICS architecture meet the systematicity and productivity challenges? In: P. Calvo, J. Symons (Hrsg.): The Architecture of Cognition. Rethinking Fodor and Pylyshyn's Systematicity Challenge. MIT Press, Cambridge/MA, London, 2014, S. 31-76.
- ^ J.A. Fodor, B. McLaughlin: Connectionism and the problem of systematicity: Why Smolensky's solution doesn't work. Cognition. Band 35, 1990, S. 183-184.
- ^ J.A. Fodor: The language of thought. Harvester Press, Sussex, 1976, ISBN 0-85527-309-7.
- ^ J.A. Fodor: LOT 2: The language of thought revisited. Clarendon Press, Oxford, 2008, ISBN 0-19-954877-3.
- ^ J.A. Fodor, Z.W. Pylyshyn (1988), S. 33-48.
- ^ P. Smolenky: Reply: Constituent structure and explanation in an integrated connectionist / symbolic cognitive architecture. In: C. MacDonald, G. MacDonald (Hrsg.): Connectionism: Debates on psychological explanation. Blackwell Publishers. Oxford/UK, Cambridge/MA. Vol. 2, 1995, S. 224, 236-239, 242-244, 250-252, 282.
- ^ P. Smolensky, G. Legendre: The Harmonic Mind: From Neural Computation to Optimality-Theoretic Grammar. Vol. 1: Cognitive Architecture. A Bradford Book, The MIT Press, Cambridge, London, 2006a, ISBN 0-262-19526-7, S. 65-67, 69-71, 74-75, 154-155, 159-202, 209-210, 235-267, 271-342, 513.
- ^ M. Werning: Neuronal synchronization, covariation, and compositional representation. In: M. Werning, E. Machery, G. Schurz (Hrsg.): The compositionality of meaning and content. Vol. II: Applications to linguistics, psychology and neuroscience. Ontos Verlag, 2005, S. 283-312.
- ^ M. Werning: Non-symbolic compositional representation and its neuronal foundation: towards an emulative semantics. In: M. Werning, W. Hinzen, E. Machery (Hrsg.): The Oxford Handbook of Compositionality. Oxford University Press, 2012, S. 633-654.
- ^ A. Maye und M. Werning: Neuronal synchronization: from dynamics feature binding to compositional representations. Chaos and Complexity Letters, Band 2, S. 315-325.
- ^ Bechtel, W., Abrahamsen, A.A. Connectionism and the Mind: Parallel Processing, Dynamics, and Evolution in Networks. 2nd Edition. Blackwell Publishers, Oxford. 2002
- ^ G.F. Marcus: The algebraic mind. Integrating connectionism and cognitive science. Bradford Book, The MIT Press, Cambridge, 2001, ISBN 0-262-13379-2.
- ^ H. Maurer: Cognitive science: Integrative synchronization mechanisms in cognitive neuroarchitectures of the modern connectionism. CRC Press, Boca Raton/FL, 2021, ISBN 978-1-351-04352-6. https://doi.org/10.1201/9781351043526
References
- Feldman, Jerome and Ballard, Dana. Connectionist models and their properties(1982). Cognitive Science. V6, Iissue 3 , pp205-254.
- Rumelhart, D.E., J.L. McClelland and the PDP Research Group (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, Cambridge, Massachusetts: ISBN 978-0-262-68053-0
- McClelland, J.L., D.E. Rumelhart and the PDP Research Group (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 2: Psychological and Biological Models, Cambridge, Massachusetts: MIT Press, ISBN 978-0-262-63110-5
- Pinker, Steven and Mehler, Jacques (1988). Connections and Symbols, Cambridge MA: MIT Press, ISBN 978-0-262-66064-8
- Jeffrey L. Elman, Elizabeth A. Bates, Mark H. Johnson, Annette Karmiloff-Smith, Domenico Parisi, Kim Plunkett (1996). Rethinking Innateness: A connectionist perspective on development, Cambridge MA: MIT Press, ISBN 978-0-262-55030-7
- Marcus, Gary F. (2001). The Algebraic Mind: Integrating Connectionism and Cognitive Science (Learning, Development, and Conceptual Change), Cambridge, Massachusetts: MIT Press, ISBN 978-0-262-63268-3
- David A. Medler (1998). "A Brief History of Connectionism" (PDF). Neural Computing Surveys. 1: 61–101.
- Maurer, Harald (2021). Cognitive Science: Integrative Synchronization Mechanisms in Cognitive Neuroarchitectures of the Modern Connectionism, Boca Raton/FL: CRC Press, https://doi.org/10.1201/9781351043526, ISBN 978-1-351-04352-6
External links
- Dictionary of Philosophy of Mind entry on connectionism
- Garson, James. "Connectionism". In Zalta, Edward N. (ed.). Stanford Encyclopedia of Philosophy.
- A demonstration of Interactive Activation and Competition Networks Archived 2015-07-03 at the Wayback Machine
- "Connectionism". Internet Encyclopedia of Philosophy.
- Critique of connectionism