Neuro-symbolic AI

Neuro-symbolic AI is a type of

Approaches for integration are diverse. Henry Kautz's taxonomy of neuro-symbolic architectures,^[10] along with some examples, follows:

• Symbolic Neural symbolic is the current approach of many neural models in
  large language models. Examples include BERT, RoBERTa, and GPT-3
  .
• Symbolic[Neural] is exemplified by AlphaGo, where symbolic techniques are used to invoke neural techniques. In this case, the symbolic approach is Monte Carlo tree search and the neural techniques learn how to evaluate game positions.
• Neural | Symbolic uses a neural architecture to interpret perceptual data as symbols and relationships that are reasoned about symbolically. Neural-Concept Learner^[11] is an example.
• Neural: Symbolic → Neural relies on symbolic reasoning to generate or label
  training data that is subsequently learned by a deep learning model, e.g., to train a neural model for symbolic computation by using a Macsyma-like symbolic mathematics
  system to create or label examples.
• Neural_{Symbolic} uses a
  neural net that is generated from symbolic rules. An example is the Neural Theorem Prover,^[12] which constructs a neural network from an AND-OR proof tree generated from knowledge base rules and terms. Logic Tensor Networks^[13]
  also fall into this category.
• Neural[Symbolic] allows a neural model to directly call a symbolic reasoning engine, e.g., to perform an action or evaluate a state. An example would be
  Wolfram Alpha
  .

These categories are not exhaustive, as they do not consider multi-agent systems. In 2005, Bader and Hitzler presented a more fine-grained categorization that considered, e.g., whether the use of symbols included logic and if it did, whether the logic was propositional or first-order logic.^[14] The 2005 categorization and Kautz's taxonomy above are compared and contrasted in a 2021 article.^[10] Recently, Sepp Hochreiter argued that Graph Neural Networks "...are the predominant models of neural-symbolic computing"^[15] since "[t]hey describe the properties of molecules, simulate social networks, or predict future states in physical and engineering applications with particle-particle interactions."^[16]

Artificial general intelligence

Gary Marcus argues that "...hybrid architectures that combine learning and symbol manipulation are necessary for robust intelligence, but not sufficient",^[17] and that there are

...four cognitive prerequisites for building robust artificial intelligence:

• hybrid architectures that combine large-scale learning with the representational and computational powers of symbol manipulation,
• large-scale knowledge bases—likely leveraging innate frameworks—that incorporate symbolic knowledge along with other forms of knowledge,
• reasoning mechanisms capable of leveraging those knowledge bases in tractable ways, and
• rich
  knowledge bases.^[18]

This echoes earlier calls for hybrid models as early as the 1990s.[19]^[20]

History

A series of workshops on neuro-symbolic AI has been held annually since 2005 Neuro-Symbolic Artificial Intelligence.[23] In the early 1990s, an initial set of workshops on this topic were organized.^[19]

Research

Key research questions remain,^[24] such as:

• What is the best way to integrate neural and symbolic architectures?
• How should symbolic structures be represented within neural networks and extracted from them?
• How should common-sense knowledge be learned and reasoned about?
• How can abstract knowledge that is hard to encode logically be handled?

Implementations

Implementations of neuro-symbolic approaches include:

• Scallop: a language based on Datalog that supports differentiable logical and relational reasoning. Scallop can be integrated in Python and with a PyTorch learning module.^[25]
• Logic Tensor Networks: encode logical formulas as neural networks and simultaneously learn term encodings, term weights, and formula weights.
• DeepProbLog: combines neural networks with the probabilistic reasoning of ProbLog.
• SymbolicAI: a compositional differentiable programming library.
• Explainable Neural Networks (XNNs): combine neural networks with symbolic hypergraphs and trained using a mixture of backpropagation and symbolic learning called induction.^[26]

Citations