Deep reinforcement learning, symbolic learning and the road to AGI by Jeremie Harris

There have been several efforts to create complicated symbolic AI systems that encompass the multitudes of rules of certain domains. Called expert systems, these symbolic AI models use hardcoded knowledge and rules to tackle complicated tasks such as medical diagnosis. But they require a huge amount of effort by domain experts and software engineers and only work in very narrow use cases. As soon as you generalize the problem, there will be an explosion of new rules to add (remember the cat detection problem?), which will require more human labor. To summarize, one of the main differences between machine learning and traditional symbolic reasoning is how the learning happens. In machine learning, the algorithm learns rules as it establishes correlations between inputs and outputs.

Machine learning algorithms build mathematical models based on training data in order to make predictions. Although deep learning has historical roots going back decades, neither the term “deep learning” nor the approach was popular just over five years ago, when the field was reignited by papers such as Krizhevsky, Sutskever and Hinton’s now classic (2012) deep network model of Imagenet. One of the main stumbling blocks of symbolic AI, or GOFAI, was the difficulty of revising beliefs once they were encoded in a rules engine.

Career development

Many adults can perform a variety of motor tasks (typing, sewing a shirt, operating a lawn mower) that they would find difficult to describe in iconic (picture) or symbolic (word) form. Maybe in the future, we’ll invent AI technologies that can both reason and learn. But for the moment, symbolic AI is the leading method to deal with problems that require logical thinking and knowledge representation.

Word meanings are changing across the meta-training episodes (here, ‘driver’ means ‘PILLOW’, ‘shoebox’ means ‘SPEAKER’ etc.) and must be inferred from the study examples. This test episode probes the understanding of ‘Paula’ (proper noun), which just occurs in one of COGS’s original training patterns. In this paper, we introduce TPSR, a novel transformer-based planning framework for symbolic regression by leveraging priors of large-scale pretrained models and incorporating lookahead planning. TPSR incorporates Monte Carlo Tree Search (MCTS) into the transformer decoding process of symbolic regression models. Unlike conventional decoding strategies, TPSR enables the integration of non-differentiable feedback, such as fitting accuracy and complexity, as external sources of knowledge into the transformer-based equation generation process. At this age, children start playing side-by-side and noticing what others kids do.

Agents and multi-agent systems

Optimization closely followed the procedure outlined above for the algebraic-only MLC variant. The key difference here is that full MLC model used a behaviourally informed meta-learning strategy aimed at capturing both human successes and patterns of error. Using the same meta-training episodes as the purely algebraic variant, each query example was passed through a bias-based transformation process (see Extended Data Fig. 4 for pseudocode) before MLC processed it during meta-training. Specifically, each query was paired with its algebraic output in 80% of cases and a bias-based heuristic in the other 20% of cases (chosen to approximately reflect the measured human accuracy of 80.7%). To create the heuristic query for meta-training, a fair coin was flipped to decide between a stochastic one-to-one translation and a noisy application of the underlying grammatical rules. For the one-to-one translations, first, the study examples in the episode are examined for any instances of isolated primitive mappings (for example, ‘tufa → PURPLE’).

As in SCAN, the main tool used for meta-learning is a surface-level token permutation that induces changing word meaning across episodes. These permutations are applied within several lexical classes; for examples, 406 input word types categorized as common nouns (‘baby’, ‘backpack’ and so on) are remapped to the same set of 406 types. The other remapped lexical classes include proper nouns (103 input word types; ‘Abigail’, ‘Addison’ and so on), dative verbs (22 input word types; ‘given’, ‘lended’ and so on) and verbs in their infinitive form (21 input word types; such as ‘walk’, ‘run’). Surface-level word type permutations are also applied to the same classes of output word types. Other verbs, punctuation and logical symbols have stable meanings that can be stored in the model weights. Importantly, although the broad classes are assumed and could plausibly arise through simple distributional learning68,69, the correspondence between input and output word types is unknown and not used.

Extended Data Fig. 4 Example meta-learning episode and how it is processed by different MLC variants.

Limitations were discovered in using simple first-order logic to reason about dynamic domains. Problems were discovered both with regards to enumerating the preconditions for an action to succeed and in providing axioms for what did not change after an action was performed. The General Problem Solver (GPS) cast planning as problem-solving used means-ends analysis to create plans.

The power of human language and thought arises from systematic compositionality—the algebraic ability to understand and produce novel combinations from known components. Fodor and Pylyshyn1 famously argued that artificial neural networks lack this capacity and are therefore not viable models of the mind. Neural networks have advanced considerably in the years since, yet the systematicity challenge persists. Here we successfully address Fodor and Pylyshyn’s challenge by providing evidence that neural networks can achieve human-like systematicity when optimized for their compositional skills. To do so, we introduce the meta-learning for compositionality (MLC) approach for guiding training through a dynamic stream of compositional tasks. To compare humans and machines, we conducted human behavioural experiments using an instruction learning paradigm.

AI ‘breakthrough’: neural net has human-like ability to generalize language

These permutations induce changes in word meaning without expanding the benchmark’s vocabulary, to approximate the more naturalistic, continual introduction of new words (Fig. 1). 4 and detailed in the ‘Architecture and optimizer’ section of the Methods, MLC uses the standard transformer architecture26 for memory-based meta-learning. MLC optimizes the transformer for responding to a novel instruction (query input) given a set of input/output pairs (study examples; also known as support examples21), all of which are concatenated and passed together as the input. On test episodes, the model weights are frozen and no task-specific parameters are provided32. The specific procedure of optimizing over many related grammar-based tasks is not developmentally plausible, but there are several ways in which the greater principle—that systematicity can be honed through incentive and practice—has developmental merit. First, children are not born with an adult-like ability to compose functions; in fact, there seem to be important changes between infancy58 and pre-school59 that could be tied to learning.

Integrating this form of cognitive reasoning within deep neural networks creates what researchers are calling neuro-symbolic AI, which will learn and mature using the same basic rules-oriented framework that we do. Symbolic artificial intelligence is very convenient for settings where the rules are very clear cut,  and you can easily obtain input and transform it into symbols. In fact, rule-based systems still account for most computer programs today, including those used to create deep learning applications. This section introduces the methods used in neural-symbolic learning systems in three main categories. We aim to distill the representative ideas that provide evidence for the integration between neural networks and symbolic systems, identify the similarities and differences between different methods, and offer guidelines for researchers. The main characteristics of these representative methods are summarized in Table 3.

Symbolic Learning

For example, once a child learns how to ‘skip’, they can understand how to ‘skip backwards’ or ‘skip around a cone twice’ due to their compositional skills. Fodor and Pylyshyn1 argued that neural networks lack this type of systematicity and are therefore not plausible cognitive models, leading to a vigorous debate that spans 35 years2,3,4,5. Counterarguments to Fodor and Pylyshyn1 have focused on two main points.

We investigate an unconventional direction of research that aims at converting neural networks, a class of distributed, connectionist, sub-symbolic models into a symbolic level with the ultimate goal of achieving AI interpretability and safety. To that end, we propose Object-Oriented Deep Learning, a novel computational paradigm of deep learning that adopts interpretable “objects/symbols” as a basic representational atom instead of N-dimensional tensors (as in traditional “feature-oriented” deep learning). For visual processing, each “object/symbol” can explicitly package common properties of visual objects like its position, pose, scale, probability of being an object, pointers to parts, etc., providing a full spectrum of interpretable visual knowledge throughout all layers. It achieves a form of “symbolic disentanglement”, offering one solution to the important problem of disentangled representations and invariance. Basic computations of the network include predicting high-level objects and their properties from low-level objects and binding/aggregating relevant objects together. These computations operate at a more fundamental level than convolutions, capturing convolution as a special case while being significantly more general than it.

Deep learning and neural networks excel at exactly the tasks that symbolic AI struggles with. They have created a revolution in computer vision applications such as facial recognition and cancer detection. Symbolic AI involves the explicit embedding of human knowledge and behavior rules into computer programs. The practice showed a lot of promise in the early decades of AI research. But in recent years, as neural networks, also known as connectionist AI, gained traction, symbolic AI has fallen by the wayside. The above paper introduces the current research status and research methods of neural-symbolic learning systems in detail.

Even if you take a million pictures of your cat, you still won’t account for every possible case. A change in the lighting conditions or the background of the image will change the pixel value and cause the program to fail. The botmaster then needs to review those responses and has to manually tell the engine which answers were correct and which ones were not. 2) The two problems may overlap, and solving one could lead to solving the other, since a concept that helps explain a model will also help it recognize certain patterns in data using fewer examples.

In turn, connectionist AI has been criticized as poorly suited for deliberative step-by-step problem solving, incorporating knowledge, and handling planning. Finally, Nouvelle AI excels in reactive and real-world robotics domains but has been criticized for difficulties in incorporating learning and knowledge. Similar to the problems in handling dynamic domains, common-sense reasoning is also difficult to capture in formal reasoning.

Below is a quick overview of approaches to knowledge representation and automated reasoning. Alain Colmerauer and Philippe Roussel are credited as the inventors of Prolog. Prolog is a form of logic programming, which was invented by Robert Kowalski. Its history was also influenced by Carl Hewitt’s PLANNER, an assertional database with pattern-directed invocation of methods. For more detail see the section on the origins of Prolog in the PLANNER article. The key AI programming language in the US during the last symbolic AI boom period was LISP.

Kahneman describes human thinking as having two components, System 1 and System 2. System 1 is the kind used for pattern recognition while System 2 is far better suited for planning, deduction, and deliberative thinking. In this view, deep learning best models the first kind of thinking while symbolic reasoning the second kind and both are needed. A key component of the system architecture for all expert systems is the knowledge base, which stores facts and rules for problem-solving.[52]

The simplest approach for an expert system knowledge base is simply a collection or network of production rules. Production rules connect symbols in a relationship similar to an If-Then statement.

There were six pool options, and the assignment of words and item order were random. One participant was excluded because they reported using an external aid in a post-test survey. On average, the participants spent 5 min 5 s in the experiment (minimum 2 min 16 s; maximum 11 min 23 s). Deep neural networks are also very suitable for reinforcement learning, AI models that develop their behavior through numerous trial and error. This is the kind of AI that masters complicated games such as Go, StarCraft, and Dota. Also, some tasks can’t be translated to direct rules, including speech recognition and natural language processing.

• Supplementary 1–3 (additional modelling results, experiment probing additional nuances in inductive biases, and few-shot instruction learning with OpenAI models), Supplementary Figs.
• There are similarities between Piaget and Bruner, but a significant difference is that Bruner’s modes are not related in terms of which presuppose the one that precedes it.
• So the main challenge, when we think about GOFAI and neural nets, is how to ground symbols, or relate them to other forms of meaning that would allow computers to map the changing raw sensations of the world to symbols and then reason about them.
• For example, Andrews et al. (1995) and Townsend et al. (2019) center around knowledge extraction techniques, which aligns with the first category discussed in Section 2.

Read more about https://www.metadialog.com/ here.