Search Results

One oft-envisioned function of search is planning actions (e.g., by exploring routes through a cognitive map). Yet, among the most prominent and quantitatively successful neuroscentific theories of the brain’s systems for action choice is the temporal-difference account of the phasic dopamine response. Surprisingly, this theory envisions that action sequences are learned without any search at all, but instead wholly through a process of reinforcement and chaining. This chapter considers recent proposals that a related family of algorithms, called model-based reinforcement learning, may provide a similarly quantitative account for action choice by cognitive search. It reviews behavioral phenomena demonstrating the insufficiency of temporal-difference-like mechanisms alone, then details the many questions that arise in considering how model-based action valuation might be implemented in the brain and in what respects it differs from other ideas about search for planning.

Psychiatry faces numerous challenges: the reconceptualization of symptoms and diagnoses, disease prevention, treatment development and monitoring of its effects, and the provision of individualized, precision medicine. To confront the complexity and heterogeneity intrinsic to brain disorders, psychiatry needs better biological, quantitative, and theoretical grounding. This chapter seeks to identify the sources of complexity and heterogeneity, which include the interplay between genetic and epigenetic factors with the environment and their impact on neural circuits. Computational approaches provide a framework to address complexity and heterogeneity, which cannot be seen as noise to be eliminated from diagnosis and treatment of disorders. Complexity and heterogeneity arise from intrinsic features of brain function, and thus present opportunities for computational models to provide a more accurate biological foundation for diagnosis and treatment of psychiatric disorders. Challenges to be addressed by a computational framework include: (a) improving the search for risk factors and biomarkers, which can be used toward primary prevention of disease; (b) representing the biological ground truth of psychiatric disorders, which will improve the accuracy of diagnostic categories, assist in discovering new treatments, and aid in precision medicine; (c) representing how risk factors, biomarkers, and the underlying biology change through the course of development, disease progression, and treatment process.

This chapter assesses alternative approaches of reinforcement learning that are developed by machine learning. The initial goal of this branch of artificial intelligence, which appeared in the middle of the twentieth century, was to develop and implement algorithms that allow a machine to learn. Originally, they were computers or more or less autonomous robotic automata. As artificial intelligence has developed and cross-fertilized with neuroscience, it has begun to be used to model the learning and decision-making processes for biological agents, broadening the meaning of the word ‘machine’. Theoreticians of this discipline define several categories of learning, but this chapter only deals with those which are related to reinforcement learning. To understand how these algorithms work, it is necessary first of all to explain the Markov chain and the Markov decision-making process. The chapter then goes on to examine model-free reinforcement learning algorithms, the actor-critic model, and finally model-based reinforcement learning algorithms.

The discussion here considers a much more common learning condition where an agent, such as a human or a robot, has to learn to make decisions in the environment from simple feedback. Such feedback is provided only after periods of actions in the form of reward or punishment without detailing which of the actions has contributed to the outcome. This type of learning scenario is called reinforcement learning. This learning problem is formalized in a Markov decision-making process with a variety of related algorithms. The second part of this chapter will use function approximators with neural networks which have made recent progress as deep reinforcement learning.

Delving further into development, adaptation, and learning, this chapter considers the potential of reward-oriented optimization of behavior. Reinforcement learning (RL) is motivated from the Rescorla–Wagner model in psychology and behaviorism. Next, a detailed introduction to RL in artificial systems is provided. It is shown when and how RL works, but also current shortcomings and challenges are discussed. In conclusion, the chapter emphasizes that behavioral optimization and reward-based behavioral adaptations can be well-accomplished with RL. However, to be able to solve more challenging planning problems and to enable flexible, goal-oriented behavior, hierarchically and modularly structured models about the environment are necessary. Such models then also enable the pursuance of abstract reasoning and of thoughts that are fully detached from the current environmental state. The challenge remains how such models may actually be learned and structured.