Search Results

This chapter argues that investigation of reinforcement learning is a complement to the study of belief learning, rather than being a ‘dangerous antagonist’. It begins at the low end of the scale, to see how far simple reinforcement learning can get us, and then move up. Exactly how does degree of reinforcement affect the strengthening of the bond between stimulus and response? Different answers are possible, and these yield alternative theories of the law of effect.

The entire cerebral cortex projects down to the basal ganglia that plays an important role in coordinating cognition. These structures serve as the repository for our procedural knowledge. They have the ability to recognize appropriate cortical patterns and take actions directly without further deliberation. In contrast to declarative memory, procedural memory is a slow-learning system in which new capacities only gradually emerge. This chapter describes how new production rules are acquired in procedural memory and how reinforcement learning mechanisms serve to select among alternative productions. Three examples are described that focus on procedural learning in language acquisition, learning from instructions, and brain imaging changes that occur with procedural learning.

One oft-envisioned function of search is planning actions (e.g., by exploring routes through a cognitive map). Yet, among the most prominent and quantitatively successful neuroscentific theories of the brain’s systems for action choice is the temporal-difference account of the phasic dopamine response. Surprisingly, this theory envisions that action sequences are learned without any search at all, but instead wholly through a process of reinforcement and chaining. This chapter considers recent proposals that a related family of algorithms, called model-based reinforcement learning, may provide a similarly quantitative account for action choice by cognitive search. It reviews behavioral phenomena demonstrating the insufficiency of temporal-difference-like mechanisms alone, then details the many questions that arise in considering how model-based action valuation might be implemented in the brain and in what respects it differs from other ideas about search for planning.

Current computational models of dopamine (DA) modulation have worked either from a more abstract neuroalgorithmic level, starting with specific assumptions about DA's computational role and then working out its implications at a higher cognitive level, or have used a more biophysical/physiological implementation to unravel the dynamic and functional consequences of DA's effects on voltage-gated and synaptic ion channels. This chapter focuses on the latter, and in addition will specifically review models of DA-innervated target regions rather than models of ventral tegmental area/substantia nigra (VTA/SN) DA neurons themselves. It begins with a brief discussion of how DA may change the input/output functions of single striatal and cortical neuron. It considers the network level and the potential computational role of DA in higher cognitive functions, and then reviews DA-based models of reinforcement learning.

This chapter discusses evidence for the applicability of reinforcement learning models to reward-learning and reward-based action selection in humans, with a particular emphasis on data derived from functional magnetic resonance imaging (fMRI) studies. It begins with a description of basic mechanisms by which predictions of future reward as well as punishment can be learned, and their associated neural bases. It then considers the mechanisms underlying action selection for reward, as well as in learning to avoid punishers. It also reviews instances where findings from functional neuroimaging studies are inconsistent with a simple reinforcement learning framework, and discusses the implications of these results for the development of a more complete theory of human choice.

This chapter reviews the anatomy of the neural circuits associated with dopamine and valuation; presents theories of reinforcement learning from psychology and computer science that predate dopaminergic studies of valuation; examines experimental work that forged a relationship between dopamine and these pre-existing computational theories; and reductively follows these insights all the way down to the behavior of ion channels. It shows that a neuroeconomic theory of valuation sweeps effectively from cell membranes to utility theory, and that constraints derived from each interlocking level of analysis provide useful tools for better understanding how we decide.

Reinforcement is an empirical principle which states that the higher the payoff from taking an action in the past, the more likely it will be taken in the future. This approach has long occupied a prominent place in psychology; more recently it has begun to migrate into experimental economics and game theory. This chapter sketches some of the better-known models and what is known about their asymptotic behaviour in particular situations.

The assumption that people possess a repertoire of strategies to solve the inference problems they face has been raised repeatedly. However, a computational model specifying how people select strategies from their repertoire is still lacking. The proposed strategy selection learning (SSL) theory predicts a strategy selection process on the basis of reinforcement learning. The theory assumes that individuals develop subjective expectations for the strategies they have and select strategies proportional to their expectations, which are then updated on the basis of subsequent experience. The learning assumption was supported in four experimental studies. Participants substantially improved their inferences through feedback. In all four studies, the best-performing strategy from the participants' repertoires most accurately predicted the inferences after sufficient learning opportunities. When testing SSL against three models representing extensions of SSL and against an exemplar model assuming a memory-based inference process, the authors found that SSL predicted the inferences most accurately.

This chapter looks at selection of mainly autonomic responses and their classical conditioning. Selection of approach or withdrawal, and their classical conditioning are also mentioned. It then goes on to describe a selection of fixed stimulus-response habits; and a selection of arbitrary behaviours to obtain goals, action-outcome learning, and emotional learning. The roles of the prefrontal cortex in decision-making and attention are described. The chapter then goes on to talk about neuroeconomics, reward magnitude, expected value, and expected utility; delay of reward, emotional choice, and rational choice; reward prediction error, temporal difference error, and choice; reciprocal altruism, strong reciprocity, generosity, and altruistic punishment; and dual routes to action, and decision-making.

This chapter contrasts two approaches to the study of mechanism and function in decision making: rules of thumbs (or heuristics) and the contributions of experimental psychology and psychophysics. The first approach is most frequently used by behavioral ecologists. It implements a behavioral gambit by which researchers address hypothetical decision problems without reference to independently known cognitive processes. The second approach shares interest in the functional consequences of behavior, but shows greater subordination to empirical research on behavioral and cognitive mechanisms. Here natural selection is seen to act on processes that tune behavior to the environment across broad domains. Associative learning and Weber’s Law are two putative evolutionary responses to such challenges. In the second approach these independently known traits, rather than ad-hoc rules or heuristics, are considered as candidates for effecting decisions, and this can often lead to asking for the functional problem a posteriori, querying what selective pressures might have led to the presence of the trait. Based on examples from foraging research, it is argued that for a majority of decision problems investigated across vertebrates, the second approach is preferable. It is also recognized that dedicated rules are preferable when the relevant information acts across generations and involves little learning.

Values, rewards, and costs play a central role in economic, statistical, and psychological notions of decision making. They also have surprisingly direct neural realizations. This chapter discusses the ways in which different value systems interact with different decision-making systems to fashion and shape affectively appropriate behavior in complex environments. Forms of deliberative and automatic decision making are interpreted as sharing a common purpose rather than serving different or non-normative goals.

Behavioral performance of humans and other animals and the neural activity related to various behaviors commonly display substantial variability, but the relationship between these two different types of variability remains poorly understood. The dynamics of choice behaviors during a computer-simulated competitive game were studied and the chapter characterizes the variability in the activity of individual neurons recorded in the parietal, cingulate, and prefrontal areas of the primate cortex. It was found that the monkeys approached the optimal strategy during the competitive game using a reinforcement learning algorithm. In addition, the activity in multiple cortical areas commonly modulated their activity according to the variables necessary to update the animal’s decision-making strategies using the reinforcement learning algorithm. In particular, neurons in the dorsolateral prefrontal cortex and lateral intraparietal area encoded signals related to the value functions of the two alternative targets in addition to the animal’s choice and outcome history. Nevertheless, the residual variability in the activity of neurons in the cortical areas tested in this study that could not be accounted for by task-related variables was still substantial and higher than expected from additive Poisson noise. Moreover, the amount of this residual variability was particularly high for the anterior cingulate cortex. The functional significance of this regional difference in the variability of neural activity remains largely unknown.

This chapter is devoted to consideration of adaptive processes in neural processing. This adaptation includes both weight changes (what is usually meant by ‘learning’), and dynamic stability of network activity states (what is usually taken to be a model of ‘memory’). A serial working memory model is described in detail, that exploits the syntactic representations of the SPA. In addition, a biologically detailed spike-based learning rule is presented and applied to learning arbitrary vector functions, reinforcement learning, and explaining STDP effects. Finally, this rule is applied to a network that explains the results of the Wason Card Selection task. Tutorial: Learning in Nengo

Evolutionary and psychological approaches to decision making remain largely separate endeavors. Each offers necessary techniques and perspectives which, when integrated, will aid the study of decision making in both humans and nonhuman animals. The evolutionary focus on selection pressures highlights the goals of decisions and the conditions under which different selection processes likely influence decision making. An evolutionary view also suggests that fully rational decision processes do not likely exist in nature. The psychological view proposes that cognition is hierarchically built on lower-level processes. Evolutionary approaches to decision making have not considered the cognitive building blocks necessary to implement decision strategies, thereby making most evolutionary models of behavior psychologically implausible. The synthesis of evolutionary and psychological constraints will generate more plausible models of decision making.

This chapter proposes a model in which the activity of the anterior cingulate cortex (ACC) is modulated in part by reinforcement learning processes in the basal ganglia. It describes how this relationship may lead to a better understanding of the error-related negativity (ERN). The conflict monitoring theory of the ERN is also discussed. Finally, the chapter describes the core basal ganglia model and its role in reinforcement learning and action selection.

This chapter discusses the relevance of reinforcement learning (RL) to its hierarchical structure. It first reviews the fundamentals of RL, with a focus on temporal-difference learning in actor-critic models. Next, it discusses the scaling problem and the computational issues that stimulated the development of hierarchical reinforcement learning (HRL). The potential neuroscientific correlates of HRL are also described. The chapter also presents the results of some initial empirical tests and ends with directions for further research.

The learning of cognitive programs faces many technical difficulties, but the most important is the valuation of programs that have delayed rewards. The algorithms of Reinforcement Learning that the brain uses tackle this head-on are logically situated in the Basal Ganglia, which represent the abstract sequential components of motor and cognitive plans. Such sequences are evaluated in terms of their expected reward and risk, which in turn are coded, by using the neurotransmitters dopamine and serotonin respectively, which serves as a common evaluative currency. Reinforcement learning algorithms learn by adjusting deviations in expected reward, a signal, which can also be used to program the Cortex’s memory representations. Tesauro’s use of reinforcement to learn the game of Backgammon provides a superb example of the putative integration of the process between the two forebrain subsystems.

This chapter surveys eight research areas organized around one question: As learning systems become increasingly intelligent and autonomous, what design principles can best ensure that their behavior is aligned with the interests of the operators? The chapter focuses on two major technical obstacles to AI alignment: the challenge of specifying the right kind of objective functions and the challenge of designing AI systems that avoid unintended consequences and undesirable behavior even in cases where the objective function does not line up perfectly with the intentions of the designers. The questions surveyed include the following: How can we train reinforcement learners to take actions that are more amenable to meaningful assessment by intelligent overseers? What kinds of objective functions incentivize a system to “not have an overly large impact” or “not have many side effects”? The chapter discusses these questions, related work, and potential directions for future research, with the goal of highlighting relevant research topics in machine learning that appear tractable today.

The reinforcement memory systems evolved early in the history of animals. Through these ancient mechanisms, animals can remember which actions in which circumstances produced benefits or avoided costs (instrumental memories), as well as which contexts, objects, and places were associated with costs or benefits (Pavlovian memories). Surface similarities support a common set of terms and formalisms to describe reinforcement learning of many kinds, but the fact that they depend on a variety of unrelated brain structures and can be established by brainless animals, such as sea anemones, shows that they do not compose a single system or mechanism. As new memory systems emerged during vertebrate evolution, reinforcement learning persisted, but it cannot account for derived aspects of human cognition such as analogical, metaphorical, or relational reasoning, abstract problem-solving strategies, mental time travel, scenario construction, mental trial and error behavior, autobiographical narratives, language, a theory of mind, or explicit (declarative) memory.

This chapter discusses a new integrative theory of anterior cingulate cortex (ACC) function that proposes that the dorsal ACC supports the selection and execution of coherent behaviors over extended periods. It first presents the current theories of ACC function and its role in four key aspects of behavior: performance monitoring, action, reinforcement learning, and motivation. The chapter then outlines a new theory that proposes that ACC contributes to hierarchical reinforcement learning or high-level, temporally extended behaviors.