Exploring the Representational Spaces of States, Values, and Goals in Cognitive Maps

Abstract

Maximizing current or future expected rewards is one of the most common goals of many decisions we make. Decisions are always made within the context of a given task. To optimally achieve our goals, we must combine knowledge of the structure of the environment with the current goal to optimally predict different potential rewards. Even the same exact choice can provide different rewards depending on the task at hand. How does the brain combine task demands, environmental structure, and reward maximization in the service of decision-making? Prior works have shown that the ventromedial prefrontal cortex (vmPFC) and adjacent orbitofrontal cortex (OFC) are known to contain signals corresponding to anticipated outcomes of decisions, known as expected value signals, that inform our choices. The hippocampal formation is known to maintain a representation of the environment and potential courses of action in it, known as a cognitive map. This thesis comprises three projects that explore the interaction between task structure, value representation, and cognitive maps within the vmPFC, OFC, and hippocampal formation.

In the first project (Moneta et al., Nature Communications, 2023), we investigate how the vmPFC flexibly switches between different value representations in a task-dependent manner. Thirty-five participants completed a random dot-motion task in which either stimulus color or motion predicted rewards. Multivariate MRI analyses showed that vmPFC signals contain a rich representation that includes the current task state or context (motion/color), the expected value associated with the state, and crucially, the irrelevant expected value of the alternative context. We also find that irrelevant value representations in vmPFC compete with relevant value signals, interact with task-state representations, and relate to behavioral signs of value competition. Our results shed light on vmPFC's role in decision-making, bridging between its role in mapping observations onto the task states of a mental map, and computing expected values for multiple potential states.

In the second project (Moneta et al., in prep.), taking a broader perspective on task structure, we examine how non-spatial cognitive maps are affected by value generalization. A non-spatial cognitive map refers to mental representations of abstract relationships between different states, helping an agent navigate decision-making by encoding how various choices or actions relate to one another. Animal work has shown that neural representations of spatial cognitive maps are affected by reward. In this project, we tested if similar effects generalize to non-spatial maps in humans. Seventy-two participants (38 of which underwent MRI scanning) performed two sessions of a perceptual discrimination task, before and after extensive reward learning. In all sessions, stimuli varied along two perceptual dimensions, forming a continuous two-dimensional cognitive map. After reward learning, performance in the perceptual discrimination task improved among previously rewarding stimuli. The effect of reward also generalized to areas of the cognitive map that were never rewarding. The precise pattern of changes in perceptual similarity judgments is consistent with the idea that reward learning leads to increased psychological distance between stimuli in the rewarding area, and decreased spacing in neighboring areas. Simulations show that a shift of representational fields towards the rewarded location, akin to a gravitation pulling, can explain the behavioral changes. In line with this, preliminary fMRI data analysis shows evidence for such gravitational pull in the hippocampus and, to some degree, in the medial OFC representations. Future analyses including additional regions in the hippocampal formation and the prefrontal cortex are planned. These results indicate that reward affects non-spatial cognitive maps and suggests accompanying neural representational changes.

In the third project (Moneta, Grossman, Schuck, Trends in Neurosciences, 2024), we review recent literature connecting value and state representations in the OFC/vmPFC, proposing that these regions integrate stimulus, context, and outcome information into a unified representational space. Comparable encoding principles emerge in late layers of deep reinforcement learning models, where single nodes exhibit similar forms of mixed-selectivity, which enables flexible readout of relevant variables by downstream neurons. Based on these lines of evidence, we suggest that outcome-maximization leads to complex representational spaces that are insufficiently characterized by linear value signals that have been the focus of most prior research on the topic. We also discuss major outstanding questions concerning the role of OFC/vmPFC in learning across tasks, encoding of task-irrelevant aspects, and the role of hippocampus-PFC interactions.

Collectively, these projects shed light on the dynamic relationship between task states and value representations, their integration into a cognitive map, and the representational capacities of the OFC/vmPFC and the hippocampal formation. The findings provide new insights into the neural mechanisms guiding behavior and suggest future research directions.