Sequence Learning and the Role of the Hippocampus in Rodent Navigation

Introduction

Navigation is a complex phenomenon, encompassing a number of different strategies and behaviors that preclude a straightforward association with any particular brain system. Nonetheless, the one brain system that is most frequently associated with rodent navigation is the hippocampal system. Hippocampal place cells fire selectively when an animal occupies a restricted location in an environment (Box 1), and they have long been hypothesized as the neural substrate of an internal, cognitive map of the environment that is necessary for flexible navigation, map-based spatial learning, and episodic memory [1]. However, a vexing problem for such notions has been that, although place cells can convey information about the current location of the animal, navigation needs a system that conveys information about desired targets and routes to those targets. It has been difficult to find strong evidence that the hippocampus represents this crucial information needed to guide navigation to specific locations away from the current location of an animal. However, the development of multi-electrode recording techniques, which allow the simultaneous recording of dozens of neurons, has allowed quantitative analyses of neural populations that have begun to shed light on how the hippocampus can create representations of spatial (and perhaps temporal [2;3]) sequences that can extend many meters away from the current location of the animal, and that may constitute the neural substrate for online navigation decisions.

Hippocampal place cells offer an ideal representation for navigational learning

We can begin to understand the hippocampal contribution to navigation by comparing the neural responses found in the hippocampus with the highly geometrical and relatively task-independent spatial representations that have been found outside the hippocampus proper. Cells of one class, called grid cells, fire in a geometrically regular pattern of locations that form a pattern of equilateral triangles that tessellate the floor of an environment [4;5]. Cells of another class, called head direction cells, fire whenever the animal's head points in a particular, allocentric direction in an environment [6]. Both grid cells and head direction cells appear to be rather rigid and inflexible spatial representations, firing in similar patterns relative to each other across environments [7;8]. In contrast, the hippocampal representation of space is very different, in that it is extremely flexible and adapts to the task at hand [9–12]. Within the framework of Markov decision problems (MDPs) [13], of which navigation is an example, the hippocampus appears to learn a Markovian state space representation (Box 2). What this means in simple terms is that cells in the hippocampus come to be active in such a way that simple associations with their activity can specify navigationally useful actions or represent navigationally relevant quantities. This imposes severe constraints upon hippocampal responses. They need to individuate places at the same spatial scale as the animal makes navigational decisions, despite the fact that sensory stimuli do not in general correspond to places, and even grid cell and head direction cell inputs are active in spatially distributed patterns. Nevertheless, the place fields of hippocampal cells are both highly localized and cover the entire environment in a relatively uniform way [14]. Moreover, it has been shown that place fields do not straddle impenetrable barriers to movement [15], which accords with the fact that correct actions on two different sides of a barrier are unlikely to be the same despite the Euclidean proximity of the two regions. Place fields are task specific, in that different navigational tasks, and hence different sets of actions, within the same environment are handled by different sets of place cells [9]. When within the same task the same place must be associated with different actions at different times, this distinction can be handled by using different cells for the same place [10;16–20]. Given these properties, place cells can be modeled as the input to an actor-critic model of navigational learning, using the temporal difference learning rule (Box 3), which can learn short paths in a classic spatial task as quickly as real rats do [21].

Recent evidence supports the notion that areas downstream of the hippocampus may act in accord with this conception of hippocampal place cell function, with the ventral striatum being the downstream critic [22;23] and the dorsal striatum being the downstream actor [24]. Particularly, van der Meer and Redish recently provided evidence that a subset of ventral striatum neurons ramp up in activity during approaches to reward sites, which matches what would be expected from a critic [21;25;26]. Moreover, ventral striatum neurons exhibit a close connection to fine-timescale dynamics in the hippocampal representation of state, specifically theta phase precession [25], forward sweeps[23], and replay [22;27;28], which are discussed in the following sections.

Evidence for encoding and expression of spatial sequences during active movement

During exploration of an environment, an animal can learn to navigate to a goal by encoding the sequence of locations required to get to the goal from a given starting point. Computational models have shown that the temporal asymmetry of long-term potentiation can encode such sequences in the synaptic weights of a place cell matrix [29–31]. Because long-term potentiation (LTP) is preferentially induced when a presynaptic cell fires before a postsynaptic cell, whereas long-term depression (LTD) is induced in the opposite situation [32], a place cell that fires early in a trajectory toward a goal will enhance its ability to fire a cell that fires slightly later in the trajectory. A prediction of the models is that the second cell will begin to fire earlier and earlier during repeated runs of the trajectory, as it is increasingly driven by the presynaptic cells that fire earlier in the sequence. This prediction was confirmed in recordings of CA1 place cell populations [33], and the phenomenon is dependent on NMDA receptors [34]. This effect was interpreted as CA1 cells learning to predict the future location of the animal based on its current trajectory and its past experience [35]. CA3 place cells show a similar phenomenon. Unlike CA1, which reverts back to its baseline firing locations after 24 hours, CA3 retains the learned sequences for long periods of time, suggesting that the recurrent collateral network of CA3 is the long-term repository of the spatial sequence information [36].

While storing such sequence information may be a useful property to aid the hippocampus in its role in navigation, these mechanisms are not specific to the hippocampus. Head direction cells show a similar backward shift of their tuning properties when the animal runs repeated laps on a track [37]. Furthermore, repeated stimulation of visual cortex neurons with sequences of oriented bars can shift the orientation tuning of a neuron in a manner analogous to the shifting of place fields and head direction tuning curves [38]. Thus, these mechanisms are likely to be broadly prevalent in the nervous system, as many brain systems require the capacity to encode commonly experienced sequences of stimulation in order to predict future states based on prior experience.

Hippocampal cell activity also expresses sequences within individual cycles of the prevalent theta rhythm. This finding was suggested indirectly from the observation of theta phase precession, a temporal relationship between place cell spikes and the running-related theta rhythm in the hippocampal EEG [39]. However, theta precession is typically extremely variable over repeated passes through a place field, as the theta phases associated with spiking at a particular location can span a wide range of phases when the spikes are collated over a session. This variability, coupled with the variability of place cell firing rates from trial to trial [40] and the dependence of the rate of precession on the size of a place field [41;42], made the prediction of sequences nontrivial. By recording the activity of large numbers of place cells simultaneously, the existence of theta sequences was confirmed [43]. Moreover, shuffling spike phases between trials in a way that preserved the theta precession relationship for individual cells had the effect of disrupting theta sequences across cells, indicating that the relative timing of spikes during theta sequences is more tightly controlled than would be predicted from the theta precession relationship alone [43]. One possible corollary of this result is the recent observation that theta precession may be less variable when considered within a single trial rather than averaged across trials [44]. Place field expansion can be interpreted as the elongation of theta sequences (Figure 1). Conceptually, theta sequences have been proposed to compress behavioral spatiotemporal sequences into a timescale conducive to LTP-induction mechanisms [45]. Moreover, since the sequences tend to be repeated over multiple theta cycles as an animal passes through a region of space, the chunking of behavioral sequences into multiple, repeated theta sequences effectively turns one-trial learning into multiple trials, which may help to solidify the sequence memory.

Expression of entire navigational sequences during navigation tasks

Like other MDPs, navigation suffers from the well known temporal credit assignment problem, which is how to relate reward or outcome information that may occur only at the end of a long sequence of decisions to the individual decisions within that sequence [46]. Theta sequences appear to be restricted in spatial extent to overlapping place fields. Thus, although they potentially allow information to be accessed relating to places immediately ahead of an animal, and may even allow exploration of alternatives at a choice point up to ~1 m [47], they do not allow information to be accessed corresponding to longer trajectories. However, much longer sequences called replay sequences have been reported. First shown in sleep [48–50], where they were hypothesized to be involved in long-term memory consolidation [51], they have more recently been shown to occur robustly during the awake state as well. It was reported that when an animal stops at the end of a trajectory, the entire behavioral sequence of activity across place cells is replayed in reverse order, beginning with those place cells corresponding to the current position at the trajectory endpoint and ending with those place cells that were active at the beginning of the trajectory [52]. Moreover, any single behavioral episode can give rise to large numbers of reverse replays. The phenomenon has been extensively replicated [53–57]. Consistent with a role in navigational planning, distance information is faithfully represented in replay sequences [54]. Consistent with a role in learning, reverse replay appears to be more prevalent on a novel track than a familiar one [52], whereas forward ordered sequences display the opposite pattern [54]. Moreover, reward appears to play a critical role in modulating replay levels [58]. Unlike theta sequences, awake replay sequences can indeed extend through the entirety of a navigational trajectory, even up to 10 m in length [55], which is the limit that has so far been tested.

Theoretically, because of the Markovian property of MDPs, reverse replay offers a much more efficient mode of planning than forward look-ahead, since trajectories from multiple starting positions are evaluated simultaneously in the context of the same outcome, and in a way that respects the Markovian structure of the problem. Interestingly, reverse replay has been proposed repeatedly in the machine learning literature as a way to speed up learning within temporal difference learning models [59–61]. A key insight is that reverse replay does not replace the previous actor-critic model, but instead naturally augments that model, in that offline reverse playback will improve value estimates in the critic. From a modeling perspective, reverse replay is most useful early in learning, when the critic knows almost nothing, and so getting value information out from the reward point is the central task. Later on, when estimates are available everywhere, this may not be so useful, and short-range comparisons might be preferred in order to make finer scale changes to the critic's value function. As noted above, this pattern does appear to hold [52;54]. Thus, awake replay, and reverse replay in particular, may be an ideal mechanism for learning and planning in navigational tasks.

Navigation in larger state spaces

A recent report tracking the movements of a single wild rat revealed that it completely explored a 9.5 hectare island and then swam 400 m across open water to a neighboring island [62]. Therefore, the state space for real-life navigation is much larger than what is typically studied in the laboratory. Another way in which navigational state spaces can be very large is when considering the multiplicity of related navigational problems, such as navigating to different goals within the same space. Theoretically, one way to deal with larger state spaces is to incorporate hierarchical planning and control [63]. One simple approach is to augment the state space representation with a set of larger states, thus permitting generalization between states, that can speed learning in large state spaces and allow generalization to novel goal positions [64]. Interestingly, place cells with larger fields are found at the temporal (ventral) end of the hippocampus [65;66], and more recently it was shown that place field size and grid field scale increase gradually along the septo-temporal (dorsal-ventral) axis [67;68]. According to the model, it would be sufficient for cells with larger place fields to project to the same downstream value and action areas, for the effects of generalization and accelerated learning to take place [64]. However, a critical requirement in the model is that larger fields should respect the structure of the navigational environment, and this remains to be tested.

Cross-species comparisons

The studies reviewed here are almost exclusively from rats, the species that has provided the overwhelming majority of data about hippocampal place cells. A question of interest is whether the phenomena described here are specific to rat navigation, or whether they are common to other species. There are little data in the literature on other species regarding the backward expansion, theta sequence, and awake replay sequence phenomena described here for rats. However, there are accumulating data from other species that suggest a commonality of mechanisms across species, as well as hints of intriguing differences. For example, as reviewed above, replay sequences in the alert state may be related to active navigation decisions. Awake replay events have been reported in mice [50], whereas sleep replay has been reported in the song system of zebra finches [69]. The circuitry and network mechanisms that allow sleep replay may also be present in the awake state, and if so, they may allow similar mechanisms for behavioral control.

Place cells have been recorded from the hippocampus of mice [70;71], rabbits [72], bats [73], and monkeys [74;75], and there is evidence that they may exist in humans as well [76]. Interestingly, hippocampal recordings from pigeon hippocampus failed to find robust place cells, although there was evidence of hemisphere-dependent spatial modulation and trajectory encoding [77;78]. Moreover, the theta rhythm has also been recorded during alert behavior in different species [79–81], and place cells in mice demonstrate theta phase precession [82], thereby raising the possibility that theta sequences are present as well across species. An interesting difference comes from the bat, however. When bats crawl along a surface, they show both place-cell firing in hippocampus [73] as well as grid-cell firing in the medial entorhinal cortex [83]. However, under these conditions, there is no evidence of strong, movement-related theta in either the LFP or the spike-train autocorrelograms. Rather, the theta appears to be correlated with the echolocating vocalizations, at least in the brown bat [73]. Thus, it is not clear whether theta sequences would be an important mechanism for navigation in bats. It remains to be tested whether theta is present during the bat's more natural mode of navigation through flight, as opposed to very slow crawling in the lab, and whether the same principles of theta sequences in rats might apply to bats as they explore their environment remotely through echolocation. Theta is a complex, multi-dimensional phenomenon even within an individual species, and it will be informative to understand in more detail the commonalities and differences in the behavioral and cognitive correlates of theta across species.

Conclusions

In summary, the hippocampus provides a number of mechanisms that might support navigation, including place representations for association with values and actions, representations at multiple spatial scales for larger navigational tasks, short forward-ordered sequences which might support lookahead mechanisms to access information about locations immediately ahead of an animal, and longer replay sequences which might support learning and planning during offline periods. These mechanisms can be considered together as a highly dynamic representation of state that projects to downstream evaluation and action selection networks. One implication is that subcortical networks which were previously thought to mediate relatively inflexible learning in conditioning tasks may in fact mediate far more flexible learning and decision-making when driven by the hippocampus. Two key sets of issues remain relatively underexplored. First, few experiments have been conducted during spatial tasks that do not artificially constrain the animal's trajectory by the structure of the task or apparatus. Given technological constraints, it can be difficult to obtain sufficient sampling of place cell activity over an extended, two-dimensional environment when behaviors are restricted to navigationally directed trajectories. Second, there is very little direct evidence of a causal role for place cell phenomena in navigation. However, with recent advances in recording technology [84] and the ability to stimulate neural activity [85], these issues are likely to be addressed.