Confidence estimation as a stochastic process in a neurodynamical system of decision making

Network model.

We employed a spiking neural network model, which has been previously used to simulate a categorical decision of an analog feature, like motion direction (Furman and Wang 2008; Liu and Wang 2008). This model consists of 2,048 pyramidal cells and 512 interneurons. Both pyramidal cells and interneurons are modeled as integrate-and-firing neurons; excitatory postsynaptic currents from pyramidal cells are mediated by α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid (AMPA) and N-methyl-d-aspartate (NMDA) receptors, whereas inhibitory postsynaptic currents from interneurons are mediated by GABA receptors. Pyramidal cells are uniformly placed on a ring according to their preferred motion directions and continuously span 360° of possible motion directions (Fig. 1B), while the interneurons constitute a nonselective neural pool. The recurrent connectivity strength between two pyramidal cells is a Gaussian function of the difference between their preferred motion directions, whereas those from and onto the interneurons are broad and uniform (Fig. 1B). All the cells receive a background noise mediated by AMPA receptors, which is modeled as uncorrelated Poisson spike train. We used the neuronal and synaptic parameters from Furman and Wang (2008), which are fully specified therein, with a change of the background noise rate to 2,200 Hz, which ensures a choice is generated even if the motion strength is weak and stimulus duration is short. With these parameters, the network is endowed with winner-take-all competition so that only one of the neural pools wins (reaching an average population firing rate >50 Hz for at least 50 ms), and the decision is maintained in the form of a bell-shaped persistent activity pattern (“bump attractor”) in a delay period.

Simulation protocol of FD discrimination-decision task.

Our model assumes that neurons in area LIP incorporate sensory evidence (Cook and Maunsell 2002; Hanks et al. 2006; Roitman and Shadlen 2002) and reward signals (Platt and Glimcher 1999; Sugrue et al. 2004; Tobler et al. 2005). For simplicity, we assumed that the amount of reward for each target (i.e., two directional targets and a sure target) is associated with the instantaneous input strength of its current at the moment of the target onset (Soltani and Wang 2006). That is to say, the amplitude of the sure target input does not correspond to its physical properties (like the luminance) in the experiment; instead it is related to the behavioral significance of the sure target that a monkey learned, i.e., the amount of reward it received by choosing the sure target (Fig. 8).

In an FD version of the two-alternative direction-discrimination task (Fig. 1A), two directional targets, T_A (90°) and T_B (270°), are first presented to the network. A random-dot stimulus with net motion to T_A is presented at 500 ms after target onset. The difficulty of the task is modulated by the stimulus duration (randomly chosen from 110, 130, 152, 178, 208, 244, 289, 348, 439, and 627 ms) and the percentage of coherently moving dots (the motion strength). We modeled the external input to pyramidal cell i (at θ_i) as a sum of two target signals, I^tar_i(t) (tar = {A, B}; Fig. 1D, black line), and the motion stimulus, I^m_i(t) (Fig. 1C). The target inputs to T_A and T_B are identical:

I^tar_i(t) = I^tar(t)exp[−(θ_i − θ_tar)²/2σ²_tar]

where θ_A = 90°; θ_B = 270°; σ_tar = 10°.

I^tar(t) = I₁ + I₂exp[−(t − t_d − 200)/τ_d], t_d + 100 < t ≤ t_m + 80

I^tar(t) = I₃ + (I₁ − I₃)exp[−(t − t_m − 80)/τ_ss], t_m + 80 < t

where t_d = 400 ms and t_m = 800 ms are the onset times of targets and motion, respectively; τ_d = 500 ms and τ_ss = 15 ms are the time constants of the adaption and suppression, respectively; I₁ = 250 pA, I₂ = 50 pA, and I₃ = 60 pA. Specially, the target (motion, respectively) input onset time is 100 (200, respectively) ms after the target (motion, respectively) onset time, and the target input is suppressed by the motion stimulus with a latency of 80 ms (Roitman and Shadlen 2002); with the high intensities of the target inputs, winner-take-all competition between the two targets does not take place before the motion stimulus onset (Furman and Wang 2008; Liu and Wang 2008; Wong et al. 2007).

In simulation, motion input is modeled to imitate the neural response in the middle-temporal area (MT) to the random-dot stimuli. We constructed such a population activity as a Gaussian function with a tuning width independent of motion strength while motion presented (t_m + 200 < t < t_mo, t_mo is the moment of motion input offset)

I^m(i) = m₀ + coh {−m₁ + m₂exp[−(θ_i − θ_m)²/2σ²_m]}

where the motion strength 0 ≤ coh ≤ 1; net direction θ_m = 90°; σ_m = 40°. We kept the activity normalized, i.e., <I^m(i)> = m₀ = 4 pA; m₁ = 4.93 pA; m₂ = 25 pA.

In trials with T_s (θ_s = 180°), where there was the opt-out safe target presented (Fig. 1D, red line), we modeled its time-dependent current, I^s(i), as:

I^s(i) = I^s(t)exp[−(θ_i − θ_s)²/2σ²_tar]

which is added to the external input. We used:

I^s(t) = I₃ + I₄exp[−(t − t_s − t_mo)/τ_s]

for t > t_s + t_mo, where t_s is T_s input onset time after the motion input offset, t_mo, with a latency of 100 ms to the network after T_s onset. In simulations, we used t_s = 575 ms; τ_s = 90 ms; I₄ = 240 pA (see Fig. 8 for a discussion on the choice of I₄), expect Fig. 5 (t_s is equal to 575, 750, or 925 ms).

The network model is taken from Furman and Wang (2008), with a few parameter changes, i.e., background noise that ensures a choice is generated even if the motion strength is weak and stimulus duration is short, and a different set of parameter values for the choice target input, motion input, and sure target input that are adopted to the new experimental protocol of the Kiani-Shadlen experiment. Although the network was not originally designed for a confidence-estimation experiment, unexpectedly it can reproduce the behavioral and neurophysiological observations that are similar to those in the Kiani-Shadlen experiment [Figs. 1, B and C, and 2, A and B; Fig. 5, B and C in Kiani-Shadlen study (2009)]: 1) neurons inside T_A and T_B display firing activities with small differences when T_s is chosen, while their firing activity is divergent when T_s is shown but waived [Fig. 2, B and D, top, vs. Fig. 2, A and B, in Kiani and Shadlen (2009)]; 2) neurons inside T_s response field have weak and uninformative spontaneous activity before T_s onset and then exhibit a fast ramping followed by a decay after T_s onset [Fig. 2D, bottom, vs. Fig. 5, B and C, in Kiani and Shadlen (2009)]. In the model, we assumed that the input strength of each target goes to the same level attributable to the adaptation. For simplicity, we fixed I₄, while adjusting τ_s to match a majority of points on the performance curves of P_sure from Kiani and Shadlen (2009) [Fig. 3B, vs. Fig. 1, B and C, in Kiani and Shadlen (2009)].

We also studied the choice confidence in a reaction time (RT) version of the task. In this task, a network can generate choice at any time after the motion onset and, at the same time, report directly its choice confidence. We followed the same simulation protocols as those in the FD task without T_s, except that the motion input is terminated when one of the activity bumps crosses the decision threshold, 60 Hz for at least 50 ms. We measured the corresponding time, t_r, and calculated RT as t_R = t_r − t_m + 80, where 80 ms is the latency period for implementation of saccadic eye movement (Roitman and Shadlen 2002). In the FD version of the task, an initial choice is assumed to be made when one of the two competing neural pools reaches a decision threshold of 50 Hz for at least 50 ms after motion onset because the decision threshold of an FD task was experimentally observed lower than that of an RT task (Roitman and Shadlen 2002).

In FD tasks, we performed 1,500 trials at each motion strength and stimulus duration level, where T_s was not presented, and 3,500 trials at each motion strength and stimulus duration level, where T_s was presented. In Fig. 5, we simulated 1,500 trials for each condition. In RT task, we carried out 3,000 trials for each motion strength level. All the simulated behavioral data reported were computed using all trials for each simulation set. The integration method was a modified second-order Runge-Kutta algorithm with firing-time interpolation (Hansel et al. 1998) and a time step dt = 0.02 ms.

Measurements of activity trajectories.

We calculated the average response of the population of units associated with targets T_A, T_B, and T_s, namely R_A, R_B, and R_s, as the average firing activity of the neurons within 8.4° around each target with a time window of 100 ms preceding the time point (e.g., the moment of decision and onset of T_s) for R_A, R_B, and R_s in analysis, except for Figs. 2, B and D, and 7A. In Figs. 2, B, D, E, F, G, and H, and 7A, each trajectory or point represents the activity of a single neuron at each target. We applied a 100-ms Gaussian sliding window to smooth the peristimulus time histograms for the temporal evolutions of the firing rates of R_A, R_B, and R_s in Figs. 2, B, D, E, F, G, and H, and 7A.

Choice confidence assessment.

In the monkey experiment, as well as in our model simulations, the introduction of a sure target only serves as a probe examining the confidence of the system. That is to say, with carefully choosing the ratio of sure target reward to that of choice targets (i.e., I₄ in our simulation), one can access the choice confidence across trials. In our simulation, the probability of opting for the sure target is bounded (Fig. 3B); it thus represents a good choice for estimating confidence. Furthermore, we will show later in results that the probability of choosing the sure target, P_sure, reflects the uncertainty of a choice in an opt-out task. Here, we defined choice confidence, cc, as the probability of waiving a sure target, i.e., cc = 1 − P_sure, using the trials in which T_s is presented (binned by 0.5 Hz; Fig. 6A, black circles). We also assumed that this probability can be predicted as a function of the differential activity |R_A − R_B|. We then performed the fit of a logistic function between |R_A − R_B| and cc = 1 − P_sure. At each differential activity level, i, ccⁱ = 1 − P_sure is computed as the mean of the decision result across the sample trials, k, <sⁱ_k> for T_s. The decision result, s, is a binary variable, i.e., s = 1, if T_s is waived; s = 0, if T_s is chosen. To perform the fit, we used the firing activity within a 100-ms time window before T_s onset in the FD task (Fig. 6A) for R_A and R_B:

ccⁱ = 1 − P_sure(|Rⁱ_A − Rⁱ_B|) = b₁ + a[1 + exp(k|Rⁱ_A − Rⁱ_B| − b₀)]⁻¹

Using all trials in FD task, we obtained b₀ = 2.22 Hz; b₁ = 1.01; a = −1.01; k = 0.089 (R² = 0.98, Fig. 6A, red dashed line). Importantly, a real result here is to quantify confidence as a function of the neural activity. Confidence estimation is thus applicable to all trials, even without sure target presentation. We then used these estimated parameters to calculate ccⁱ for each sample trial in both FD and RT tasks, where R_A and R_B are the average firing rates within a 100-ms time window before T_s onset in FD task (Figs. 6, 7A, and 8C) and those before one of the bumps reaching a decision threshold in RT task (Fig. 9).

Network dynamics in an FD task with PDW at zero motion strength.

In experiments using single-unit recording, each neuron was recorded at a time, and its selectivity and dynamics were evaluated across trials, whereas, in our model, all neurons are monitored simultaneously in a single trial. At the population level, the ramping activity is demonstrated as the gradual development of a bell-shaped activity pattern (bump) around the direction of a selected target. The stimulated neural dynamics in Fig. 2 can be compared directly with single-neuron data from area LIP for a neuron with the preferred direction at T_A; T_A and T_B are equivalent to T_in and T_opp in the Kiani-Shadlen experiment (2009).

Figure 2A shows the spatiotemporal spiking activity of the network model in a trial without T_s presented. Although the input is identical to all the pyramidal cells at zero motion strength, the two activity bumps compete with each other through shared inhibitory feedback and stochastic recurrent dynamics. Eventually one bump ramps up, while the other one decays, leading to a categorical choice (Furman and Wang 2008; Wang 2002; Wong et al. 2007). The ramping-up bump is maintained by the persistent activity in delay (Compte et al. 2000). At the single-unit level, the firing activities of neurons at T_A and T_B (R_A and R_B, respectively) diverge over time after motion onset, leading to the choice of the network (T_A). The winning neural pool persists its firing activity until the end of delay (Fig. 2B).

In trials when T_s is presented (Fig. 2, C and D), again the attractor dynamics dictates the choice behavior of the network, and we identified a choice of T_s by the firing activity of neurons at T_s (R_s); the network selects T_s (Fig. 2D, dashed blue line) if R_s persists at a high rate, and it waives T_s (Fig. 2D, solid blue line) if R_s decays to a low rate. In the trial where T_s is waived, we observed the same divergence and persistent activity of R_A (choice) and R_B as that without T_s presented (Fig. 2, C, top, and D, solid lines); however, if the network selects T_s, R_A and R_B are indistinguishable at some intermediate rates without significant divergence (Fig. 2C, bottom, and D, dashed lines; R_A = 32.5 Hz, R_B = 33.6 Hz at T_s input onset), and then both decay to low rates as neurons around T_s win the competition. The average firing activities of R_A, R_B, and R_s across different motion strengths (100 trials for each condition) are shown in Fig. 2, E and F, which follow the same conventions as those in Fig. 2, B and D. All these are similar to the observed LIP neuronal data (Kiani and Shadlen 2009) from motion input onset time to the end of trial, which is crucial to capture the neural dynamics of choosing or waiving T_s, although a difference of neuronal activities could exist before motion input onset in our model compared to the experimental observations, which is not important to predict a choice of T_A, T_B, or T_s.

We further studied the neural dynamics that underlies a choice of the network among T_A, T_B, and T_s. Taking trials in Fig. 2C, for example, we visualized the neural trajectories in a 3D decision space (R_A, R_B, R_s) following the sequences marked in Fig. 2D. In the trial where T_s is waived (Fig. 2G, red line), the network first converges to a choice attractor (T_A), then leaves away from and returns to it again after the presentation of T_s; in the trial where T_s is selected (Fig. 2G, gray line), the network wanders around the diagonal line (R_A = R_B, R_s ≈ 0) and then converges to the attractor T_s. These trajectories in decision space imply that the presentation of T_s could induce a sure attractor, which behaves similarly to choice attractors; the network could thus act like a three-way competition after the presentation of T_s. To test this, we visualized more sampling trials at the same stimulus condition and explored the basin of attraction for each attractor (Fig. 2H). We found that, for trials choosing T_A, T_B, and T_s (Fig. 2H, red, green, and gray lines, respectively), the networks converge to the choice attractors T_A (near the R_A axis) and T_B (near the R_B axis) and the sure attractor T_s (near the R_s axis), respectively. The whole decision space is thereby separated into three attractor regions, and, because of this structure of decision space, the location of each neural trajectory at the moment of T_s onset (namely the initial location) can potentially predict the choice of a network (Kiani and Shadlen 2009). Particularly, if the initial location of a network is close to a choice attractor, it would eventually converge back to that choice attractor again after T_s onset, whereas, if its initial location is around the diagonal line, it more likely converges to a sure attractor. Notably, if the initial location of a network is between the diagonal line and a choice attractor, it could either continue converging to that choice attractor or change its mind to T_s (discussed later in Fig. 7). Therefore, our model demonstrates that a categorical choice of a network in this task could be generated by a three-way competition among attractors T_A, T_B, and T_s, which relies internally on the stochastic neural dynamics (Wang 2008).

Behavioral performance.

The performance of the model is quantified by the fraction of trials corresponding to a particular behavioral response. Figure 3 exhibits the probability of choosing T_s (P_sure) and accuracy (P_correct) for trials when T_s is not presented or T_s is shown but waived. At a fixed stimulus duration, our model shows that P_sure decreases while P_correct increases with the motion strength; P_correct improves in trials where T_s was shown but waived (Fig. 3A). Moreover, Fig. 3B shows that P_sure decreases with the stimulus duration, wherefore the network selects T_s more often for weaker motion strength or shorter duration; P_correct increases monotonically with stimulus duration for trials with or without T_s presented (Fig. 3B, right, solid and dashed lines, respectively). For a given motion strength and stimulus duration, P_correct is higher for trials where T_s is shown but waived than those without T_s presented, implying that P_sure is a probe of uncertainty (Kepecs and Mainen 2012; Whiteley and Sahani 2008). In conclusion, the model successfully reproduces the salient behavioral observations in the monkey experiment (Kiani and Shadlen 2009) (Fig. 3C).

Choice confidence as a logistic function of the differential activity.

Consistent with the neurophysiological observation (Kiani and Shadlen 2009), R_A and R_B undergo brief decreases after T_s onset in our model (Fig. 2C). This is because T_s stimulates neurons selective for the sure target, and their increased firing activity recruits more feedback inhibition that reduces R_A and R_B in a three-way competition (Fig. 2H). We therefore hypothesized that the network would opt to T_s if it has not converged to a stable attractor for T_A (R_A >> R_B) or T_B (R_A << R_B); the differential activity |R_A − R_B| at T_s onset determines P_sure.

We first observed that R_A and R_B could diverge in the late phase of delay. The long divergent time (>T_s onset) implies that there is a quasistable state at R_A = R_B, around which the network could wander, but eventually the network would escape from it and converge to a stable attractor, generating a choice. We next studied whether the network in a state around R_A = R_B would opt to T_s (e.g., Fig. 2H, gray lines). For individual sample trials, we visualized R_A and R_B against each other on a decision space. Figure 4, A and B, shows that the activity of the network falls down along a diagonal line and then separates into three groups for choices of T_A, T_B, and T_s. In trials where T_s is waived, the network converges to one of two stable attractors, T_A or T_B (Fig. 4, A and B, gray lines), while, in those where T_s is selected, the network walks randomly around the quasistable state at R_A = R_B ≈ 35 Hz (Fig. 4, A and B, black lines). In summary, once the network converges to a stable attractor before T_s onset, T_s is waived; if it wanders around R_A = R_B, the network is likely to opt to T_s.

The studies of the similar attractor models (Wang 2002; Wong and Wang 2006; Wong et al. 2007) showed that early divergence of R_A and R_B (bias to one attractor) determines the probability of choosing T_A and T_B as a function of the motion strength and stimulus duration. One can thus expect that early divergence would also result in a decrease of P_sure as a function of the motion strength and stimulus duration. To examine this, we investigated the distributions of R_A − R_B with different motion strengths at T_s input onset. Figure 4, C and D, shows that the percentage of the trials around R_A = R_B decreases with higher motion strength or longer stimulus duration, resulting in a decrease of P_sure.

Although the early divergence plays a dominant role in the network dynamics, network continues to evolve via NMDA-mediated reverberatory dynamics; the slow stochastic dynamics could thus drive the network away from R_A = R_B in the later phase of the delay. Consequently, P_sure depends on T_s onset time. Figure 5A displays the evolution of the distribution of R_A − R_B at different times from motion offset, demonstrating that the slow stochastic dynamics also plays an essential role in the behavior of the network. Across trials, our model predicts that P_sure decreases with longer T_s onset times (Fig. 5B) because more trials settle down to a stable attractor later in the delay, i.e., the percentage of trials with R_A = R_B decreases with T_s input onset time. Interestingly, we found that P_correct was nearly constant with different T_s input onset times (Fig. 5C). This happens because P_correct ≈ r_AB/(1 + r_AB), where r_AB is the ratio of the number of trials at attractor T_A (choice) to that at attractor T_B at T_s onset and is saturated at 575 ms after motion offset (Fig. 5A). In short, P_sure is directly related to the percentage of trials around R_A = R_B, whereas P_correct is associated with the number of trials at T_A and T_B at T_s onset. In this sense, there is a dissociation of confidence estimation from performance (Graziano and Sigman 2009; Graziano et al. 2010). Moreover, without T_s presented, the network continues to converge to T_A or T_B via stochastic dynamics, and the probability to one of them is biased and relies on R_A − R_B in the early phase of delay (Fig. 5D). In conclusion, we found that |R_A − R_B| at the moment of T_s input onset determines P_sure probabilistically and reflects a degree of the stability of a choice: if a categorical choice is achieved but with small |R_A − R_B|, it could be altered to T_s; whereas, if |R_A − R_B| is large, the choice of the network would not be changed by T_s.

Here we define choice confidence as a function of the instantaneous differential activity |R_A − R_B| at sure target input onset time for each trial i, i.e.,

ccⁱ = f(|Rⁱ_A − Rⁱ_B|).

In our model, |R_A − R_B| shows the position of a network in the (R_A,R_B) plane (Fig. 2, G and H) related to choice attractors in the decision space, i.e., the larger |R_A − R_B| is, the closer a network is to a choice attractor T_A or T_B; f(·) is therefore required to be an increasing function. In the previous studies (Beck et al. 2008; Kepecs and Mainen 2012; Kepecs et al. 2008; Vickers 1979), functions f(·) were given in a variety of ways. One can picture that, as long as f(·) is a monotonic function, we can always equate f(·) from one model to another. Of note, our definition of the confidence stems from the structure of the attractor basin in decision space (Fig. 2, G and H), i.e., if a choice is confident, then it is easily resistant to the other external input such as a sure target, whereas confidence estimations from models like Beck et al. (2008) and Kepecs and Mainen (2012) are compared directly to log odds of a choice in Bayesian framework. In the studies of Beck et al. (2008), they found from both the experimental data and their model that log odds of choice at A, namely confidence across trials for choice A, is proportionate to <R_A> − <R_B>, where <·> is the average across trials. However, such a read-out of confidence would predict a strong correlation between confidence and performance on single trials, which is somewhat inconsistent with experimental observation of a broad performance variation in different confidence categories (Graziano et al. 2010; Juslin and Olsson 1997). Second, this “optimal decoder” <R_A> − <R_B> relies explicitly on the equal variance hypothesis for likelihood (Kepecs and Mainen 2012) or “left-right” symmetry of a linear decoder (Beck et al. 2008). It remains unclear what the biologically plausible mechanisms are to achieve such a fine-tuned neural circuit to compute confidence signals in these models.

In our model, confidence is defined using a monotonic increasing function of |R_A − R_B|. Particularly in an opt-out task, confidence can be probed by the probability that a choice stays in its attractor after presenting a sure target. If the choice is confident at |R_A − R_B|, then this probability, 1 − P_sure, is low. Using this probe, we found that choice confidence increases as a sigmoid function [i.e., function f(·)] of |R_A − R_B| (Fig. 6A). Next, we asked whether, across trials, our definition of confidence can also reflect probability of correct at each |R_A − R_B| level as those defined in Bayesian framework (Beck et al. 2008; Kepecs and Mainen 2012; Kepecs et al. 2008). This seems possible as indicated from Fig. 5D. Moreover, a detailed analysis should compare probability of correct choice and confidence simultaneously. We performed this analysis using trials without presenting T_s, where we computed both the performance and the confidence at the end of decision averaged across all stimulus conditions (Fig. 6B). Figure 6B demonstrates that confidence in our model increases monotonically with the performance. Importantly, our model indicates that confidence can be computed as a function of the instantaneous neural activities (like a population code; Beck et al. 2008) at any time in a decision circuit without explicit use of elapsed time for integration of the sample (Drugowitsch et al. 2012; Kiani and Shadlen 2009; Kiani et al. 2014; Moreno-Bote 2010). Therefore, although confidence in our model is not defined as a log odds function of the choice (Beck et al. 2008; Drugowitsch et al. 2012; Kepecs and Mainen 2012; Kepecs et al. 2008; Kiani and Shadlen 2009; Kiani et al. 2014), confidence can be a good measurement of the subjective correct rate across trials. Importantly, on single trials, choice confidence in our model is dissociable from performance (Graziano et al. 2010), whereas Bayesian models would predict a strong correlation.

Despite the similarity of f(·), choice confidence in our model is, however, conceptually distinct from those from Bayesian decision models because our definition of confidence fundamentally comes from the structure of the attractor basin in the decision space. Therefore, our model predicts that confidence would be different when estimated at the different times after motion offsets (Fig. 5), and that would be nearly the same in a Bayesian model (Beck et al. 2008; Drugowitsch et al. 2012; Kiani and Shadlen 2009; Moreno-Bote 2010). To test this, we estimated choice confidence using neural activities R_A and R_B in trials without T_s presented at different times after motion offsets. We first estimated the choice confidence at 575 ms after motion input offset (Fig. 6C; compared directly with Fig. 3B, left), where the distribution of R_A − R_B is still evolving, namely the confidence estimation after a short delay (Fig. 5A). One can thus expect an increase of choice confidence in trials with longer stimulus durations (Fig. 6C), according to the variation of the bimodal distribution of R_A − R_B at different stimulus durations (Fig. 4, C and D). We next estimated the choice confidence at 1,550 ms after motion input onset, i.e., the same time of a trial, where the internal noise is nearly identical at different stimulus conditions and the strength of the input dominates the choice confidence of the network. In this case, one would expect that confidence should increase as a function with the motion strength and stimulus duration for a noiseless integrator (Beck et al. 2008), unless it is bounded (Drugowitsch et al. 2012; Kiani and Shadlen 2009). While in our model, the attractor dynamics implies that the bimodal distribution of R_A − R_B depends dominantly on the early divergence (Fig. 4, C and D). As a result, Fig. 6D shows that all confidences saturate at stimulus duration >400 ms, suggesting that the early evidence has the greatest effect on confidence estimation. Of note, the saturation time prolongs with lower motion strength.

Low confidence results in changes of mind to T_s.

The whole dynamic space, R_A-over-R_B decision space, can be divided into three regions: choice attractor regions (R_A >> R_B or R_A << R_B) and an unstable region in between them (Wang 2008). In the previous study, we focused on the trials along the diagonal line (R_A = R_B), where a choice of network remains undecided before the presentation of T_s. We then investigated the dynamics of networks that are between the diagonal lines (R_A = R_B) and a choice attractor (R_A >> R_B or R_A << R_B) on R_A-over-R_B decision space preceding the presentation of T_s, where a trial could be still in the unstable region (and thus could go to the sure attractor after T_s onset) or in a stable attractor region (where the network trajectory would stay in the same choice attractor even after T_s onset). For these trials, we could define an initial choice of the network by its nearby choice attractor T_A (T_B, respectively), where R_A (R_B, respectively) fires above a decision threshold (>50 Hz). Particularly, we explored under which condition the network would more likely continue converging to the attractor of its initial choice, or shift to the sure attractor, when T_s is offered.

Figure 7A compares the neural activity in trials with low- and high-confidence initial choices, the analysis of which is on single trials and is missing in Kiani and Shadlen (2009). In low-confidence trials (Fig. 7A, top), only one of the activity bumps fires above the decision threshold, but the other remains similar (|R_A − R_B| is small). After T_s onset, both R_A and R_B decay to a low level, while R_s grows to a high level and T_s is chosen. By contrast, in high-confidence trials (Fig. 7A, bottom), when one of the firing rates reaches a steady state, the other is much smaller (|R_A − R_B| is large). Neurons activated by T_s are suppressed, and T_s is waived. In the latter case, the activity of the winning neural pool exhibits a brief dip upon T_s input onset and then ramps up again to its steady state.

Across trials, the probability of changes of mind to T_s is negatively correlated with choice confidence, i.e., the network exhibits low confidence in trials at low motion strength (Fig. 6C) and high probability of changes of mind to T_s (Fig. 7B, left). To further test whether the network bases the probability of changes of mind to T_s on its performance and confidence, we categorized the trials with initial choices, where either R_A or R_B reaches a decision threshold, 50 Hz (if both of them do not reach the decision threshold, we considered the choice remaining undecided at T_s onset), into correct and error groups and found that network changes its choice to T_s more often in error trials. Furthermore, the probability of choosing T_s in correct (error, respectively) choice decreases (increases, respectively) with the motion strength (Fig. 7B, right). This finding is reminiscent of the experimental observation that, in a decision making task with a delayed reward, animals moved back to self-restart port more often when the task became more difficult or the confidence was low (Kepecs et al. 2008).

In conclusion, we identified two possibilities for choosing T_s; either an initial choice was not made (along the diagonal line; Fig. 2), or it was made with a low confidence (between the diagonal line and choice attractors; Fig. 7A, top). For the latter case, |R_A − R_B| reveals the confidence about an initial choice; low confidence of a choice is likely to result in changes of mind to T_s.

A sure target as a probe about the confidence of the system.

The introduction of a sure target plays a role of probing the confidence of the system. Specifically, in the monkey experiment, the physical luminance of the sure target was the same as the choice targets. Monkeys were trained to understand what the sure target meant behaviorally, which depended on the amount of reward by choosing it. Therefore, in our model, the amplitude of the sure target input (I₄ in our model) does not correspond to its physical properties but is related to the behavioral significance of the sure target that a monkey learned as the amount of reward he receives by choosing the sure target. One can imagine that, if choosing the sure target yields a negligible (significant, respectively) amount of reward, monkeys would never (always, respectively) have learned to choose it. To test this, we studied the effect of T_s input strength, I₄, on the behavioral performance at a fixed motion strength level (i.e., 12.8%). We found that P_sure increases as a function of T_s input strength (Fig. 8A). When T_s input strength is low (e.g., I₄ = 192 pA), T_s is always waived; when T_s input strength is high (I₄ = 288 pA), T_s is mostly chosen, as stimulus duration is short. Moreover, in the trials where T_s is shown but waived, our simulation predicts an increase of correct rate at high T_s input strength (Fig. 8B). At the network level, these observations in Fig. 8, A and B, still follow a three-way competition among R_A, R_B, and R_S, e.g., when input of the sure target is weak (strong, respectively), it always behaves like a loser (winner, respectively). Last, we examined whether, in a range of T_s input strengths (a selected range of amount of T_s rewards), a sure target can serve as a probe about the confidence of the system, when the network applies the attractor dynamics. We assessed the choice confidence as a function of |R_A − R_B| at the moment of T_s input onset for different choices of T_s input strengths. Figure 8C shows that, on average, choice confidence is identical for different T_s input strengths (and increases as a function of stimulus durations), and P_sure decreases as a linear function of choice confidence for a broad range of T_s input strengths, i.e., in this range, 216 pA <I₄ < 264 pA, a sure target in our model can be considered as a probe about the confidence of the system. Therefore, a sure target is only a probe, and the confidence measure is valid even without it.

Assessment of choice confidence in an RT task.

In our model, confidence can be read out at any time and increases as a function of stimulus duration in an FD task. One may thus argue that the network would exhibit high confidence despite the task difficulty if it freely controls the viewing duration of the stimulus. However, classical literature about confidence in cognitive psychology (Vickers 1979) emphasizes an inverse relationship between confidence and response time, which can be potentially tested in an RT version of discrimination task [developed previously by Furman and Wang (2008)] with direct assessment of choice confidence. This distinguishable difference between confidence estimation in FD vs. RT task in fact comes from two distinct processes; whereas longer viewing time in an FD task enables more integration of evidence (confidence thus increases with motion viewing time), a longer RT means a higher task difficulty in an RT task (confidence thus decreases instead with motion viewing time). We thus want to further test whether our model can nicely explain such a contrasting observation. To do this, we designed an RT version of discrimination task with direct assessment of choice confidence (Fig. 9A); the network integrates the motion input until the neurons selective for one of two alternatives fire above a decision threshold and reports the confidence as the function of the instantaneous |R_A − R_B| (Fig. 6A) at the moment of choice [a similar human behavioral experiment is performed and reported by Kiani et al. (2014) recently].

Our model exhibits the typical psychometric and chronometric curves of a two-alternative discrimination task (Churchland et al. 2008; Roitman and Shadlen 2002), i.e., P_correct increases, whereas RT decreases with the motion strength (Fig. 9, B and C). Importantly, weaker motion strengths are associated with longer RTs, where |R_A − R_B| will be less at longer RTs. Choice confidence thus increases with the motion strength [Fig. 9D; see also Fig. 5 in Beck et al. (2008)] and is positively correlated with the behavioral performance across trials (Barthelmé and Mamassian 2010) (data not explicitly shown). We also found that choice confidence decreases as an inverse function of RT (Fig. 9E), which agrees broadly with the human behavioral observations (Vickers 1979). Although an erroneous choice could be associated with high confidence (Graziano and Sigman 2009), the average |R_A − R_B| across trials is higher in correct trials than that in error ones (Wang 2002). Therefore, in our model, confidence increases (decreases, respectively) with motion strength in correct (error, respectively) trials (Fig. 9F), consistent with human studies (Pierrel and Murray 1963).

Moreover, we studied correlation between the choice confidence and decision accuracy. Figure 9, B and D, implies that choice confidence is positively correlated with behavioral performance across trials. Although confidence in our model does not directly represent a subjective estimation of performance (like that in Beck et al. 2008; Drugowitsch et al. 2012; Kiani and Shadlen 2009), one can estimate the subjective performance from choice confidence using a monotonic function, g(·), in Fig. 6B. We can thus compare directly our confidence score with performance to study the “hard-easy” effect (Juslin and Olsson 1997). Here we defined underconfidence score as the difference between the choice confidence and accuracy, cc − P_correct [one can also use g(cc) − P_correct], and a hard-easy effect is the observation that the underconfidence score decreases as a function of task difficulties, i.e., in the easy (difficult, respectively) trials, the report is more likely to be overconfident (underconfident, respectively) cc − P_correct > 0 (cc − P_correct < 0, respectively). Figure 9, G and H, shows the variation of underconfidence scores as a function of choice confidence for the FD task with a delay of 627 ms and RT task, respectively; both display the hard-easy effect in the reports. Of note, these results still hold true when comparing estimated subjective performance from choice confidence with the behavioral performance using g(cc) − P_correct. Such a hard-easy effect in our model mainly stems from sampling of stochastic neural dynamics; sampling duration thus influences the underconfidence score in our model. When the sampling duration is short, the network behaves with more overconfidence. To test this, we compared the scores in the FD and RT tasks, where the average sampling durations in RT tasks are longer than those in FD tasks at low motion strength (from 0 to 6.4%). Consequently, the network exhibits overconfidence more often at low motion strengths in FD task.

Comparison with existing models.

Computational schemes have been proposed for the study of confidence (Kepecs and Mainen 2012; Kepecs et al. 2008; Kiani and Shadlen 2009; Moreno-Bote 2010; Ratcliff and Starns 2009; Rolls et al. 2010a, 2010b; Vickers 1979). These models can be classified into Bayesian inference models and neural network models.

In Bayesian inference models, one can either compute confidence based on a single decision variable (Drugowitsch et al. 2012; Kiani and Shadlen 2009) or an optimal population code (Beck et al. 2008). Kiani and Shadlen (2009) proposed that confidence could be defined in terms of the log posterior ratio for the two choices given the position of a decision variable and elapsed time at decision, using DDM. This looks promising, yet it remains unclear what is a direct representation of a decision variable exclusively for a choice. Moreover, for the RT version of the task, this kind of model implies that the position of a decision variable at decision time would be a deterministic function of RT (either a constant or a time-varying function like that in Drugowitsch et al. 2012); one can thus find that confidence would also decrease deterministically as a monotonic function of RT (Drugowitsch et al. 2012; Kiani and Shadlen 2009; Volkmann 1934) on single trials. This idea, however, failed to explain the widely overlapped RT distributions in different confidence categories (Ratcliff and Starns 2009). Such a strong correlation between confidence and RT or performance can be eliminated through a two-stage DDM (Pleskac and Busemeyer 2010), where additional process for confidence is required. Nevertheless, in our model, performance, RT, and confidence are naturally dissociated with each other on single trials. Importantly, in a classic DDM model, sensory evidence contributes equally in time to confidence estimation, whereas, in our model, confidence estimation would more be dominated by the early sensory evidence. Last, in the Kiani and Shadlen model, choosing T_s uses a hard thresholding process and has little to do with neural activity at T_s response field, whereas, in our model, it generates from the same sampling of stochastic neural dynamics as the other choices (as indicated by data in Fig. 5, Kiani and Shadlen 2009).

On the other hand, the optimal population code model (Beck et al. 2008) posits that confidence could be estimated as the instantaneous differential activity, |R_A − R_B|, without explicit use of RT as our model. A notable difference between our model and theirs is that the optimal population code model requires LIP neural circuit to be a fine-tuned noiseless integrator. This can be easily tested experimentally because our model predicts that confidence estimation would differ at different times in the delay, whereas their model would expect that to be constant. Generally, these Bayesian inference models (Beck et al. 2008; Kiani and Shadlen 2009) assert that confidence must be based on explicit neural representation of probability functions, such as likelihood at any moment in time and in single trials. Our model demonstrates an alternative perspective. Whereas probability representations may be a perfectly valid mathematical description of the aggregated statistics across trials, they should not be confused with what actually happens in single trials, which is stochastic neural dynamics.

Insabato et al. (2010) and Rolls et al. (2010a, 2010b) extended the model of Wang (2002) to account for the confidence estimation and its behavioral readout. Insabato et al. (2010) argued that confidence can be read out as a function of R_A + R_B, and Rolls et al. (2010a, 2010b) claimed that it can be further approximated as a function of the neural activity of the winning pool. All these models showed some consistencies with the existing data. However, as discussed in our model, neither of them can exclusively demonstrate the position of a neural trajectory related to the choice attractor and thus the choice confidence at any time during a decision. These results seem only true at the moment when a decision is made exactly around a choice attractor in the decision space, where |R_A − R_B| ≃ |R_A + R_B| ≃ max(R_A,R_B) because min(R_A,R_B) << max(R_A,R_B). Therefore, these models would fail to predict confidence using a fixed decision boundary or capture the relationship between neural activities in LIP with P_sure or high-confidence errors in single trials. Alternatively, our model does not require a time-varying decision threshold, estimates confidence simply as a function of instantaneous |R_A − R_B| at the moment of choice on single trials, and can correctly reproduce the salient behavioral relationships between confidence, RT, and performance on single trials and those across trials.

Confidence rating is important for monitoring cognition when there is uncertainty, and two types of uncertainty should be distinguished, namely Brunswikian (external) uncertainty originating from incomplete states of knowledge (noisy or ambiguous sensory data) and Thurstonian (internal) uncertainty attributable to variations intrinsic to the brain (Juslin and Olsson 1997). The noise level in a decision circuit has only recently begun to be examined experimentally (Brunton et al. 2013). Our work provides a computational framework to detect these two effects using a spiking-neuron circuit. Our model can also be extended in several important ways. It still remains an open question how confidence estimation, as a sigmoid function of the differential activity in downstream neural circuits, can be read out for a direct report and to guide future behavior. In fact, confidence is commonly assessed without a verbal report using a two-stage PDW task; subjects perform a first-order discrimination task and then make a high-low bet on the outcome of the decision (Middlebrooks and Sommer 2011, 2012; Smith 2009) (see also Kiani et al. 2011), where the probability for a high bet is considered as a readout of confidence estimation. A plausible neural circuit for explicit representation and memory of a confidence signal is needed for the two-stage PDW (Komura et al. 2013; Middlebrooks and Sommer 2011, 2012) and should be examined in the future. A biologically plausible neural circuit to computing |R_A − R_B| involves neurons in pulvinar (Komura et al. 2013), where the neurons fires highly in nonsure-target trials and low in sure-target trials. Moreover, Kiani et al. (2014) recently found that confidence could also decrease in error trials with the difficulty of task, which poses a challenge to our model prediction (Fig. 9F). One possible direction in the future is to understand the mechanism of error trial. Nevertheless, what is robust about the prediction in our model compared with their observation is that the difference of the confidence between correct and error decreases with the difficulty of task (Lak et al. 2014). Second, for the sake of simplicity, we assumed that the amount of T_s reward is encoded as the onset strength of its target input, which mimics the firing activity of midbrain dopamine neurons in response to the targets with different amounts of reward (Tobler et al. 2005). Our model predicts that both P_sure and P_correct increase with the reward of T_s (data not shown). This then brings up two inquiries for future studies: 1) what a reasonable amount of T_s reward used to measure confidence in a PDW task would be (Dienes and Seth 2010; Fleming and Dolan 2010; Persaud et al. 2007) and 2) how the amount of T_s reward used is learned through neural dynamics and applied to the decision circuit (Soltani and Wang 2006). Moreover, one can extend our model to investigate confidence signals for multiple-choice decision tasks and effects of microstimulation on confidence (Fetsch et al. 2014). Specifically, one can incorporate known effects of microstimulation on MT inputs in our model to perform the experiment of Fetsch et al. (2014) using computer simulation and then test its effects on confidence. Finally, confidence may be represented in a distributed network in the brain (Del Cul et al. 2009); the dynamical nature and computational principle remains to be elucidated in future research. In conclusion, we found it remarkable that a previously established model of decision making (Furman and Wang 2008) naturally accounts for all the salient behavioral and neurophysiological observations of the Kiani-Shadlen experiment (2009). Furthermore, it reproduces the observation that confidence decreases with response time in an RT version of the task. The model also offers testable predictions about the changes of mind and, unexpectedly, the hard-easy effect observed in human studies, which naturally emerges from the model. Taken together, our work establishes that a dynamical system of stochastic neural population can underlie even the seemingly abstract metacognitive concept of confidence.