In these early training trials, all responses are initially equally active in motor cortex, leading to a high degree of response conflict and a strong initial STN Global NoGo signal. Nevertheless the Go signal for the correct response prevails in Striatum (via random initial synaptic weights or as a result of prior learning). A dopamine burst subsequently "rewards" the response, allowing the striatum to learn about Go vs NoGo activity states for each response at the time of the DA burst. This learning allows the striatum to facilitate selection of the rewarding response in future presentations of the same stimulus context. The mechanisms by which DA affects activity and plasticity are motivated by several biological experiments and are consistent with effects of dopamine D1 and D2 pharmacological agents on activity and long term plasticity.
In these trials a dopamine dip during negative feedback leads to increased NoGo activity and decreased Go activity (relative to the response selection phase), allowing the striatum to learn to avoid selecting that particular response in the future. Note that the model is never "told" the correct response (as in error-driven learning algorithms). Motor cortex continues to activate the incorrect response during negative feedback, but instead has to learn which responses lead to positive vs negative reinforcement by trial and error and associated changes in BG/DA. (The Output layer shows the response the correct response during feedback, but this is only used for display purposes and does not affect learning. )
In this trial the model has been trained for 25 "epochs" on a probabilistic learning task. Note that here, R1 "wins" the competition in motor cortex and sends output activation prior to its associated BG Go signal. This is because as a response is increasingly facilitated by the BG in response to an input stimulus, Hebbian principles drive learning directly between the stimulus input and response units in premotor cortex. In effect, the BG/DA system trains the cortical system. This is consistent with observations in both animals and humans that learning related activity occurs in the striatum prior to frontal cortex, and that the BG/DA system is particularly important in the learning of new behaviors, but less important for well ingrained habits. In this particular example, due to the probabilistic nature of the task, negative feedback was delivered.
After choosing one response (R3), a new stimulus Input is presented. Note that at the beginning of the trial, R3 motor cortical units are fully active while others are suppressed. But because R3 had not been positively reinforced in response to this new input stimulus, a BG NoGo signal suppresses the initial R3 selection and allows switching to an alternative response (R2). This R2 response is then reinforced with a dopamine burst because it was the correct choice in this particular task context.
In these trials, networks were faced with making a choice in response to two simultaneously presented stimulus cues (two columns of input units), each of which had been separately associated with a different response in the past. In this case, R4 is the correct choice, because it had been associated with the highest probability (80%) of reward, whereas R1 had been associated with 60% reward in response to the other stimulus. This is a high-conflict "win/win" decision, in which the STN is important for preventing premature responding. Noise in motor cortex was increased in this example for demonstration purposes. When the STN is intact it prevents early responding and allows integration of noisy signals; as result the model correctly chooses R4. In contrast, with the STN lesion (inactive STN), the model responds prematurely to R1 as it happens to become more active early in settling of network activity states and is impulsively facilitated.
In this spurious trial, the BG initially facilitates two responses simultaneously, which is not a good thing when having to make choices! However, note that when these two responses are fully excited in premotor cortex, the additional response conflict drives a second STN Global NoGo signal; this leads to excitation of GPi and inhibition of the Thalamus. The lack of bottom-up support for both responses makes it easier for one to dominate and suppress the other (via lateral inhibition that is present in cortex), leading to the selection of just one response. At this point, the conflict in cortex goes down, and the STN Global NoGo signal turns off.
Incorporating Norepinephrine Function into the Model
The below simulations explore how these effects play out within
the context of the overall BG/DA action selection circuitry (see Frank, Scheres & Sherman (2007) and Frank,
Santamaria, O'Reilly & Willcutt (2007) for simulation results,
discussion, and implications for ADHD). We showed that (a) the tonic
LC mode leads to increased representation of multiple cortical
responses, (b) more reaction time variability, and (c) more erratic
trial-to-trial response switching.
In the phasic LC mode, tonic LC firing is low but punctate phasic
bursts are elicited via top-down excitatory projections from premotor
cortex. In this manner stimulus-evoked premotor activity (which arises
from prior stimulus-response learning; see above) elicits a phasic LC burst, which in turn
reciprocally modulates the gain of premotor units and facilitates the
selection and execution of the desired response. These effects turn
out to be especially critical in the presence of noisy cortical
activity. To explore effects of LC/NE on noisy premotor activity, we
delay the stimulus onset so that noisy activity is present in premotor
cortex prior to processing of a task-relevant stimulus (as is likely
the case in natural environments, but is typically not simulated).
This model explores the effects of norepinephrine (NE) in modulating
cortical response selection processes, as simulated by Aston-Jones,
Cohen and colleagues (see Aston-Jones & Cohen, 2005, Annual Review of
Neuroscience). Like DA cells in the SNc, firing states of NE-releasing
neurons in the locus coeruleus (LC) come in both tonic and phasic
modes. In electrophysiological recordings, LC cells release phasic NE
bursts during periods of focused attention, infrequent target
detection, and good task performance. This phasic NE burst is thought
to reflect the outcome of the response selection process and serves to
facilitate response execution. In contrast, poor performance is
accompanied by a high tonic, but low phasic, state of LC firing. The
authors simulated the effects of these LC modes on action selection
such that NE modulated the gain of the activation function in cortical
response units (Usher et al, 1999). They showed that phasic NE
release leads to ``sharper'' cortical representations and a tighter
distribution of reaction times, whereas the high tonic state was
associated with noisy activity and more RT variability, as observed in
their empirical work with monkeys. They further hypothesized that
increases in tonic NE during poor performance may be adaptive, in that
it may enable the representation of alternate competing cortical
actions during exploration of new behaviors.
In these trials, noisy activity in premotor units prior to stimulus onset happens to favor the correct cortical response (R1) associated with the particular stimulus. In this "good noise" case, both tonic and phasic LC modes are associated with swift facilitation of the correct response.
In the "bad noise" case, noisy premotor activity prior to stimulus
onset happens to favor R1 units, but R2 is the correct response for
the particular input stimulus. Once the stimulus is presented premotor
R2 units begin to become active. This is because the network had
already been trained sufficiently such that cortical units had
developed strong synaptic strengths directly from the simulus units in
the input layer.
High tonic LC activity and associated NE nondiscriminately enhances
cortical activity, including initial noisy representations. Thus when
R2 units become active in response to the stimulus, these have to
compete with the already-active R1 units. In the BG model, in addition
to leading to increased inhibitory competition in premotor cortex
itself, the resulting increased response conflict drives a strong STN
Global NoGo signal (see above), slowing
reaction time until R2 can be selected and R1 suppressed. If the
correct response units happen to be more active during initial noisy
activity (as in the "good noise" case above), this slowing does not
occur. The resulting effects across multiple noisy trials lead to
increased RT variability, and somewhat slowed overall responses, as is
seen in ADHD. This same tonic NE / cortical noise can also lead to
exploration of new responses and erratic trial-to-trial switching of
responses. Indeed, we found that increased RT variability and
trial-to-trial switching were strongly correlated in non-medicated
ADHD (and not in controls), suggestive of a common mechanism ( Frank et al
(2007)). Critically, these measures were independent from other,
putative DA-dependent measures such as positive reinforcment learning
and working memory updating.
In contrast, the phasic LC mode performs more efficiently in the face of
noise: lower tonic LC activity prior to stimulus onset reduces the effects of
membrane potential noise in cortex. The phasic LC burst selectively enhances
premotor responses evoked by the stimulus, leading to reduced conflict and
swift response facilitation. The resulting distribution of reaction times is
more narrow for the phasic case, because it is less susceptible to
pre-stimulus noise. This phasic mode thereby supports exploitation in our model, as in the models proposed by Aston Jones & Cohen, and McClure et al (2006).
This graph from the supplement of Frank, Scheres & Sherman (2007) shows RT variability (standard deviation of the number of processing cycles needed to facilitate a response) and choice accuracy in a simple selection task,
as a function of tonic LC firing rate (with no phasic response).
Intermediate tonic LC levels are associated with high RT variability, while
high tonic (supra-tonic) levels are associated with narrower distributions,
with a cost in accuracy. This demonstrates the need for a dynamic LC modulation of cortical gain during exploitative behavior, with low tonic levels prior to a response, but increased levels to facilitate swift response execution when appropriate. This dynamic mode is associated with minimal RT variability (as in the very high tonic levels above), and also high accuracy (not shown here).