Motivation of Extended Behaviors by Anterior Cingulate Cortex. Holroyd, Yeung. Trends in Cognitive Sciences 2012

  1. Proposes that anterior cingulate cortex (ACC) performs selection and “maintenance” <learning?> of options
    1. This view can be seen as a way of tying together two different believed functions of ACC
    2. One theory is that ACC is seen as being related to motivation and starting behavior (lesions to ACC may lead to what seems to be outwardly catatonic behavior such as not responding to questions or speech even though physically the person is completely capable of doing so)
    3. The other is that it is thought to be related to cognitive control and RL
  2. Prior theories of ACC function actually are related to the deployment of cognitive control, as opposed to cognitive control itself
  3. Argue that differences in findings is due to narrow examination of deficits
  4. “In this Opinion we argue that ACC supports the selection and execution of coherent behaviors over extended periods…” which is hierarchical RL
  5. “In this view, ACC is more concerned with the selection and maintenance of the task itself than with the minutiae of task execution.  Thus, ACC would be responsible for engaging in a psychology experiment until its completion as opposed to implementing subtle behavioral adjustments along the way.”
  6. Previous work by Matt proposes that much of hierarchical control can be done in prefrontal cortex “… a region widely believed to be involved in supporting task sets […].”
    1. In this work they argue for ACC doing this instead of PFC
  7. Based on actor-critic model, but this work specifically proposes that ACC is “at the apex of bot pathways.”
    1. ACC selects and maintains options
    2. dorsolateral prefrontal cortex  and motor structures execute options (the actor)
    3. Orbitofrontal cortex and ventral striatum (the critic) evaluates progress to the terminating state of the option
  8. “… the dorsal striatum implements the policy of the actor, for example, by stopping the car at red lights…”
  9. Ventral striatum is the critic does long term credit assignment for actions selected
  10. “… oribtofrontal cortex provides the ventral striatum with information related to abstract goals, consistent with its role in contextually based action and reward evaluation […]; this information affords the basal ganglia flexibility to learn not only about primary rewards and punishments … but also about goal-related outcomes.”  <not clear to me what they mean exactly by this distinction>
  11. A few problems with current models
    1. Generally models have basal ganglia learning stimulus-response mappings, but this explanation seems to simplistic to describe how we can go about complex real world behaviors, as something else would be needed to get learning to scale.
    2. “… the architecture does not specify what task DLPFC should implement nor what goal orbitofrontal cortex should take as appropriate for the current task context.”
    3. Doesn’t determine “… the degree of vigor with which the task should be executed.”
  12. The ACC (and its proposed function in terms of HRL) removes these problems
  13. “…ACC decides what task to perform and then directs DLPFC to implement that task, which in turn provides top-down biasing signals to the dorsal striatum to facilitate execution of the chosen policy.  Thus whereas ACC selects the option-specific policy, DLPFC and the dorsal striatum together execute that policy… Furthermore, orbitofrontal cortex associates the termination state of each option with pseudo-reward, providing contextually appropriate reward information to the ventral striatum…”
  14. ACC also dictates how much effort should be expended in executing the option, and when to terminate the option
    1. If ACC doesn’t push much, basal ganglia will dominate behavior with actions selected at the low-level it functions on
  15. ACC also works on dopamine
  16. Theories hold that both phasic (short) and tonic (long) dopamine signals are used for shaping behavior
  17. From the HRL model,
    1. Low dopamine levels do option selection: facilitating “… the gating of a high-valued option into working memory…”
    2. High dopamine levels  do option maintenance: maintaining “… information in working memory until the option is completed …” 
  18. Further evidence of ACC for higher level behavior is patients with ACC lesions have mostly unimpared functioning in tasks where behavior is simple immediate stimulus-response tasks
    1. Conversely, lesions in ACC are associated with the lack of initialization of more effortful or spontaneous behavior.  The slowdown in initialization occurs because ACC is most active when starting an option
  19. Consistent with proposal ACC also activates when switching between high-level tasks of a problem

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: