RL in Nonstationary Environment Navigation Tasks. Lane, Ridens, Stevens

  • Uses deictic (not familiar with this) representations that lead to partial observability, but seems to also allow for generalization
  • “The goal of the paper is… to study the effects of representational choices and the topology of the environment on an already well-understood RL algorithm.”
  • They claim dealing with nonstationarity is an instance of a more general generalization problem.  I can see how this is true, but it isn’t how I always think about it
  • The representation used here is in terms of logical predicates that describe the state, this allows for more generalization than just enumerating states
    • Predicates describe properties of objects, n-ary predicates describe relationships among objects
  • Problems arise when the relational language can’t adequately distinguish between two configurations that require different policies
    • “The difficult, then is ensuring that the relational representation captures enough of the state description to function effectively, while discarding enough to generalize well”
  • Strategy employed is of state envelopes, that the agent maintains an explicit representation of states near the envelope and disregards anything outside it (basically makes the agent work in a local fashion).  Here the envelope is centered around the agent
  • Discuss a gridworld with multiple types of terrain
  • Say the classic discrete state representation doesn’t work if environment (such as goal location changes)
  • Results applied to Q-learning.  A very low γ is used, which sort of creates an envelope around the state anyway, so naturally the representation with the smaller state space will learn faster, as I don’t think it can learn much better with the small γ
  • Doesn’t seem that nonstationary to me, because with their method they are representing everything that is needed to remove nonstationarity, whereas it seems like QL is deprived that information (such as where the goal is)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: