Model-Free Monte Carlo–like Policy Evaluation. Raphael Fonteneau, Susan A. Murphy, Louis Wehenkel, Damien Ernst. AISTATS 2010


(I accidentally had the wrong title here before, fixed)

  1. Seems similar in spirit to (same 1st author): Inferring bounds on the performance of a control policy from a sample of trajectories.
  2. Attempts to do value approximation by using corpus of SARSes, which is made of a number of “broken trajectories”
  3. Assumption everything is Lipshitz
  4. Strength is method doesn’t use FAs
  5. Stochasticity is tolerated
  6. Gives equations for bias and variance of monte carlo estimators
  7. Algorithm is setup to minimize discrepancy between rollout resulting from corpus vs that from true model if it was available
  8. Each sample in the corpus is used at most once (not sure why thats necessary)
  9. Sample used for the transition is one that minimizes a particular distance metric in the SxA space
  10. Gives bias and variance of their evaluator
    1. “When the sample sparsity becomes small, the bias of the estimator decreases to zero and its variance converges to the variance of the Monte Carlo estimator.”
Advertisements

One thought on “Model-Free Monte Carlo–like Policy Evaluation. Raphael Fonteneau, Susan A. Murphy, Louis Wehenkel, Damien Ernst. AISTATS 2010

  1. Michael Littman says:

    “Seems similar in spirit to (same 1st author): Inferring bounds on the performance of a control policy from a sample of trajectories.”: I assume that’s the paper you shared with me recently.

    “Stochasticity is tolerated”: Do they do anything interesting to extend the other work to handle stochastic transitions?

    “Each sample in the corpus is used at most once (not sure why thats necessary)”: I assume it’s to keep the statistical estimates from getting biased.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: