Informing sequential clinical decision-making through reinforcement learning: an empirical study. Susan M. Shortreed · Eric Laber · Daniel J. Lizotte · T. Scott Stroup · Joelle Pineau · Susan A. Murphy. Machine Learning 2010

  1. Primarily concerned with Schizophrenia treatment, but discusses a number of factors relevant to other types of treatment:
    1. No controlled exploration
    2. Limited data
    3. Partial observability
    4. Importance of understanding level of uncertainty
  2. Corpus covers 1460 patients in a two stage clinical trial
  3. Specifically mention issues of drug resistance.
  4. Defines state as patient’s history
  5. Using SMART trials: sequential multiple assignment randomized trials
  6. Using finite horizon MDPs (since it was a 2-stage study, the horizon is length 2)
  7. The study lasted 18 months, with data recorded every 3 months.  20 demographic variables and 30 variables relevant to treatment progress
  8. Talk about patient dropout as problematic in two ways: reduces data therefore increasing variance, and introduces bias.
    1. Removing data from trial dropouts can also introduce another type of bias, so they left it in
  9. On average symptoms improved over the course of the study, but on a case-by-case basis there was a very large amount of noise/variance
  10. An additional issue in this particular case is that all the medications were expected to perform similarly, so teasing them apart would be tricky
  11. Discuss use of multiple imputation where (I think) data that is missing is filled in probabilistically (repeatedly) and then that completed data is treated as the true data
    1. The exact method of multiple imputation used is outlined
  12. They use LSPI – want to use linear method because it is simple and there is concern related to use of high variance training data
  13. Not sure I understand their method of voting for treatment.  Looks like they base results on the results of all possible 2-permutations of treatment together (saying goodness of treatment i is as good as the average of doing i and then all other possible treatments) why not i is as good as it is with the best possible treatment?
    1. Perhaps it could be argued its intended to reduce impact of variance, but that isn’t stated.
    2. They mention choosing to do “double bootstrapping” to reduce variance, but not as an argument for the method as a whole
  14. The method of estimating confidence intervals is different from the traditional Hoeffding bounds
    1. Use bootstrap resampling methods
  15. In all cases there is at least some overlap for the suggestion of optimal treatment
  16. Mention that SMART studies are underway for treatment of autism, substance abuse, and ADD
  17. “While this foundation is a good start, using RL methods to optimize treatment policies is not as simple as applying off-the-shelf RL methods. Though the planning horizon is very short (only 2 stages in our particular application) and the exploration policy is fixed, upon closer inspection, a number of previously unexplored challenges arise when tackling prob-lems in this domain. In particular, our case study highlights issues such as pervasive missing data, the need to handle a high-dimensional and variable state space and the need to communicate the evidence for a recommended action and estimate confidence in the actions selected by the learned policy.”
  18. “The adaptive confidence interval method presented here assumes that the linear approxi-mation provides a high quality approximation to the optimal Q-function. The adaptive confi-dence interval method has not yet been generalized for use with a non-linear approximation to the Q-function such as, for example, trees or a nearest neighbor estimators.”

Notes on what type of data the algorithm requires/consumes:

  1. Very short horizon (2 step)
  2. Small action set (5 different medications).
    1. Initial action/medication used was randomized
  3. Data was recorded every three months
    1. 20 demographic variables recorded at start
    2. 30 variables measured over time
    3. Examples are:
      1. Symptom levels
      2. Quality of life
      3. Side effect burden
      4. Medication compliance
  4. They differentiate between two types of state variables, there are the normal ones as we think of them, and then other state variables that “can aid in the accurate estimation of the Q-function, but that are not needed to inform the policy because their influence on the Q-function is the same for all actions.”
    1. They argue including these types of state variables help make the problem more Markovian, even though it is still highly partially observable by nature.
    2. There were 5 of these types of variables in this study for example:
      1. Binary variable related to a particular side effect
      2. Whether the patient was recently hospitalized in the past 3 months
      3. Where was the patient treated if so (private, state, VA, etc)
      4. Length of time in study prior to current stage
      5. Previous treatment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: