Temporal Segmentation and Activity Classification from First-person Sensing, Spriggs, De La Torre, Hebert. CVPR 2009.


  1. “We explore first-person sensing through a wearable camera and Intertial Measurement Units (IMUs) for temporally segmenting human motion into actions and performing activity classification in the context of cooking and recipe preparation in a natural environment.”
  2. Try:
    1. Gaussian Mixture Models
    2. HMMs
    3. K-NN
    4. Also try unsupervised methods so that annotation isn’t necessary
  3. Large number of references to prior work
  4. The IMUs used here are set up as:
    1. 5 on the wrist
    2. <5?> on ankles
    3. 1 on waist
  5. And then there is head-cam
  6. There is ambiguity in how to label what is going, especially because people did the same task differently.
    1. Also huge partial observability (even in terms of what is in view of the camera)
  7.  Action clauses are distinct per recipe (at least partially), 29 clauses for brownies
  8. For unsupervised segmentation, they try PCA, then cluster that
    1. Performance is about 70% correct with this method
    2. These features can also be used to try and classify what recipe is being made
  9. Recipe classification is perfect on the small dataset when using data from IMUs
  10. Tried unsupervised clustering of IMU data with HMM but it came up with garbage
  11. They then merge video and IMU data <but its not totally clear how they accomplish this in terms of representation> again, run through PCA and HMM
  12. Recipe classification with this method was ~93%, so just using IMUs is better
  13. Then they move onto supervised, with annotation
  14. ~80% of frames annotated, “stirring the mix” takes up about 25% of labeled frames
    1. They trained only on frames where annotation was available
  15. Then do classification with supervised HMM (poor classification~ 10%) and k-NN (much better at ~60%)
    1. They argue perf of k-NN is from high D data <not sure why that is a good argument though>
Advertisements
Tagged , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: