Temporal Segmentation and Activity Classification from First-person Sensing, Spriggs, De La Torre, Hebert. CVPR 2009.

1. “We explore first-person sensing through a wearable camera and Intertial Measurement Units (IMUs) for temporally segmenting human motion into actions and performing activity classification in the context of cooking and recipe preparation in a natural environment.”
2. Try:
   1. Gaussian Mixture Models
   2. HMMs
   3. K-NN
   4. Also try unsupervised methods so that annotation isn’t necessary
3. Large number of references to prior work
4. The IMUs used here are set up as:
   1. 5 on the wrist
   2. <5?> on ankles
   3. 1 on waist
5. And then there is head-cam
6. There is ambiguity in how to label what is going, especially because people did the same task differently.
   1. Also huge partial observability (even in terms of what is in view of the camera)
7.  Action clauses are distinct per recipe (at least partially), 29 clauses for brownies
8. For unsupervised segmentation, they try PCA, then cluster that
   1. Performance is about 70% correct with this method
   2. These features can also be used to try and classify what recipe is being made
9. Recipe classification is perfect on the small dataset when using data from IMUs
10. Tried unsupervised clustering of IMU data with HMM but it came up with garbage
11. They then merge video and IMU data <but its not totally clear how they accomplish this in terms of representation> again, run through PCA and HMM
12. Recipe classification with this method was ~93%, so just using IMUs is better
13. Then they move onto supervised, with annotation
14. ~80% of frames annotated, “stirring the mix” takes up about 25% of labeled frames
    1. They trained only on frames where annotation was available
15. Then do classification with supervised HMM (poor classification~ 10%) and k-NN (much better at ~60%)
    1. They argue perf of k-NN is from high D data <not sure why that is a good argument though>