Computer Vision Lab Seminar: Jeanette Bohg, Stanford, "Multimodal Data in Robot Manipulation"

Thursday, April 18, 2019
2:00 p.m.
3137 Iribe Building
Janice Perrone
301 405 1736
janice@umiacs.umd.edu

Vision, Touch and Motion: On the Value of Multimodal Data in Robot Manipulation and How to Leverage it

Jeannette Bohg
Assistant Professor, Robotics
Computer Science Department
Stanford University
https://web.stanford.edu/~bohg/

Abstract
Recent approaches in robotics follow the insight that perception is facilitated by physical interaction with the environment. First, interaction creates a rich sensory signal that would otherwise not be present. And second, knowledge of the regularity in the combined space of sensory data and action parameters facilitate the prediction and interpretation of the signal. In this talk, I will focus on what this rich sensory signal may consist of and how it can be leveraged for better perception and manipulation. I will start with our recent work that exploits RGB, Depth and Motion to perform instance segmentation of an unknown number of simultaneously moving objects. The underlying model estimates dense, per-pixel scene flow that is then followed by clustering in motion trajectory space. We show how this outperforms state-of-the-art in scene flow estimation and multi-object segmentation.

Furthermore, I will present our recent work on fusing vision and touch for contact-rich manipulation tasks. It is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of visual and haptic sensory inputs, which can then be used to improve the sample efficiency of policy learning.  I present experiments on a peg insertion task where the learned policy generalises over different geometry, configurations, and clearances, while being robust to external perturbations.

Audience: Graduate  Undergraduate  Faculty 

remind we with google calendar

 

March 2024

SU MO TU WE TH FR SA
25 26 27 28 29 1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31 1 2 3 4 5 6
Submit an Event