Looking at a Few Images of Rooms and Many Interacting Hands

The long-term goal of my research is to let computers understand the physical world from images, including both 3D properties and how humans or robots could interact with things. This talk will cover two recent directions aimed at enabling this goal.

I will begin by talking about 3D reconstruction from two ordinary images where the camera pose is unknown and the views have little overlap -- think hotel listings. Computers struggle in this setting since standard techniques usually depend on many images, high overlap, known camera poses, or RGBD input. Nonetheless, humans seem to build a sense of a space from a few photos. We think the key to this ability is joint reasoning over reconstruction, camera pose, and correspondence. This insight is put into action with a deep learning architecture and optimization that produces a coherent planar reconstruction. Our system outperforms many baselines on Matterport3D, but there is plenty of room for new work in this exciting setting.

Then, I will focus on understanding what humans are doing with their hands. Hands are a primary means for humans to manipulate the world, but fairly basic information about what they're doing is often off-limits to computers (at least in unconstrained data). I'll describe some of our efforts on understanding hand state, including work on learning to segment hands and hand-held objects in images via a system that learns from large-scale video data.