Learning Single-Image 3D from the Internet

Abstract: Single-image 3D refers to the task of recovering 3D properties such as depth and surface normals from an RGB image. Although significant progress has been made in this field, the current best systems still struggle to perform well on arbitrary images “in the wild”, i.e. images that depict all kinds of contents and scenes. One major obstacle is a lack of diverse training data, as existing datasets are limited to indoor or driving scenes (e.g. NYU Depth, KITTI). Systems trained on such data have been shown to fail in generalizing in the wild.

In this dissertation, I will present my research on acquiring high-quality and diverse 3D data from the Internet. I will discuss my work in collecting the first-ever 3D datasets for images in the wild through manual and automated methods, as well as the new research opportunities opened up by the introduction of these datasets. I will also talk about how to utilize the collected data to significantly advance single-image 3D perception in the wild.