Learning from Geometry

Deep neural networks have proved very successful in domains where large training sets are available, since their capacity can be increased by adding layers or by increasing the number of units in a layer. When the number of training samples is small, their performance suffers from overfitting.

In the first part of this talk we focus on a single layer and the design of linear features that are able to discriminate classes of inputs. We use wireless information theory to derive fundamental limits on the maximum number of classes that can be discriminated with low probability of error and on the tradeoff between the number of classes and the probability of misclassification.

In the second part of the talk we focus on overfitting to random error or noise instead of the underlying signal. Prior work includes methods such as weight decay, Dropout and DropConnect that are data independent and designed to make it more difficult to fit to random error or noise. We will describe GraphConnect, a complementary method that is data dependent, and is motivated by the empirical observation that data of interest typically lies close to a manifold.

This is joint work with Jiaji Huang, Matt Nokleby, Qiang Qiu, Miguel Rodrigues and Guillermo Sapiro