Engineering and Learning Visual Representations: Invariance, Sufficiency, and the role of Control

Visual Representations are functions of visual data tailored for decision and control tasks, where much of the variability is due to nuisance factors such as viewpoint, illumination, partial occlusion. An optimal representation would be maximally insensitive to those, and maximally "informative" for the task.

I will formalize an expression for the maximal-invariant/minimal-sufficient representation and show that, under very restrictive conditions, it is related to methods currently employed in Computer Vision, such as SIFT, except for a small but important modification. When such a modification is applied, performance on benchmark image matching datasets improves by over 30% above SIFT, surpassing even convolutional neural networks trained on millions of images.

More importantly, such restrictive conditions can be lifted. I will present multi-view representations and discuss their properties in relation to their "informative content" which is maximal when the statistic is complete. I will argue that completeness is achievable by controlling the data acquisition process, in an active sensing/experiment design setting. For visual data, this entails the ability to move in physical space. Thus, it is not just that vision is useful for mobility; conversely, mobility is needed to construct optimal representations.