On The Performance Of The Maximum Likelihood Over Large Models

This dissertation investigates non-parametric regression over large function classes, specifically, non-Donsker classes. We will present the concept of non-Donsker classes and study the statistical performance of the Least Squares Estimator (LSE) — which also serves as the Maximum Likelihood Estimator (MLE) under Gaussian noise — over such classes.

(1) We demonstrate the minimax sub-optimality of the LSE in the non-Donsker regime, extending traditional findings of Birgé and Massart '93 and resolving a longstanding conjecture of Gardner, Markus, and Milanfar '06.

(2) We reveal that in the non-Donsker regime, the sub-optimality of LSE arises solely from its elevated bias error term (in terms of the bias-variance decomposition).

(3) We introduce the first minimax optimal algorithm for multivariate convex regression with a polynomial runtime in the number of samples – showing that one can overcome the sub-optimality of the LSE efficiently.

(4) We study the minimal error of the LSE both in random and fixed design settings.