A simple asymptotic model for overparameterized learning

This talk builds on the signal-processing-centric "harmless interpolation" story that we shared at LIDS back in Sep 2019, and sets out to flesh out a simple and stylized asymptotic model for overparameterized learning. The eventual goal remains understanding the mystery: why are deep neural networks able to achieve zero training error and yet generalize well, even when the training data is noisy and there are many more parameters than data points? One nice thing about the stylized linear model we study is that the connection to signal-processing thinking gives us heuristic ways to conjecture asymptotic behavior. After recapping the essential story for regression problems, we discuss both binary classification and multi-class classification in an asymptotic regime where the number of classes can grow with the number of data points. It turns out that we are able to verify what our heuristics allowed us to conjecture: the existence of asymptotic regimes where classification can generalize even though the corresponding regression problems would not be able to generalize. Finally, reflecting upon the asymptotic behavior in these regimes allows us to engage in counterexample-style thinking that sheds light on the (dis)connection between test loss and training loss in overparameterized settings.