Train Simultaneously, Generalize Better: Stability of Gradient-Based Minimax Learners

Wednesday, December 2, 2020 - 4:00pm to 4:30pm

Event Calendar Category

LIDS & Stats Tea

Speaker Name

Farzan Farnia

Affiliation

EECS

Zoom meeting id

934 7019 0285

Join Zoom meeting

https://mit.zoom.us/j/93470190285

Abstract

The success of minimax learning problems of generative adversarial networks (GANs) and adversarial training has been observed to depend on the minimax optimization algorithm used for their training. This dependence is commonly attributed to the convergence speed and robustness properties of the underlying optimization algorithm. In this talk, we present theoretical and numerical results indicating that the optimization algorithm also plays a key role in the generalization performance of the trained minimax model. To this end, we analyze the generalization properties of standard gradient-based minimax optimization algorithms through the lens of algorithmic stability under both traditional convex-concave and general nonconvex-nonconcave minimax settings. We also compare the generalization performance of standard gradient descent ascent (GDA) and GDmax algorithms where the latter fully solves the maximization subproblem at every iteration. Our generalization analysis suggests the superiority of GDA provided that the minimization and maximization subproblems are solved simultaneously with similar learning rates. We discuss several numerical results demonstrating the role of optimization algorithms in the generalization of trained minimax models.  

Biography

Farzan Farnia is a postdoctoral research associate at the Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, where he is co-supervised by Asu Ozdaglar and Ali Jadbabaie. Prior to joining MIT, Farzan received his master’s and PhD degrees in electrical engineering from Stanford University and his bachelor’s degrees in electrical engineering and mathematics from Sharif University of Technology. At Stanford, he was a graduate research assistant at the Information Systems Laboratory advised by David Tse. Farzan's research interests include information theory, statistical learning theory, and convex optimization. He has been the recipient of the Stanford graduate fellowship (Sequoia Capital fellowship) from 2013-2016 and the Numerical Technology Founders Prize as the second top performer of Stanford electrical engineering PhD qualifying exams in 2014.