Jorge Nocedal

Jorge Nocedal is Professor of Industrial Engineering and Management Sciences at Northwestern University. He studied at UNAM (Mexico) and Rice University. His research is in optimization, both deterministic and stochastic, with emphasis on very large-scale problems. He is a SIAM Fellow, was awarded the 2012 George B. Dantzig Prize and the 2017 Von Neumann Theory Prize, for contributions to theory and algorithms of nonlinear optimization. He is a member of the US National Academy of Engineering.

Title of talk: Some Open Questions in Deep Neural Network Optimization



In 1961, Minsky perceived a fundamental flaw within the burgeoning field of artificial neural networks. He doubted that such a nonlinear system could be effectively trained using gradient methods, because unless the “structure of the search space is special, the optimization may do more harm than good.” Fast forward to today, and we observe deep neural networks — far more complex than those envisioned at the field's inception — being successfully trained with methods akin to gradient descent. It has, indeed, become evident that the objective function displays a highly benign structure that we are only starting to comprehend. In this lecture, I aim to summarize our current understanding of this enigmatic optimization process. I will explore a diverse array of themes, including intrinsic dimensionality, the optimization landscape, and implicit regularization, and I will highlight key open questions, all within the context of residual networks and generative models.