Distinguished Lecturer Seminar - DRSOM: A Dimension-Reduced Second-Order Mtehod for Machine and Deep Learning
We introduce a Dimension-Reduced Second-Order Method (DRSOM) for convex and nonconvex (unconstrained) optimization. Under a trust-region-like framework, our method preserves the convergence of the second-order method while using only Hessian-vector products in few directions. Moreover, the computational overhead remains comparable to the first-order such as the gradient descent
method. We show that the method has a local super-linear convergence and a global convergence rate of O(∈-3/2) to satisfy the first-order and second-order conditions under a commonly adopted approximated Hessian assumption. We further show that this assumption can be removed if we perform a step of the
Krylov subspace method periodically. The applicability and performance of DRSOM are exhibited by various computational experiments, particularly in Machine and Deep Learning. For neural networks, our preliminary implementation seems to gain computational advantages in terms of training accuracy and iteration complexity over state-of-the-art first-order methods such as SGD and ADAM. For policy optimization, our experiments show that DRSOM compares favorably with popular policy gradient methods in terms of the effectiveness and robustness.
Yinyu Ye is currently the K.T. Li Professor of Engineering at Department of
Management Science and Engineering and Institute of Computational and
Mathematical Engineering, Stanford University. His current research interests include Continuous and Discrete Optimization, Data Science and Application, Algorithm Design and Analysis, Computational Game/Market Equilibrium, Metric Distance Geometry, Dynamic Resource Allocation, and Stochastic and Robust Decision Making, etc.
He is an INFORMS (The Institute for Operations Research and The Management Science) Fellow since 2012, and has received several academic awards including: the inaugural 2006 Farkas Prize on Optimization, the 2009 IBM Faculty Award, the 2009 John von Neumann Theory Prize for fundamental sustained contributions to theory in Operations Research and the Management Sciences, the inaugural 2012 ISMP Tseng Lectureship Prize for outstanding contribution to continuous optimization (every three years), the winner of the 2014 SIAM Optimization Prize awarded (every three years), the 2015 SPS Signal Processing Magazine Best Paper Award, etc.. According to Google Scholar, his publications have been cited 51,000 times.