IoT Thrust Seminar | Accelerated Stochastic Momentum with Application to Machine Learning  

Machine learning training is a high dimension stochastic optimization problem. Most of the training algorithms are based on stochastic gradient, which is known to be slow. There are many research works trying to accelerate gradient based algorithms using momentum and good success has been demonstrated in deterministic problems. However, the accelerated performance of momentum-based algorithms cannot be extended to the case when the gradient has noise. In this work, we explore the fundamental reason for the failure of stochastic momentum algorithms and propose a new framework for the step size design for stochastic momentum-based algorithms using stochastic differential equations (SDE). By designing an appropriate Lyapunov function, we obtain closed form step size rules for stochastic momentum algorithms via Lyapunov drift minimization. The proposed stochastic momentum algorithm has 5X-10X convergence speedup with analytical convergence guarantee.