Efficient Second-order Optimization for Machine Learning

  • Naman Agarwal | Princeton University

Stochastic gradient-based methods are the state-of-the-art in large-scale machine learning optimization due to their extremely efficient per-iteration computational cost. Second-order methods, that use the second derivative of the optimization objective, are known to enable faster convergence. However, the latter has been much less explored due to the high cost of computing the second-order information. We will present second-order stochastic methods for (convex and non-convex) optimization problems arising in machine learning that match the per-iteration cost of gradient-based methods, yet enjoy the faster convergence properties of second-order optimization overall leading to faster algorithms than the best known gradient-based methods.

Watch Next