Random matrix theory for high dimensional optimization, and an application to scaling laws

We describe a program of analysis of stochastic gradient methods on high-dimensional random objectives. We illustrate some assumptions under which the loss curves are universal, in that they can completely be described in terms of some underlying covariances. Furthermore, we give description of these loss curves that can be analyzed precisely.

We show how this can be applied to SGD on a power-law-random-features model. This is a simple two-hyperparameter family of optimization problems, which displays 5 distinct phases of loss curves; these phases are determined by the relative complexities of the target, data distribution, and whether these are ‘high-dimensional’ or not (which in context can be precisely defined).  In each phase, we can also give, for a given compute budget, the optimal random-feature dimensionality.

Joint work with Courtney Paquette (McGill & Google Deepmind), Jeffrey Pennington (Google Deepmind), and Lechao Xiao (Google Deepmind).

_________________________________________________________________

About the series:

The Vector Distinguished Talk series is a formal gathering of academic and industrial data scientists across the Greater Toronto Area (GTA) to discuss advanced topics in machine learning and its goal is to build a stronger machine learning community in Toronto.

The talks will be given by international and local faculty and industry professionals. The seminar series is intended for university faculty and graduate students in machine learning across computer science, ECE, statistics, mathematics, linguistics, and medicine, as well as PhD-level data scientists doing interesting applied research in the GTA. The Toronto machine learning community will be stronger when we know each other and know what problems people are working on. At the end of each talk there will also be an opportunity to have in-person meetings with the speaker.

Vector Distinguished Lecture Series is currently being held remotely. We look forward to welcoming everyone in-person at Vector’s new office space soon. Our talks will continue to be streamed online for those not able to come in-person.

Register here!

This event is open to the public with emphasis on graduate students in machine learning, computer science, ECE, statistics, mathematics, linguistics, medicine, as well as PhD-level data scientists in the GTA.