New neural network models offer the potential for better predictions in medicine, finance, and human genetics

January 30, 2020

Vector Faculty Member David Duvenaud and collaborators have published a new paper, “Scalable Gradients for Stochastic Differential Equations,” in Artificial Intelligence and Statistics. The paper uses backpropagation to fit stochastic continuous-time models and offers the potential for building more complex prediction models in fields like physics, finance, and human genetics. 

Stochastic differential equations (SDEs) — differential equations that account for uncertainty due to unseen interactions — have a long history in fields like finance where they help forecast how stock prices might evolve over time. However, they’ve been limited in the number of parameters (generally 10 or 20) that can be fit at any one time and, generally speaking, were not scalable to the large neural networks with millions of parameters that are used to fit data in other domains.

“Scalable Gradients for Stochastic Differential Equations” specifies the dynamics of these models with neural networks, and trains them with gradient-based optimization.

Duvenaud and his co-authors Ricky T. Chen, Ting-Kam Leonard Wong, and Xuechen Li combine these processes with deep neural networks.  Duvenaud had previously worked on the idea of switching from discrete time — data sampled at regular intervals — to continuous time — data sampled at any point in the flow, in the paper “Neural Ordinary Differential Equations,” which won a Best Paper Award at NeurIPS 2018. In this new paper, they generalized the math allowing for neural ODEs to be trained for SDEs, a much larger family of models. 

Continuous time backpropagation already existed for neural ODEs, but no such reverse mode method existed for SDEs. The algorithm ended up being a straightforward extension of the ODE method with fixed noise, a sort of continuous-time reparameterization trick.

These continuous time models offer a more fine-grained and flexible way to incorporate or sample data time series data.  They have the potential to help better model medical data, predict prices across the stock market, or track the evolution of populations over long periods of time.


Vector logo blog image

Vector Institute partners with World Economic Forum on responsible artificial intelligence and research insights

Vector logo blog image

Vector Institute 2022-23 annual report: accelerating AI in Ontario

How SMBs can manage the opportunities and risks of deploying AI