Jacobians of neural networks and Free Probability

Abstract

Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing robustness. Following the works of Pennington et al., such Jacobians are modelled using free multiplicative convolutions from Free Probability Theory. In a substantial first part of the talk, I will present the problem from Machine Learning and introduce the necessary tools from Free Probability. It will then be clear that a good computational method is missing. In the second part of the talk, we present our solution: a reliable and very fast method for computing the spectral densities associated to a neural network. Beyond machine learning, the method is certainly of independent interest in Free Probability Theory and high dimensional statistics. This is joint work with Tariq DAOUDA and Ezéchiel KAHN.