Benign overfitting in adaptive non-parametric regression

Abstract

Benign overfitting is an intriguing phenomenon recently discovered in the deep-learning community. In specific cases, deep neural networks can be experimentally trained to perfectly overfit a noisy training dataset, while having excellent generalization performances to predict new data points. This talk aims to explore benign overfitting in the simplified setting of nonparametric regression. Using local polynomials, we construct an estimator of the regression function with two main properties. Firstly, the proposed estimator is minimax-optimal over Hölder classes. Secondly, it is a continuous function interpolating the set of observations with high probability. Furthermore, we propose an additional overfitting estimator that attains optimality adaptively to the unknown Hölder smoothness. Our results are non-asymptotic and highlight that in the nonparametric regression model, interpolation can be fundamentally decoupled from the bias-variance tradeoff.