Universality of Gradient Descent Neural Network Training

[Submitted connected 27 Jul 2020]

View PDF HTML (experimental)

Abstract:It has been observed that creation choices of neural networks are often important for their successful optimization. In this article, we truthful talk the mobility if it is ever imaginable to redesign a neural web truthful that it trains good pinch gradient descent. This yields the pursuing universality result: If, for a fixed network, location is immoderate algorithm that tin find bully web weights for a classification task, past location exists an hold of this web that reproduces these weights and the corresponding guardant output by specified gradient descent training. The building is not intended for applicable computations, but it provides immoderate predisposition connected the possibilities of meta-learning and related approaches.