An analysis of weight initialization methods in connection with different activation functions for feedforward neural networks

Abstract

The selection of weight initialization in an artificial neural network is one of the key aspects and affects the learning speed, convergence rate and correctness of classification by an artificial neural network. In this paper, we investigate the effects of weight initialization in an artificial neural network. Nguyen-Widrow weight initialization, random initialization, and Xavier initialization method are paired with five different activation functions. This paper deals with a feedforward neural network, consisting of an input layer, a hidden layer, and an output layer. The paired combination of weight initialization methods with activation functions are examined and tested and compared based on their best achieved loss rate in training. This work aims to better understand how weight initialization methods in neural networks, in combination with activation functions, affect the learning speed in comparison after a fixed number of training epochs.