Improving the Accuracy of Deep Neural Networks Through Developing New Activation Functions

摘要

Without activation functions, it would be only possible for the neural network to learn very basic tasks, so the activation function is a key point in the neural network’s architecture. The function allows us to learn more complicated tasks and also it impacts the performance to obtain the outcome. So, activation functions represent the continuous and widespread interest of research to identify the most suitable activation function to a specific task. In this paper, we propose four activation functions that bring improvements for different datasets in the Computer Vision task. These functions are a combination of the popular activation functions such as sigmoid, bipolar sigmoid, Rectified Linear Unit (ReLU), and tangent (tanh). By allowing activation functions to be learnable we obtain models more robust. To validate these functions, we tested using more datasets and more architectures with different depths, showing that their properties are significant and useful. Also, we compared them with other powerful activation functions to see how our proposed activation functions impact accuracy.