Evolutionary and Constructive Approaches to Supervised Learning in Hybrid Neural Networks

摘要

Single hidden layer neural networks with supervised learning have been successfully applied to approximate unknown mappings defined in compact functional spaces. The most advanced results also give rates of convergence, stipulating how many hidden neurons, with a given activation function, should be used to achieve a specific order of approximation. However, independently of the activation function employed, these connectionist models for function approximation suffer from a severe limitation: all hidden neurons use the same activation function. If each activation function of a hidden neuron is properly and automatically defined, then better rates of convergence will be achieved. This is exactly the purpose of constructive learning using projection pursuit techniques, where each hidden neuron presents a particular activation function. In spite of its greater flexibility, the resulting transfer function associated with this constructive learning process still presents a strong restriction: additive composition is the only way to combine the possibly distinct activation functions. So, we propose here a hybrid composition for the combination of the activation functions, that accepts also the multiplicative composition as a candidate, besides the additive one. Two alternative implementations toward optimal design are proposed: one based on evolutionary computation and nonlinear optimization techniques, and another based on an extended version of projection pursuit learning. Although demanding a higher computational cost to obtain the solution, the amount of computational resources required by the resulting neural network architecture is impressive inferior, when compared to the solution produced by traditional supervised learning approaches.