Improving Generalization Performance by Information Minimization

摘要

In the present paper, we attempt to show that the information about input patterns must be as small as possible for improving the generalization performance under the condition that the network can produce targets with appropriate accuracy. The information is defined with respect to the hidden unit activity and we suppose that the hidden unit has a crucial role to store the information content about input patterns. The information is defined by the difference between uncertainty of the hidden unit at the initial stage of the learning and the uncertainty of the hidden unit at the final stage of the learning. After having formulated an update rule for the information minimization, we applied the method to a problem of language acquisition: the inference of the past tense forms of regular and irregular verbs. Experimental results confirmed that by our method, the information was significantly decreased and the generalization performance was greatly improved.