首页>
外国专利>
Asymmetric Functionality Activation for Improved Stability in Neural Networks
Asymmetric Functionality Activation for Improved Stability in Neural Networks
展开▼
机译:神经网络中提高稳定性的不对称功能激活
展开▼
页面导航
摘要
著录项
相似文献
摘要
Thus, aspects of the present disclosure address model “blow up” by changing the functionality of the activation, thereby providing “dead” or “dying” neurons with the ability to recover from this situation. As one example, for activation functions that have an input region in which the neuron is turned off by a 0 or close to 0 gradient, a training computing system can keep the neuron turned off when the gradient pushes the unit farther into the region (e.g., by applying an update with zero or reduced magnitude). However, if the gradient for the current training example (or batch) attempts to push the unit towards a region in which the neuron is active again, the system can allow for a non-zero gradient (e.g., by applying an update with standard or increased magnitude).
展开▼