Adding external information from unlabeled data to Named Entity Recognition task is the direction of research for solving the scarcity of labeled data. In this issue, adding pre-trained context embeddings as external information from language model is the state-of-the-art technology. However, it has combined external information to the model with simple concatenation, which assumed the same contribution of both internal and external information. While with the use of pre-trained language models, model parameters were not be updated during the iterative process of named entity recognition model. The language model cannot fully capture the semantic and syntactic roles of the contextual words in the labeled data. In view of that case, this paper proposed an improved approach by using pre-trained context embeddings based on attention mechanism. The attention mechanism layer can dynamically balance the difference between the internal information learned from bidirectional Long Short-term Memory model and external information from neural language model. Experimental results have shown that our model has achieved substantial improvement over previous one in Mongolian Named Entity Recognition task.
展开▼
