METHOD AND APPARATUS FOR NEURAL NETWORK-BASED WORD SEGMENTATION AND PART-OF-SPEECH TAGGING, DEVICE AND STORAGE MEDIUM

摘要

A method and apparatus for neural network-based word segmentation and part-of-speech tagging, a computer device, and a storage medium, which relate to the technical field of artificial intelligence. The method comprises: acquiring a corpus to undergo word segmentation and inputting same into a pre-trained first DNN neural network model, and acquiring a plurality of initially segmented words outputted by the first DNN neural network model in response to the corpus to undergo word segmentation (201, 202); and calculating an internal aggregation degree and information entropy of each initially segmented word, and determining an intially segmented word of which both the internal aggregation degree and information entropy exceed set thresholds to be a final segmented word (203). The final segmented word is inputted into a pre-trained second DNN neural network model and KNN model for use in analyzing the candidate word part-of-speech and candidate word part-of-speech probabilities as well as the similar word part-of-speech and similar word part-of-speech probabilities of the final segmented word (204, 205), and the part of speech that has the highest probability is returned as the part of speech of the final segmented word (206). The described method completes part-of-speech tagging at the same time as word segmentation, further improves the accuracy of word segmentation, and provides word segmentation results for different scenarios that best fit the scenarios.