A Novel Statistic Based Chinese Word Segmentation Approach

摘要

Chinese word segmentation is by no means trivial process. In this paper, a novel segmentation model is presented and a two ways searching algorithm is also given. Unlike the raditional model, the model proposed scores the caniddate word seuences with the word formation power of Chinese characters string(word form model, WFM) and the affinity of character junctures(CJM), which results in improvements in segmentation performance especially disambiguation capability. Moreover, the model has a strong adaptability since all the parameters can be learned directly from unsegmented train texts. The approach has proved efficient through our primary experiments.