Optimal Selection of Different Order N-Grams

摘要

We subjectively introduce different assumptions about the mutual information among words for separate usage of different N-grams regardless of the fact that some assumptions cannot be true. An approach is proposed to evaluate different order N-grams based on the concept of confidence of assumption or assumption probability. The assumption probabilities are estimated on the basis of discriminative estimation criterion. To reduce the redundant information among different order N-grams, we bring up with a method to select M-gram elements from different order N-grams with reference to the assumption probabilities. We conduct experiments on the platform of conversion from Chinese pinyin to Chinese character. At first, we emplov Newton Gradient method to estimate the assumption probabilities and then lest the optimally selected language model. The experimental results show that the memory capacity of language model can be remarkably lowered with little loss of accuracy.