Using Statistical Properties to Enhance Text Categorization

摘要

Statistical properties extracted from text are useful in many areas. Knowing who authored some text or knowing the category of a text is among the uses of collecting such statistics. In this paper, language-independent properties of text are studied using two categorized corpora of news articles. It is observed that the properties do not depend on the corpus or on its size. Several interesting properties are identified which enable minimizing the training set for an intelligent categorization system. Some other applications for such statistics could be to compare the information content and rate of new information between different corpora as well as to enhance the categorization of text.