Unsupervised Taxonomy of Large Document Corpora Utilizing Idiomatic Character of Natural Languages

摘要

This paper presents a novel approach to unsupervised taxonomy. It is based on the idiomatic character of natural languages, rather then on statistical calculations. The term "idiomatic" is hereby utilized in two complementary senses: as an intersubjective agreement among the members of a speech community regarding the meaning of a phrase, or as an objective agreement that a particular message is customarily expressed by some particular phrase. The idiomatic character of natural languages makes it extremely likely that across entire document corpora similar ideas will be consistently expressed by some particular phrases. This allows main ideas of a corpus to be faithfully represented by a handful of idiomatic phrases, which can serve as a directory that significantly improves the navigation through the underlying corpus.