When Information Retrieval Measures Agree About the Relative Quality of Document Rankings

摘要

The variety of performance measures available for infor- mation retrieval systems, search engines, and network filtering agents can be confusing to both practitioners and scholars. Most discussions abut these measures address their theoretical foundations and the character- istics of a measure that make it desirable for a particular application. In this work, we consider how measures of performance at a point in a search may be formally compared. Criteria are developed that allow one to de- termine the percent of time or conditions under which two different performance measures suggest that one document ordering is superior to another ordering, or when the two measures disagree abut the relative value of document orderings. As an example, graphs provide illustrations of the relationships between preci- sion and F.