URL-Based Relevance-Ranking Approach to Facilitate Domain-Specific Crawling and Searching

摘要

The WWW is a vast repository of all the types of information known to mankind and thus is capable of serving the frequent varying needs of its users. Classifying and organizing the webpages according to their domain or topic will help the search engine in retrieving and returning a set of fairly relevant pages to the users. This classification is generally done on the basis of their underlying text or content. This paper brings in a novel approach that tries to predict the relevance of a webpage in a domain not by downloading its content but based on the web documents it is linked to. The approach offers advantages of efficiency in cost and performance as the most easily and the least expensive information available about a webpage is its uniform resource locator (URL) [1]. Since the URLs serve as the unique identifier, they are assumed to be an important source for the content of a web page, and therefore, the proposed approach associates the domain information with the web pages based on their URLs.