An architecture for cooperative distributed Internet resource discovery systems.

摘要

This dissertation presents an architecture of a distributed cooperative Internet resource discovery system which organizes a set of autonomous and heterogeneous index servers on the Internet into a large virtual index server. The system is built in a horizontal manner based on peer-to-peer communication between neighboring servers rather than in a more traditional hierarchical manner. The entire system consists of self-contained servers each of which is capable of accepting and processing user queries, as well as acting as a broker to a group of other servers. As a broker, a server ranks and selects the best servers which potentially carry information relevant to a given query among the members of its server group, forwards the query to the selected servers, and merges the search results. The servers form server groups with overlapping memberships among the groups, which ultimately results in a web-like network topology. By extending the brokerage mechanism and taking advantage of the web-like topology, a global query routing is obtained. This dissertation proposes a set of techniques employed by the system at various levels of interconnectivity, starting with the basic Internet resource indexing and searching at the index server level, collection fusion methods at the broker server level, to the system-wide query routing and the optimization of the routing's dynamics at the global level. All of these techniques are based solely on simple statistical data about the distribution of keywords across the index databases. We demonstrate that using the word distribution statistics alone, an efficient and effective Internet-wide distributed multi-database information retrieval can be built. We empirically evaluate each of the proposed techniques based on its contribution to the system's effectiveness in locating resources relevant to user queries.