Implementation and validation of a master-slave distributed crawler applied in Internet Information Retrieval

摘要

This study describes a preliminary research on Information Retrieval (IR) systems. We developed a prototype tool that runs on a master-slave model and uses distributed processing in order to decentralize the workload while retrieving information from the Internet. Later, its viability is demonstrated with a set of executions and the discussion follows with an analysis of the master process overload versus the number of slaves connected.