Web toolkit: an agent based scalable search engine using cellular automata based classification for duly ranked retrieved data

摘要

Web page classification is a major issue for categorising web documents to facilitate indexing, search and retrieval of web pages for search engine. Different crawling techniques have been utilised to accumulate web pages of different domains under separate databases depending on practical scenario. Downloaded web pages are being parsed for further processing. A classifier is designed dynamically using single cycle multiple attractor cellular automata for mapping downloaded web pages of different domains into specific structure. This paper proposes alternate technique for automatic categorisation of web pages into different domains. Retrieved web pages have been ranked automatically at the time of classifier formation. Typically, our system consists of crawling, ranking and storage parts created in a different way. Hierarchical concept has been used over parallel crawler. GF(2P) concept is introduced in ranking. The concept of SMACA has been utilised in indexing storage. Overall, a search engine module has been created using agent-based method.