Social-based optimization of web crawling for online social networks

摘要

In one embodiment, a method includes a search engine of an online social network crawling a first webpage of a first web domain, where the first webpage includes links to one or more second webpages, each of which may be within a second web domain, accessing a domain ranking for each second web domain, where for each second web domain the domain ranking may be based on one or more domain-quality signals associated with the second web domain, where the domain-quality signals may include a measure of activations of social plug-ins of the online social network associated with webpages of the second web domain, selecting one or more of the second webpages to crawl based at least in part on the domain ranking of the second web domain associated with the second webpage, and the search engine of the online social network crawling each selected second webpage.