A partitioning technique for improving the performance of PageRank on Hadoop

摘要

There are a lot of research results in large scale graph analysis on Hadoop. The performance of the graph analysis based on Hadoop is impacted by data partitioning. The effectiveness of data partitioning depends on how the data partitioning maintains data locality in each node of cluster, and this would be different from the problems faced with. One way of data partitioning known to be effective is partitioning data by domains. For instance, this technique could be very useful in partitioning data by areas analyzing web graphs. But this kind of improvement from the data partitioning is limited to specific problems. In this paper, we propose a data partitioning technique based on semi-clustering for analyzing web graphs with PageRank algorithm on Hadoop. With experiment, PageRank computation with our partitioning technique improves the performance, as the number of iterations increases. This method can be very effective in the case of large scale graph processing.