A new data-grouping-aware dynamic data placement method that take into account jobs execute frequency for Hadoop

摘要

Recent years have seen an increasing number of scientists employing data parallel computing frameworks, such as Hadoop, in order to run data-intensive applications. Research on data-grouping-aware data placement for Hadoop has become increasingly popular. However, we observe that many data-grouping aware data placement schemes are static, without taking MapReduce job execution frequency into consideration. Such data placements scheme will lead to severe performance degradation that is way below the potential efficiency of optimal data distribution when executing MapReduce jobs that are executed frequency. In this paper, we propose a new data-grouping-aware dynamic (DGAD) data placement method based on the job execution frequency. Firstly, we build a job access correlation relation model among the data blocks according to the relationships provided by the records about historical data block access. Then we use a clustering algorithm to divide data blocks into clusters according to the job access correlation relation model among the data blocks and propose a data placement algorithm based on data block clusters in order to put correlated data blocks within a cluster on the different nodes. Finally, a series of experiments are carried out in order to verify the method proposed in this paper. Experimental results show that the proposed method can effectively deal with the mass data and can obviously improve the execution efficiency of MapReduce. (C) 2016 Elsevier B.V. All rights reserved.