基于异构Hadoop集群的负载均衡策略研究

摘要

In the heterogeneous Hadoop environment,processing capabilities of the nodes are diverse and various,among which each node may be continuously added or removed in the clustering and its load slope may increase obviously with tasks.Apparently,load balancing is one of the most important factors that affect the performance of Hadoop clustering.Thus a new load balancing algorithm has been proposed and employed in MapReduce task scheduling of the heterogeneous environment,which makes full use of the node performance and current computation resources according to the cluster load balancing measurement value to allocate task to suitable node for balancing the cluster load gradually and promoting coefficient of utilization of cluster nodes.Since the nodes in the Hadoop clustering are connected with network to save the costs of network transmission and the data locality should be considered with priority to decrease execution time for each task according to the characteristics of data distribution during load balancing scheduling.Simulation results show that the proposed load balancing algorithm has improved performances of whole system significantly and shorten the execution time of the MapReduce task.%异构Hadoop环境中,每个节点的处理能力各不相同,且集群中的节点会不断增加和删除,随着作业量的增大,负载倾斜会越来越明显.显然,负载均衡也成为影响Hadoop集群性能的重要因素之一.针对异构Hadoop环境中MapReduce任务调度,提出了一种新的负载均衡算法.该算法充分利用节点性能和当前的计算资源,根据集群负载平衡度量值进行任务分配,将任务分配给适合的节点,使集群负载逐渐趋于平衡,以提高集群节点利用率.由于Hadoop集群中各节点通过网络连接,以节省网络传输代价,因此在负载均衡调度时,根据数据分布特点,优先考虑数据的本地性,以缩短任务执行时间.仿真实验结果表明,所提出的负载均衡算法能明显改善系统性能,有效缩短MapReduce作业执行时间.