HDFS Write Operation Using Fully Connected Digraph DataNode Network Topology

摘要

Hadoop is an open source implementation of the MapReduce Framework in the realm of distributed processing. A Hadoop cluster is having the capability to handle large amounts of data. To handle massive scale data, Hadoop exploits the Hadoop Distributed File System termed as HDFS. Client will write data to DataNodes by taking the blocks info from NameNode. The DataNodes containing the blocks will be connected in pipeline fashion. While writing the data if DataNode/network fails the failed DataNode will be removed from the pipeline. Based on the available DataNodes in the cluster the new DataNode will be included in the pipeline. If there are very less number of spare nodes in the cluster users may experience an unusually high rate of pipeline failures since it is impossible to find new DataNodes for replacement. If network failure happend, the data packet cannot be reached to the target DataNode since they are connected in pipeline fashion. If each DataNode is connected to each other DataNode then there will not be any issue with network failure since they have number of paths through other DataNodes. In pipeline connectivity the copy operation will take longertime, where as in DataNode is having direct connection to all other DataNodes, it will take very less time because datapacket is not required to traverse through all other DataNodes to reach the end DataNode. In this paper we will address the network failure issues among the DataNodes and reducing the copy operation time to copy data packet to DataNodes which are in pipeline by fully connected digraph network topology. Using this topology network complexity will be high, but we can reduce the time to copy one data packet to all replica locations (DataNodes) and nullify the network failure issues among the DataNodes.