A novel performance aware real-time data handling for big data platforms on Lambda architecture

摘要

Big data is becoming a popular technology for analytics. But, its techniques and tools are very limited to solve the energy aware real time data handling problems. The real time data handling can be in one of the two computing areas: 1) batch computing; 2) stream computing. Stream computing environment uses round robin algorithm as default scheduling strategy whereas batch process uses distributed scheduling for allocation of its resources. But these computing are not considered proper energy aware distributed scheduling policies for allocation of its resources. This paper presents development of management policies that reduces the energy for the allocation of resources. The big data fusion has been used to improve the efficiency for handing different data types: Batch data, online data, and real-time data. A hybrid computational model has been applied to improve the performance further through Lambda architecture. Finally, experimental results have shown 20% performance improvement.