A Granularity-Aware Parallel Aggregation Method for Data Streams

摘要

This paper focuses on the parallel aggregation processing of data streams based on the shared-nothing architecture. A novel granularity-aware parallel aggregating model is proposed. It employs parallel sampling and linear regression to describe the characteristics of the data quantity in the query window in order to determine the partition granularity of tuples, and utilizes equal depth histogram to implement partitioning. This method can avoid data skew and reduce communication cost. The experimentresults on both synthetic data and actual data prove that the proposed method is efficient, practical and suitable for time varying data streams processing.