Shark is a research data analysis system built on a novel coarse grained distributed shared-memory abstraction. Shark pairs query processing with deep data analysis, providing a unified system for easy data manipulation using SQL while pushing sophisticated analysis closer to its data. It scales to thousands of nodes in a fault tolerant manner. Shark can answer queries over 40 times faster than Apache Hive and run machine learning programs on large datasets over 25 times faster than equivalent MapReduce programs on Apache Hadoop. Unlike previous systems, Shark shows that it is possible to achieve these speedups while retaining a MapReduce-like execution engine, with the fine- grained fault tolerance properties that such an engine provides. Shark additionally provides several extensions to its engine, including table and column-level statistics collection as well as a cost-based optimizer, both of which we describe in depth in this paper. Cost-based query optimization in some cases improves the performance of queries with multiple joins by orders of magnitude over Hive and over 2 compared to previous versions of Shark. The result is a system that matches the reported speedups of MPP analytic databases against MapReduce while providing more comprehensive fault tolerance and complex analytics capabilities.
展开▼
