Improved Performance of Hive Using Index-Based Operation on Big Data

摘要

Indexing provides fast searching over big data. It is a data structure technique to efficiently retrieve records from the database files based on some attributes on which the indexing has been done. With the help of indexing, queries usually results in much better performance. Hive is batch-oriented data warehouse software. With the help of hive we can perform query processing and data analysis task. Hive is popular because it supports a bulk of the SQL operations in relational database management systems. To improve performance of database systems join has been the focus of several query optimization techniques. As a result the aim of this work is in two folds: Firstly we implement index based join technique and integrated in Hive and secondly performance is estimated, after perform join operation. we run relevant test queries on datasets generated using the industry standard benchmark, TPC-H. Our results indicate significant performance gain over highly selective queries.