Optimization of generic progressive queries based on dependency analysis and materialized views

摘要

Progressive queries (PQ) are a new type of queries emerging from numerous contemporary database applications such as e-commerce, social network, business intelligence, and decision support. Such a PQ is formulated on the fly in several steps via a set of inter-related step-queries (SQ). In our previous work, we presented a framework to process a restricted type of PQs. However, how to process generic PQs remains an open problem. In this paper, we develop a novel technique to efficiently process generic PQs based on materialized views. The SQs of an in-process PQ can utilize the results of previous SQs not only from the same PQ but also from other in-process and completed PQs. The key idea is to create a multiple query dependency graph (MQDG), which captures the data source dependency relationships among SQs from multiple PQs. A mathematic model is developed to estimate the benefit of keeping the result of an SQ as a materialized view (critical SQ/ node) based on the MQDG. The kept materialized views are used to improve the performance of the future SQs. Strategies for constructing the MQDG and identifying the critical SQs for materialization by using the MQDG are presented. To manage the storage of the materialized views, we introduce two approaches - one employs a greedy method and the other adopts a dynamic programming (DP) based method. Strategies are also suggested to reduce the input problem size for the DP procedure. Experimental results demonstrate that our technique is quite promising in efficiently processing PQs.