One of the challenges in dealing with distributed large data is to transfer massive amounts of data from multiple data server(s) to users. Unless data transfers are planned, organized and regulated carefully, they can become a potential bottleneck and may necessitate changes in queries and database design which involves costly maintenance work. This is a pronounced problem in the case of virtual observatories where data is to be brought from multiple astronomical databases from all around the world. In this paper, we present adaptive middle ware caching using sub-query fragmentation. When groups of users working on related projects query multiple databases, often their queries are overlapped only partially. We develop a cooperative cache framework with dynamic maintenance algorithms to capture user query patterns in the workload that adapts itself to provide as much data available from cache units as possible. Initial results in the simulated environment with known query inputs show significant reduction in the data to be transferred in comparison with full query caching.
展开▼
