一种基于Spark的分布式时态索引方法

摘要

Based on the Spark distributed computing platform, a distributed temporal indexing method is proposed. Firstly, this paper proposes a segment-indexing strategy of temporal data set, and designs Spark-based tense index construction method and Spark RDD-based parallel query strategy for each segment.According to the Spark RDD partition pattern involved in the temporal query,it is divided into domain independent query,cross-domain query,and cross-segment query,and proposes the optimized auxiliary index efficiency.Finally,the experiments on benchmark data verifies the practicability and efficiency of the proposed indexing strategy, indicating the method is effective on the scalability of the data scale and reduces the cluster configuration requirements.%基于Spark分布式计算平台提出一种分布式时态索引方法.该方法提出时态数据集的分段索引构造策略,对每一分段设计基于Spark的时态索引构建方法及基于Spark RDD的并行查询策略;根据时态查询所涉及的Spark RDD分区模式的不同,将其分为分区独立查询,跨区查询以及跨段查询,并分别针对不同模式的时态查询提出优化的辅助索引结构,提高查询效率;在基准数据上进行实验,验证了所提索引策略的实用性和高效性,同时表明所提方法对数据规模的有效自扩展性以及降低了集群硬件配置需求.