海量自动气象站分钟数据存储检索系统设计

摘要

地面自动气象站的时空密度不断增加,产生的观测数据量呈指数级增长,传统的关系型数据库在海量数据存储与检索方面存在能力不足、检索性能下降等问题.鉴于此,本文设计一种自动站分钟数据存储与检索系统.使用Quartz定时采集自动站分钟文件并解码入库;应用HBase分布式数据库建立分钟数据存储模型;针对多要素查询需求,应用Elasticsearch建立辅助索引,实现HBase的二级索引.系统测试结果表明,分钟数据入库平均耗时54.6 s,二级索引完整可靠,数据检索结果返回时效达到毫秒级,能够满足业务应用中对自动站分钟数据存储和检索时效的要求.%With the rapid growth of temporal and spatial density of ground automatic weather station,the quantity of observed data increases exponentially,traditional relational database has insufficient ability and low performance in mass data storage and retrieval.In view of this,an automatic station minute data storage and retrieval system is designed.Quartz is used to collect automatic station minute files and then the system decodes the minute files and storage decoded data into database.The system uses HBase to build minute data storage model.In order to meet the query demands of multiple elements,the system applies Elasticsearch to build secondary index of HBase.Test results show that the average storage time for minute data is 54.6 s and the seeondary index with Elasticsearch is complete and reliable and the retrieval results returned time to milliseconds.Therefore,the system can satisfy the demand for storage and query performance of automatic weather station minute data.