ElasticSearch分布式搜索引擎在导航大数据检索中的应用_孟方园

Elasticsearch distributed search engine application in

Navigation Big Data Retrieval

Fangyuan Meng 1, Tao Wei1,2

1. Beijing Satellite Navigation Center, Beijing, China

2. Sichuan University, Chengdu, China

1. 185427910@,

2. 675279854@

Abstract: Nowadays, satellite navigation has been applied to all aspects of people's livelihood, infrastructure and national defense because of its high precision, all-weather and non-ground station restrictions. How to ensure the stability of the system related to the national economy and the people's livelihood. In the large-scale satellite navigation and positioning system, the analysis and processing of the massive log and the monitoring of the system status are the key factors to ensure the stability of the system. The existing system through the log troubleshooting issues are through the remote login server to the appropriate directory query,time-consuming and laborious. This paper presents a log analysis and monitoring system based on Elasticsearch, Logstash and Kibana, which is used to collect and retrieve massive log and system information in real time. This system can find, locate and deal with system abnormity more quickly and improve the usability of satellite system. This paper describes the framework of the system, the function and working principle of each component, and describes in detail the process of collecting, filtering and indexing the log and system information to the final query and display.The results show that the method can be used to improve the system in the event of failure through the log for investigation and processing speed.

Keywords: Navigation Big Data;Elasticsearch;Real-Time Search;Distributed

ElasticSearch分布式搜索引擎在导航

大数据检索中的应用

孟方园1，魏涛1,2

1．北京卫星导航中心，北京，中国，100094

2．四川大学，成都，中国，610065

1. 185427910@,

2. 675279854@

【摘要】人类利用太阳,地磁及其他天体进行导航已有数千年历史,随着上世纪70年代GPS的出现,卫星导航开启了人类导航的新篇章。在互联网迅猛发展的带动下，基于卫星导航的应用和产品层出不穷，随之而来的是日益增长的海量卫星导航数据，导航数据区别于其他数据的一大特点是它的时空特性。本文提出一种基于ElasticSearch分布式搜索引擎的导航大数据处理技术，可高效、可靠的对导航大数据进行导入、索引和分布式存储,并以近实时的性能进行全文检索及按字段检索，特别对包含位置信息的数据实现了地理检索功能，同时提供冲突管理，容灾备份,RESTful API等多项特性。利用该技术可大幅提高分析和处理海量卫星导航数据的效率。文中使用实际的GSP数据进行测试，实测表明在百万到千万级导航数据量下，该方法具有较好的可靠性和性能。

【关键词】Elasticsearch；实时搜索；分布式

1 引言

现如今，卫星导航技术已充分运用在各行各业之中，具体地说，就是把导航芯片嵌入和装备到电网、铁路、桥梁、隧道、公路、建筑、供水系统、大坝、油气管道等各种物体中，然后将其与现有的互联网整合起来，实现人类社会与物理系统的整合，在这个整合的网络当中，存在能力超级强大的中心计算机群，通过获取各类设备产生的数据，并对这些海量数据进行分析处理，从而实现提高资源利用率和生产力水平，改善人与自然间关系的目的。然而随着数据量的日益增长以及导航数据的时空特性，对如何高效的处理这类数据提出了挑战。

利用分布式[1,2]的方式处理大数据是一个有效的措施，分布式ElasticSearch构建的检索系统近年来也不断出现，如陈亚杰等人研究的基于分布式ElasticSearch 的天文大数据检索[3]，姜康等人讨论了基于