| MICAPS4服务端系统架构设计 |
| |
| 引用本文： | 王若曈,王建民,黄向东,董一峰,龙明盛.MICAPS4服务端系统架构设计[J].应用气象学报,2018,29(1):1-12. |
| |
| 作者姓名： | 王若曈 王建民 黄向东 董一峰 龙明盛 |
| |
| 作者单位： | 1.国家气象中心, 北京 100081 |
| |
| 基金项目： | 中国气象局“2015年山洪地质灾害防治气象保障工程建设” |
| |
| 摘 要： | MICAPS4体系采用客户端/服务器的系统架构，其中服务端系统是MICAPS4的重要部分，利用分布式存储与分布式计算技术，构建可容纳102 TB量级的气象实时数据，千万数据总量，面向数百并发用户的服务器集群系统。MICAPS4服务端系统在国内率先实现全部气象实时数据由文件到数据库、从集中式系统到分布式系统的迁移，该系统自2015年起在全国推广使用。在海量气象数据和大量用户并发访问的环境下，表现出很高的稳定性和优越的读写性能，同时具有便捷的扩展性和可维护性。MICAPS4服务端系统分为分布式存储系统、分布式前处理系统、站点实况轮询系统、查询服务器系统和监控系统5个子系统，分布式存储子系统为MICAPS4客户端提供了近实时数据的高速随机与顺序读取服务，分布式前处理系统利用对等分布式架构实现了海量气象实时数据的流式计算，站点实况轮询系统实现了跨系统的实况数据异构副本的同步功能，查询服务器系统利用多线程服务器技术实现了MICAPS4客户端的实时计算请求，监控系统利用部署于每个节点的探针实现监控信息的主动上报。
|
| 关 键 词： | MICAPS4 大数据 分布式存储 分布式计算 实时数据 |
| 收稿时间： | 2017/7/25 0:00:00 |
| 修稿时间： | 2017/12/1 0:00:00 |
The Architecture Design of MICAPS4 Server System |
| |
| Wang Ruotong,Wang Jianmin,Huang Xiangdong,Dong Yifeng and Long Mingsheng.The Architecture Design of MICAPS4 Server System[J].Quarterly Journal of Applied Meteorology,2018,29(1):1-12. |
| |
| Authors: | Wang Ruotong Wang Jianmin Huang Xiangdong Dong Yifeng and Long Mingsheng |
| |
| Institution: | 1.National Meteorological Center, Beijing 1000812.School of Software, Tsinghua University, Beijing 100084 |
| |
| Abstract: | Meteorological data are typical non-structure data, which reach dozens of TBs per day. Data pre-processing,data storage and data access based on RDBMS and file system become the bottleneck of MICAPS3. To fulfill MICAPS4 users'' need of fast, in-time query of meteorological real-time data, according to the multi-dimension model and the user query behavior of meteorological data, using non-relational key-value DDBMS, a high performance massive meteorological data storage system and a stable 7×24 distributed data pre-processing system is designed and established. MICAPS4 uses a client/server system architecture, and high-performance server cluster system is the critical component of MICAPS4. Using distributed key-value data model and P2P infrastructure, MICAPS4 server system distributes all real-time data which arrive at a very high speed to multiple servers through an automatic load balance algorithm, and all data are stored in memory initially and persistent to hard disk periodically, which can not only reduce the disk I/O operating times, but also guarantee the reduction of writing pressure accompanying the high load of reading pressure. To enhance the data and system reliability, distributed system architecture and multiple data replica are used, which also improves the throughput capacity of the system. According to statistic results gained from product environment, the performance of MICAPS4 server system improves 100 times more than MICAPS3. MICAPS4 server system transits all meteorological real-time data storage from file system to database, from centralized system to distributed system. The system becomes the core production system of China Meteorological Administration in 2015 and is popularized nationwide. Under the condition of massive meteorological data and concurrent access of many users, it shows high stability and excellent read-write performance, and it is also highly scalable and maintenance friendly. MICAPS4 high performance server system includes 5 sub-systems including distributed storage system, distributed pre-processing system, station data polling system, data query server and monitoring probe. The distributed storage system provides high performance data accessing services of meteorological real-time data in both random and sequence mode, the distributed pre-processing system implements the stream computing function of massive meteorological real-time data by adopting the peer to peer distributed system infrastructure, the station data polling system implements the heterogeneous station observation replica data synchronization function over different systems, the data query server implements MICAPS4 client real-time computing function by means of the multi-threading server technology, and the monitoring probe is deployed in each server node and reports host health messages periodically. The overall design of MICAPS4 server system is depicted, and the motivation, core technologies and the design of each sub-system are also introduced. |
| |
| Keywords: | MICAPS4 big data distributed storage distributed computation real-time data |
| 本文献已被 CNKI 等数据库收录！ |
| 点击此处可从《应用气象学报》浏览原始摘要信息 |
| 点击此处可从《应用气象学报》下载免费的PDF全文 |
|
