| 地面气象资料统计处理系统设计与实现 |
| |
| 引用本文： | 孙超,霍庆,任芝花,刘振,肖卫青,徐拥军.地面气象资料统计处理系统设计与实现[J].应用气象学报,2018,29(5):630-640. |
| |
| 作者姓名： | 孙超 霍庆 任芝花 刘振 肖卫青 徐拥军 |
| |
| 作者单位： | 国家气象信息中心, 北京 100081 |
| |
| 基金项目： | 公益性行业（气象）科研专项（GYHY201406016），国家气象信息中心“山洪地质灾害防治气象保障工程2016年建设”项目 |
| |
| 摘 要： | 为实现6万余个地面气象站资料实时统计处理，设计并实现了地面气象资料统计处理系统。系统采用灵活易扩展的技术框架，应用大数据分布式流处理技术实现高效的数据处理，较传统统计处理框架统计时效提升10掊以上，主要功能包括基于任务调度的定时统计、针对迟到数据和数据更正信息的统计结果滚动更新、自定义统计、统计处理通用算法服务等，采用国家级一级部署、各省同步应用的业务布局保证数据的一致性。2017年1月投入业务运行后，实时计算800余个多种尺度的统计项并通过全国综合气象信息共享平台（CIMISS）数据统一服务接口提供统计产品服务。分析表明:地面气象资料统计产品2017年月平均下载次数达到1951.4万次，在CIMISS所有400余种观测资料或产品中排名第三，为天气监测预报预警、气象决策服务、气候监测业务、公共气象服务等提供重要的基础数据支撑。
|
| 关 键 词： | 地面气象资料 统计处理 分布式计算 模块化 滚动更新 |
| 收稿时间： | 2018/6/6 0:00:00 |
| 修稿时间： | 2018/8/10 0:00:00 |
Design and Implementation of Surface Meteorological Data Statistical Processing System |
| |
| Sun Chao,Huo Qing,Ren Zhihu,Liu Zhen,Xiao Weiqing and Xu Yongjun.Design and Implementation of Surface Meteorological Data Statistical Processing System[J].Quarterly Journal of Applied Meteorology,2018,29(5):630-640. |
| |
| Authors: | Sun Chao Huo Qing Ren Zhihu Liu Zhen Xiao Weiqing and Xu Yongjun |
| |
| Affiliation: | National Meteorological Information Center, Beijing 100081 |
| |
| Abstract: | Statistical products of surface meteorological data (SMD) are among the most-frequently-used data in meteorological research and operations. As the improvement of surface meteorological observation system over China, statistics of SMD have encountered problems such as large number of sites, wide variety of elements, and complexity of statistical strategy. With typical features of big data, it''s possible for SMD to serve more precise and efficient operations nowadays, which is obviously beyond the capability of traditional serial processing framework.Aiming at precise and efficient statistic processing of data from more than 60000 surface weather stations, a statistical processing system for SMD is built based on big data technology. Compared to traditional serial processing framework, efficiency of the system has increased by more than 10 times and more statistics and function are provided, such as fast calculation, rolling update of statistical values according to late-arriving data and corrected information, and arbitrary time scale statistics. Storm distributed flow processing technology is applied in the system to realize efficient statistical calculations. Big data message transmission and cache technology are also applied to ensure the system''s high efficiency and stability. Modular design framework ensures strong extensibility of the system, based on which statistics, quality control and evaluation algorithms are extended to varieties of data, e.g., upper-air, radiation, oceanic and aircraft measurements. The system is deployed at national meteorology department and its products are synchronously applied at the provincial level, for this layout ensures data consistency.The system is incorporated into China Integrated Meteorological Information Sharing System (CIMISS) and become its core data processing framework. The system provides more than 800 real-time multi-scale SMD statistical values to serve meteorological users and the public through CIMISS data unified service interface since January 2017. Based on data access logs, monthly access of daily SMD statistics reach 19.51 million times in 2017, ranking the 3th among over 400 data or products, playing important roles in weather monitoring, forecasting and warning, meteorological decision, public service and climate research.In the future, the technical framework and algorithm module of the system will be integrated into the processing pipeline of meteorological large data cloud platform, with further optimization of the computational topology for full use of computing resources, which can increase convergence time for distributed node processing results. To further improve the efficiency of statistical processing, the launching mechanism of this operation can be changed from periodic to automatic scheduling based on the trigger of observed data integrity. |
| |
| Keywords: | surface meteorological data statistic processing distributed computing modularization rolling updating |
| 本文献已被 CNKI 等数据库收录！ |
| 点击此处可从《应用气象学报》浏览原始摘要信息 |
|
点击此处可从《应用气象学报》下载全文 |
|
