| 基于数据湖的省级卫星直收站数据共享服务设计 |
| |
| 引用本文： | 李显风,袁正国,周雪莹,邓卫华,刘海燕.基于数据湖的省级卫星直收站数据共享服务设计[J].气象科技,2024,52(1):45-54. |
| |
| 作者姓名： | 李显风 袁正国 周雪莹 邓卫华 刘海燕 |
| |
| 作者单位： | 江西省气象数据中心，南昌 330096 |
| |
| 基金项目： | 江西省03专项及5G项目（20212ABC03W02）、江西省气象科技重点项目(JX2020Z19)资助 |
| |
| 摘 要： | 针对省级卫星直收站数据存储分散造成共享应用不便的问题，提出卫星多源数据便捷高效访问和安全分级管理方案。引入数据湖技术，按照全国统一规范构建了基于气象大数据云平台的省级气象数据湖系统，基于此实现了FY 3、FY 4卫星地面直收站原始接收数据、加工处理产品等多种卫星数据入湖集中管理，并提供统一的数据共享服务，解决了海量多源异构数据的存储、管理和共享服务等技术难题。通过数据湖统一权限控制，实现对用户访问数据的分级管理，保障了数据应用的稳定性和安全性。通过数据湖统一文件目录服务，实现长时间序列文件的便捷快速获取，有效支撑卫星数据的服务应用和挖掘分析。
|
| 关 键 词： | 数据湖 卫星直收站数据 气象大数据云平台 共享服务 |
| 收稿时间： | 2023/2/18 0:00:00 |
| 修稿时间： | 2023/6/25 0:00:00 |
Design of Provincial Satellite Direct-Receiving Data Sharing Service Based on Data Lake |
| |
| LI Xianfeng,YUAN Zhengguo,ZHOU Xueying,DENG Weihu,LIU Haiyan.Design of Provincial Satellite Direct-Receiving Data Sharing Service Based on Data Lake[J].Meteorological Science and Technology,2024,52(1):45-54. |
| |
| Authors: | LI Xianfeng YUAN Zhengguo ZHOU Xueying DENG Weihu LIU Haiyan |
| |
| Institution: | Jiangxi Meteorological Data Center, Nanchang 330096 |
| |
| Abstract: | Meteorological satellites play an increasingly important role in meteorological observation, disaster prevention and mitigation, and weather forecasting. Direct reception, satellite broadcasting, and website download are common methods for obtaining satellite data. A number of high speed data ground satellite direct reception stations have been built in some regions of each province across China to receive and process various satellite data products such as FY 3 and FY 4 in real time. Currently, the satellite direct receiving data are stored on different devices, which poses significant inconvenience for data sharing applications. Convenient and efficient data access with hierarchical security management is urgently needed. Although satellite data product files can be read and accessed through the Tianqing unified data service interface, the files need to be downloaded time after time and stored onto a local disk directory for further processing. This traditional service mode has disadvantages of low time efficiency and short time sequences, which can not meet the demand of big data applications such as artificial intelligence for long time series product files. Therefore, there is an urgent need to build a unified big data file sharing system. A data lake can store any type of massive data and has good scalability. Various data files centralisedly stored can be directly accessed through the data lake engine for analysis and mining. Consequently, a data lake can reduce the repeated storage and migration of data files and significantly improve the support capability in big data applications. So, in this paper, we propose a convenient and efficient access and security hierarchical management scheme for satellite multi source data using a data lake. We introduce data lake technology to build a provincial meteorological data lake system based on the meteorological big data cloud platform with a set of unified national standards. A substantial volume of raw data and processed products from FY 3 and FY 4 satellite ground receiving stations are being aggregated into the data lake, enabling a unified data sharing service. The technical challenges such as massive multi source heterogeneous data storage, data management, and data sharing service are being partially resolved to a certain extent. Through the unified authority control technology, we achieve the hierarchical permission management of data access, ensuring the stability and security in data applications. Furthermore, the unified file directory service technology facilitates fast and convenient retrieval of long time series files, providing significant support for the service applications and satellite data mining in the era of big data. |
| |
| Keywords: | data lake satellite direct reception data meteorological big data cloud platform sharing service |
|
| 点击此处可从《气象科技》浏览原始摘要信息 |
| 点击此处可从《气象科技》下载免费的PDF全文 |
|
