| 基于MapReduce的空间数据并行划分算法 |
| |
| 引用本文： | 付艳丽,吴艳民,张金标,郑坤,赵长虹,郑康,方发林.基于MapReduce的空间数据并行划分算法[J].测绘通报,2017,0(11):96-100. |
| |
| 作者姓名： | 付艳丽 吴艳民 张金标 郑坤 赵长虹 郑康 方发林 |
| |
| 作者单位： | 1. 济南市勘察测绘研究院, 山东 济南 250013;2. 中国地质大学(武汉)信息工程学院, 湖北 武汉 430074;3. 北京创时空科技发展有限公司, 北京 100083;4. 广东省气象探测数据中心, 广东 广州 510610;5. 武汉兆图科技有限公司, 湖北 武汉 430070 |
| |
| 基金项目： | 国家重点研发计划（2016YFB0502603）；湖北省自然科学基金（ZRY2015001543）；中国地质大学（武汉）中央高校基本科研业务费资金（1610491B20） |
| |
| 摘 要： | 针对海量空间数据分布式存储中存在的不顾及空间邻近性、分布不均和数据倾斜的问题，基于MapReduce并行编程模型，对Hilbert空间曲线层次分解的思想和节点容量感知的方法进行了研究，提出了一种层次分解的空间数据并行划分策略，并通过临界值判定实现空间数据的均衡存储。最后通过实例分析说明该方法可以在保证空间数据邻近特性的同时，解决海量空间数据分布式存储不均和数据倾斜的问题。
|
| 关 键 词： | MapReduce Hilbert空间曲线 空间数据并行划分 |
| 收稿时间： | 2017-05-16 |
Spatial Data Parallel Partitioning Algorithm Based on MapReduce |
| |
| FU Yanli,WU Yanmin,ZHANG Jinbiao,ZHENG Kun,ZHAO Changhong,ZHENG Kang,FANG Falin.Spatial Data Parallel Partitioning Algorithm Based on MapReduce[J].Bulletin of Surveying and Mapping,2017,0(11):96-100. |
| |
| Authors: | FU Yanli WU Yanmin ZHANG Jinbiao ZHENG Kun ZHAO Changhong ZHENG Kang FANG Falin |
| |
| Institution: | 1. Jinan Geotechnical Investigation and Surverying Institute, Jinan 250013, China;2. School of Information Engineering, China University of Geosciences(Wuhan), Wuhan 430074, China;3. Beijing Create Space-time Science and Technology Limited Company, Beijing 100083, China;4. Guangdong Meteorological Observation Data Center, Guangzhou 510610, China;5. Wuhan Trillion Map Technology Limited Company, Wuhan 430070, China |
| |
| Abstract: | Spatial data partitioning method plays an important role in spatial data distributed storage, and its key problem is how topartition spatial data to distributed storage nodes in network environment. This paper discusses massive spatial data partitioning strategies and analyses their disadvantages which these partitioning methods have not taken into account spatial object size and spatial proximity. Aiming at these questions,this paper proposes a new spatial data parallelpartitioning strategy based on MapReduce and capacity-aware method to improve load balance which could avoid unevenly distributed data storage and data skew. Experimental analysis shows that the presented spatial data parallel partitioning algorithm not only achieves better storage load balance in distributed storage system,but also keeps well spatial locality of data objects after partitioning. |
| |
| Keywords: | MapReduce Hilbert space filling curve spatial data parallel partitioning |
| 本文献已被 CNKI 等数据库收录！ |
| 点击此处可从《测绘通报》浏览原始摘要信息 |
| 点击此处可从《测绘通报》下载免费的PDF全文 |
|
