排序方式： 共有30条查询结果，搜索用时 21 毫秒
1.
大数据技术为处理海量地震观测数据提供了一种新的数据存储与计算模式。为了解决现有基于关系数据库存储方案的读写速度低、用户并发度低和可扩展性差等问题,以地震前兆观测数据为例,在详细分析业务需求的基础上,提出了基于HBase和Open TSDB的地震大数据存储方案,搭建了大数据测试平台,完成了不同存储方案下查询、插入性能实验和并发性实验。实验结果表明,与关系数据库存储方案相比,基于HBase和Open TSDB的存储方案具有很好的可扩展性和并发性,经过优化后的HBase存储方案具有更高的读取和存储性能。 相似文献
2.
在Hadoop分布式计算和存储架构下,自定义ETL数据清洗规则将海量自动站小时单站文件按所属年和站号合并为大文件流转存储至HDFS中,并运用SparkSQL并行计算框架进行统计处理生成常用气象要素日统计值。结果表明,数据处理和获取时效较关系型数据库方式有显著提升。采用SparkSQL并行计算框架对多气象要素多站点和长时间序列进行数据统计处理查询均能达到秒级别响应,并随着统计站点数的不断增加和时间跨度的延长其优势更为明显,能更高效地支撑此类气象数据服务,为海量气象数据处理从关系型数据库到大数据分布式架构的转换处理提供了新思路。 相似文献
3.
针对当前湖北省交通运输数据的存储管理与应用瓶颈,结合目前大数据技术的发展背景,本文基于Hadoop生态体系设计与实现了针对湖北交通运输业务的大数据存储分析平台。文中从总体架构、存储结构、扩展架构、数据接入与治理、分布式任务调度的设计实现到平台的应用场景进行了论述,并使用10年的高速公路数据与具体业务场景对平台进行了存储与效率实验。实验结果表明,相比传统关系型数据库,该平台在数据存储、查询计算方面均具有非常明显的优越性。 相似文献
4.
云计算技术使用分布式的计算技术实现了并行计算的计算能力和计算效率,解决了单机服务器计算能力低的问题。基于长序列历史资料所计算得出的气候标准值对于气象领域实时业务、准实时业务及科学研究中均具有重要的意义。由于长序列历史资料数据量大、运算逻辑较复杂,在传统单节点计算平台上进行整编计算耗时非常长。该文基于Hadoop分布式计算框架搭建了集群模式的云计算平台,以长序列历史资料作为源数据,基于MapReduce计算模型实现了部分整编算法,提高计算时效。同时,由于数据源本身具有文件个数多、单个文件小等特点,对数据源存储形式及数据文件大小进行改造,分别利用SequenceFile方式及文本文件合并方式对同一种场景进行计算时效对比测试,分别测试了10个文件合并、100个文件合并两种情况,使时效性得到了更大程度的提升。 相似文献
5.
6.
针对时空数据存储与查询问题,传统方法存在硬件成本高,存储效率低等缺点。通过对MapReduce模型和Hadoop框架等云计算核心技术的分析和研究,提出了一种基于Hadoop的时空数据存储模型,并在此模型的基础上,设计了基于MapReduce的时空数据查询并行化框架。该框架通过对时空数据的并行操作,使其适用于海量时空数据的存储与管理。 相似文献
7.
Spatiotemporal data represent the real-world objects that move in geographic space over time. The enormous numbers of mobile sensors and location tracking devices continuously produce massive amounts of such data. This leads to the need for scalable spatiotemporal data management systems. Such systems shall be capable of representing spatiotemporal data in persistent storage and in memory. They shall also provide a range of query processing operators that may scale out in a cloud setting. Currently, very few researches have been conducted to meet this requirement. This paper proposes a Hadoop extension with a spatiotemporal algebra. The algebra consists of moving object types added as Hadoop native types, and operators on top of them. The Hadoop file system has been extended to support parameter passing for files that contain spatiotemporal data, and for operators that can be unary or binary. Both the types and operators are accessible for the MapReduce jobs. Such an extension allows users to write Hadoop programs that can perform spatiotemporal analysis. Certain queries may call more than one operator for different jobs and keep these operators running in parallel. This paper describes the design and implementation of this algebra, and evaluates it using a benchmark that is specific to moving object databases. 相似文献
8.
Zhenlong Li Fei Hu John L. Schnase Daniel Q. Duffy Tsengdar Lee Michael K. Bowen 《International journal of geographical information science》2017,31(1):17-35
Climate observations and model simulations are producing vast amounts of array-based spatiotemporal data. Efficient processing of these data is essential for assessing global challenges such as climate change, natural disasters, and diseases. This is challenging not only because of the large data volume, but also because of the intrinsic high-dimensional nature of geoscience data. To tackle this challenge, we propose a spatiotemporal indexing approach to efficiently manage and process big climate data with MapReduce in a highly scalable environment. Using this approach, big climate data are directly stored in a Hadoop Distributed File System in its original, native file format. A spatiotemporal index is built to bridge the logical array-based data model and the physical data layout, which enables fast data retrieval when performing spatiotemporal queries. Based on the index, a data-partitioning algorithm is applied to enable MapReduce to achieve high data locality, as well as balancing the workload. The proposed indexing approach is evaluated using the National Aeronautics and Space Administration (NASA) Modern-Era Retrospective Analysis for Research and Applications (MERRA) climate reanalysis dataset. The experimental results show that the index can significantly accelerate querying and processing (~10× speedup compared to the baseline test using the same computing cluster), while keeping the index-to-data ratio small (0.0328%). The applicability of the indexing approach is demonstrated by a climate anomaly detection deployed on a NASA Hadoop cluster. This approach is also able to support efficient processing of general array-based spatiotemporal data in various geoscience domains without special configuration on a Hadoop cluster. 相似文献
9.
10.
数值预报产品数据与日俱增,采用传统的关系型数据库对其进行存储和管理存在效率低和存储能力不足的问题。另外,基于文件的存储方式在数据存储处理、数据读取和算法计算等方面存在性能瓶颈。针对这一问题,基于Hadoop技术体系设计了分布式的数据存储模型,实现了数值预报产品数据的分布式存储和处理,开发了数值预报产品数据接入处理模块;并实现了基于Rest Web Service的获取数值预报产品要素场数据访问接口、时间序列数据访问接口、数据下载接口等业务应用接口。多业务用户的实际业务测试表明, 该平台在诸如数值预报产品气象数据处理和业务应用方面较传统技术架构具有一定优势。 相似文献