共查询到20条相似文献，搜索用时 46 毫秒
1.
Input/output (I/O) of geospatial raster data often becomes the bottleneck of parallel geospatial processing due to the large data size and diverse formats of raster data. The open‐source Geospatial Data Abstraction Library (GDAL), which has been widely used to access diverse formats of geospatial raster data, has been applied recently to parallel geospatial raster processing. This article first explores the efficiency and feasibility of parallel raster I/O using GDAL under three common ways of domain decomposition: row‐wise, column‐wise, and block‐wise. Experimental results show that parallel raster I/O using GDAL under column‐wise or block‐wise domain decomposition is highly inefficient and cannot achieve correct output, although GDAL performs well under row‐wise domain decomposition. The reasons for this problem with GDAL are then analyzed and a two‐phase I/O strategy is proposed, designed to overcome this problem. A data redistribution module based on the proposed I/O strategy is implemented for GDAL using a message‐passing‐interface (MPI) programming model. Experimental results show that the data redistribution module is effective. 相似文献
2.
3.
Improving the Elevation Accuracy of Digital Elevation Models: A Comparison of Some Error Detection Procedures 总被引：1，自引：1，他引：0
Carlos López 《Transactions in GIS》2000,4(1):43-64
The widespread availability of powerful desktop computers, easy‐to‐use software tools and geographic datasets have raised the quality problem of input data to be a crucial one. Even though accuracy has been a concern in every serious application, there are no general tools for its improvement. Some particular ones exist, however, and some results are presented here for a particular case of quantitative raster data: Digital Elevation Models (DEM). Two procedures designed to detect anomalous values (also named gross errors, outliers or blunders) in DEMs, but valid also for other quantitative raster datasets, were tested. A DEM with elevations varying from 181 to 1044 m derived from SPOT data has been used as a contaminated sample, while a manually derived DEM obtained from aerial photogrammetry was regarded as the ground truth to allow a direct performance comparison for the methods with real errors. It is assumed that a “better” value can be measured or obtained through some methodology once an outlier location is suggested. The options are different depending upon the user (DEM producers might go to the original data and make another reading, while end users might use interpolation). Both choices were considered in this experiment. Preliminary results show that for the available dataset, the accuracy might be improved to some extent with very little effort. Effort is defined here as the percentage of points suggested by the methodology in relation with its total number: thus 100 per cent effort implies that all points have been checked. The method proposed by López (1997) gave poor results, because it has been designed for errors with low spatial autocorrelation (which is not the case here). A modified version was then designed and compared with the method proposed by Felicísimo (1994). The three procedures can be applied both for error detection during DEM generation and by end users, and they might be of use for other quantitative raster data. The choice of the best methodology is different depending on the effort involved. The conclusions have been derived for a photogrammetrically obtained DEM; other production procedures might lead to different results. 相似文献
4.
This article mainly introduces a class of encoding and compression methods for hexagonal raster data. A new encoding mode is established with the introduction of the Gosper curve, which has good spatial aggregation. On this basis, straightforward encoding, lossless coding compression, and lossy coding compression can be carried out. First, the bidirectional correlation between the Gosper curve and the hexagonal grid data is established, and the Gosper coding value of each raster unit is determined. Then, the lossless compression of the data is completed by performing run‐length encoding on the coding set. Following the Gosper curve trend, partial grid units are semantically changed via adjacent fusion under certain threshold constraints. The run‐length encoding is carried out again to complete the lossy compression and further reduce the amount of encoding required. A break‐detection step is added during the process to avoid regional breaks. The experimental results show that the proposed method can efficiently compress data. Additionally, data compression of multi‐resolution and varying fusion thresholds is explored to summarize general patterns. Finally, the method proposed is compared with traditional row‐coding compression in hexagonal and quadrilateral grids to highlight its advantages for hexagonal raster data. 相似文献
5.
Significant trends in the processing of geographical data require increasingly powerful software and hardware, consistent
with the exploitation of parallel computing. Despite recent progress in technology, exploiting parallel processing is still
difficult so that few applications have been developed in the environmental and geographical domains.
Key issues which must be addressed in the design of parallel geographical software are described with reference to designs
for three examples which use grid and raster data. The implications for parallel processing with vector-topological data are
then explored. The emphasis is upon MIMD architectures using strategies of decomposition into subareas, and upon the need
to facilitate development of parallel geographical applications by encapsulating the parallelism in a low-level layer of software,
forming a skeletal framework upon which application algorithms can be built. The parallel layer will support distribution
of datasets across the multiple processors, and the creation and collation of datasets from those processors. 相似文献
6.
双线性插值近似网格的栅格数据投影变换 总被引：1，自引：0，他引：1
在栅格数据投影变换常用算法的基础上,介绍一种新的基于双线性内插近似网格算法的栅格数据投影变换,并详细描述其算法过程,认真分析其特点,并证明该算法在大范围栅格数据投影变换中有显著优点。 相似文献
7.
基于Applet/Servlet模式的WebGIS实现策略和技术研究 总被引：4，自引：0，他引：4
分析了采用Applet/Servlet多层应用结构的原因,提出了以栅格和矢量相混合的数据传递方式,满足了WebGIS的客户端对服务器端空间数据进行快速获取和分析应用的需求,阐述了实现中可以采用的若干策略,并以实例验证了其可行性和实用性。 相似文献
8.
Due to high data volume, massive spatial data requires considerable computing power for real‐time processing. Currently, high performance clusters are the only economically viable solution given the development of multicore technology and computer component cost reduction in recent years. Massive spatial data processing demands heavy I/O operations, however, and should be characterized as a data‐intensive application. Data‐intensive application parallelization strategies, such as decomposition, scheduling and load‐balance, are much different from that of traditional compute‐intensive applications. In this article we introduce a Split‐and‐Merge paradigm for spatial data processing and also propose a robust parallel framework in a cluster environment to support this paradigm. The Split‐and‐Merge paradigm efficiently exploits data parallelism for massive data processing. The proposed framework is based on the open‐source TORQUE project and hosted on a multicore‐enabled Linux cluster. A specific data‐aware scheduling algorithm was designed to exploit data sharing between tasks and decrease the data communication time. Two LiDAR point cloud algorithms, IDW interpolation and Delaunay triangulation, were implemented on the proposed framework to evaluate its efficiency and scalability. Experimental results demonstrate that the system provides efficient performance speedup. 相似文献
9.
Fedor Baart Gerben de Boer Wim de Haas Gennadii Donchyts Marc Philippart Mark van Koningsveld Maarten Plieger 《Transactions in GIS》2012,16(2):249-265
Numerical models produce output with a large number of variables, grid cells and time steps. The same applies to algorithms that produce gridded datasets from sparse or abundant raw data. Further use of the resulting data products has been challenging, especially for dissemination outside the institute of origin. Due to the gradually increasing size of data products, simply downloading copies of them is becoming impossible. A gradual transition from traditional download methods to web services is therefore observed. Web services allow for on‐the‐fly access to subsets of data that were hitherto considered as indivisible granules. Here we compare the most mature candidates to serve gridded data through the web: the Open‐source Project for a Network Data Access Protocol (OPeNDAP) and Web Coverage Service (WCS) protocols. In the framework of the new Dutch National Model and Data Centre (NMDC.eu) a distributed data storage has been created by coupling OPeNDAP servers. A WCS service layer is provided for the same data. This allows us to compare OPeNDAP and WCS. Using several use cases, we compare the usability, performance and features of the two protocols. 相似文献
10.
Cumulative spatial impact layers: A novel multivariate spatio‐temporal analytical summarization tool
Lucy Romeo Jake Nelson Patrick Wingo Jennifer Bauer Devin Justman Kelly Rose 《Transactions in GIS》2019,23(5):908-936
Scientific inquiry often requires analysis of multiple spatio‐temporal datasets, ranging in type and size, using complex multi‐step processes demanding an understanding of GIS theory and software. Cumulative spatial impact layers (CSIL) is a GIS‐based tool that summarizes spatio‐temporal datasets based on overlapping features and attributes. Leveraging a recursive quadtree method, and applying multiple additive frameworks, the CSIL tool allows users to analyze raster and vector datasets by calculating data, record, or attribute density. Providing an efficient and robust method for summarizing disparate, multi‐format, multi‐source geospatial data, CSIL addresses the need for a new integration approach and resulting geospatial product. The built‐in flexibility of the CSIL tool allows users to answer a range of spatially driven questions. Example applications are provided in this article to illustrate the versatility and variety of uses for this CSIL tool and method. Use cases include addressing regulatory decision‐making needs, economic modeling, and resource management. Performance reviews for each use case are also presented, demonstrating how CSIL provides a more efficient and robust approach to assess a range of multivariate spatial data for a variety of uses. 相似文献
11.
Big geospatial data is an emerging sub‐area of geographic information science, big data, and cyberinfrastructure. Big geospatial data poses two unique challenges. First, raster and vector data structures and analyses have developed on largely separate paths for the last 20 years. This is creating an impediment to geospatial researchers seeking to utilize big data platforms that do not promote heterogeneous data types. Second, big spatial data repositories have yet to be integrated with big data computation platforms in ways that allow researchers to spatio‐temporally analyze big geospatial datasets. IPUMS‐Terra, a National Science Foundation cyberInfrastructure project, addresses these challenges by providing a unified framework of integrated geospatial services which access, analyze, and transform big heterogeneous spatio‐temporal data. As IPUMS‐Terra's data volume grows, we seek to integrate geospatial platforms that will scale geospatial analyses and address current bottlenecks within our system. However, our work shows that there are still unresolved challenges for big geospatial analysis. The most pertinent is that there is a lack of a unified framework for conducting scalable integrated vector and raster data analysis. We conducted a comparative analysis between PostgreSQL with PostGIS and SciDB and concluded that SciDB is the superior platform for scalable raster zonal analyses. 相似文献
12.
随着航空航天遥感技术的不断发展,以遥感影像为代表的栅格数据分辨率越来越高,遥感影像处理呈现出数据量大、复杂度高的特点。近年来,通用GPU的运算性能不断提高为加速密集运算提供了新的途径,目前,采用GPU并行技术进行遥感影像处理成为新的研究热点。本文提出了基于GPU并行计算的巨幅遥感影像坐标转换方法,实践证明,相比于传统的转换方法基于GPU的算法有较为明显的提速。 相似文献
13.
Fei Hu Yongyao Jiang Yun Li Weiwei Song Daniel Q. Duffy 《International Journal of Digital Earth》2020,13(3):410-428
ABSTRACTEarth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency. 相似文献
14.
Harri Antikainen 《Transactions in GIS》2013,17(1):96-108
The conventional raster‐based least‐cost path technique, in which the connectivity graph is constructed by treating each raster cell as a node and linking each node to its adjacent neighbors, has been widely used to determine optimum paths in space. Unfortunately, paths calculated with this method are subject to distortions that stem from the imposed node‐link structure. Due to the distortions, the calculated paths may be longer than the true optimum paths in continuous space, and because of this, they may also look highly unrealistic when displayed on a map. This study examines how the quality of the raster‐based paths can be improved by using larger connectivity patterns on the one hand, and placing nodes on the sides of the cells of a raster instead of at their centers, on the other. In addition, the utility of the quadtree structure is examined. The different methods are evaluated against three different datasets using the cost distance of the calculated paths and processing times as evaluation criteria. The results indicate that raster‐based paths can be improved by using the evaluated techniques, although the degree of improvement is also dependent on the properties of the underlying cost raster. 相似文献
15.
随着对地立体观测体系的建立,遥感大数据不断累积。传统基于文件、景/幅式的影像组织方式,时空基准不够统一,集中式存储不利于大规模并行分析。对地观测大数据分析仍缺乏一套统一的数据模型与基础设施理论。近年来,数据立方体的研究为对地观测领域大数据分析基础设施提供了前景。基于统一的分析就绪型多维数据模型和集成对地观测数据分析功能,可构建一个基于数据立方的对地观测大数据分析基础设施。因此,本文提出了一个面向大规模分析的多源对地观测时空立方体,相较于现有的数据立方体方法,强调多源数据的统一组织、基于云计算的立方体处理模式以及基于人工智能优化的立方体计算。研究有助于构建时空大数据分析的新框架,同时建立与商业智能领域的数据立方体关联,为时空大数据建立统一的时空组织模型,支持大范围、长时序的快速大规模对地观测数据分析。本文在性能上与开源数据立方做了对比,结果证明提出的多源对地观测时空立方体在处理性能上具有明显优势。 相似文献
16.
An empirical study was performed assessing the accuracy of land use change detection when using satellite image data acquired ten years apart by sensors with differing spatial resolutions. Landsat/Multi‐spectral Scanner (MSS) with Landsat/Thematic Mapper (TM) or SPOT/High Resolution Visible (HRV) multi‐spectral (XS) data were used as a multi‐data pair for detecting land use change. The primary objectives of the study were to: (1) compare standard change detection methods (e.g. multi‐date ratioing and principal components analysis) applied to image data of varying spatial resolution; (2) assess whether to transform the raster grid of the higher resolution image data to that of the lower resolution raster grid or vice‐versa in the registration process: and (3) determine if Landsat/TM or SPOT/ HRV(XS) data provides more accurate detection of land use changes when registered to historical Landsat/MSS data. Ratioing multi‐sensor, multi‐date satellite image data produced higher change detection accuracies than did principal components analysis and is useful as a land use change enhancement technique. Ratioing red and near infrared bands of a Landsat/MSS‐SPOT/HRV(XS) multi‐date pair produced substantially higher change detection accuracies (～10%) than ratioing similar bands of a Landsat/MSS ‐ Landsat/TM multi‐data pair. Using a higher‐resolution raster grid of 20 meters when registering Landsat/MSS and SPOTZHRV(XS) images produced a slightly higher change detection accuracy than when both images were registered to an 80 meter raster grid. Applying a “majority”; moving window filter whose size approximated a minimum mapping unit of 1 hectare increased change detection accuracies by 1–3% and reduced commission errors by 10–25%. 相似文献
17.
Fang Huang Ji Zhou Xicheng Tan Shunlin Liang Jie Cheng 《International Journal of Digital Earth》2016,9(9):819-834
MODerate resolution atmospheric TRANsmission (MODTRAN) is a commercial remote sensing (RS) software package that has been widely used to simulate radiative transfer of electromagnetic radiation through the Earth's atmosphere and the radiation observed by a remote sensor. However, when very large RS datasets must be processed in simulation applications at a global scale, it is extremely time-consuming to operate MODTRAN on a modern workstation. Under this circumstance, the use of parallel cluster computing to speed up the process becomes vital to this time-consuming task. This paper presents PMODTRAN, an implementation of a parallel task-scheduling algorithm based on MODTRAN. PMODTRAN was able to reduce the processing time of the test cases used here from over 4.4 months on a workstation to less than a week on a local computer cluster. In addition, PMODTRAN can distribute tasks with different levels of granularity and has some extra features, such as dynamic load balancing and parameter checking. 相似文献
18.
S.T. Arundel L.A. Phillips A.J. Lowe J. Bobinmyer K.S. Mantey C.A. Dunn 《制图学和地理信息科学》2015,42(4):40-53
The 3D Elevation Program (3DEP) is a collaborative effort among government entities, academia, and the private sector to collect high-resolution 3-dimensional data over the United States. The United States Geological Survey (USGS) is making preparations for managing, processing, and delivering petabytes of 3DEP elevation products for the Nation. In addition to the existing 1/3, 1, and 2 arc-second seamless elevation data layers of The National Map, new 3DEP products include lidar point cloud data; a standard 1-meter DEM layer; additional source datasets; and, in Alaska, 5-meter digital elevation models. A new product generation system improves the construction and publication of the seamless elevation datasets, prepares the additional 3DEP products for distribution, and automates the data management functions required to accommodate the high-volume 3DEP data collection. Major changes in geospatial data acquisition, such as high resolution lidar data, volunteered geographic information, data processing using parallel and grid computer systems, and user needs for semantic access to geospatial data and products, are driving USGS research associated with the 3DEP. To address the research requirements, a set of inter-related projects including spatiotemporal data models, data integration, geospatial semantics and ontology, high performance computing, multi-scale representation, and hydrological modeling using lidar and other 3DEP data has been developed. 相似文献
19.
20.
Denis J. Dean 《Transactions in GIS》2011,15(1):87-107
Finding optimal paths through raster databases is limited by the very nature of the raster data itself. This limitation restricts the possible directions of movement through the database from the infinite possibilities found in the real world to a finite number of possibilities defined by the cell‐to‐cell movement that characterizes raster databases. A Triangulated Irregular Network (TIN)‐based alternative optimization model that allows unlimited possible directions of movement is presented. While not without its own limitations, this new approach offers a viable alternative to raster‐based optimal routefinding. 相似文献