气候变化领域集成服务门户(CCPortal): A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s

CCPortal

DOI	10.3390/rs14030521
	A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s
	Guo, Jifu; Huang, Chunlin; Hou, Jinliang
通讯作者	Hou, JL (通讯作者)，Chinese Acad Sci, Northwest Inst Ecoenvironm & Resources, Key Lab Remote Sensing Gansu Prov, Heihe Remote Sensing Expt Res Stn, Lanzhou 730000, Peoples R China.
发表日期	2022
EISSN	2072-4292
卷号	14 期号:3
英文摘要	As a result of Earth observation (EO) entering the era of big data, a significant challenge relating to by the storage, analysis, and visualization of a massive amount of remote sensing (RS) data must be addressed. In this paper, we proposed a novel scalable computing resources system to achieve high-speed processing of RS big data in a parallel distributed architecture. To reduce data movement among computing nodes, the Hadoop Distributed File System (HDFS) is established on nodes of K8s, which are also used for computing. In the process of RS data analysis, we innovatively use the tile-oriented programming model instead of the traditional strip-oriented or pixel-oriented approach to better implement parallel computing in a Spark on Kubernetes (K8s) cluster. A large RS raster layer can be abstracted as a user-defined tile format of any size, so that a whole computing task can be divided into multiple distributed parallel tasks. The computing resources applied by users would be immediately assigned in the Spark on K8s cluster by simply configuring and initializing SparkContext through a web-based Jupyter notebook console. Users can easily query, write, or visualize data in any box size from the catalog module in GeoPySpark. In summary, the system proposed in this study can provide a distributed scalable resources system for assembling big data storage, parallel computing, and real-time visualization.
关键词	SPATIAL DATA CHALLENGES FRAMEWORK
英文关键词	big data; parallel computing; remote sensing; HDFS on K8s; GeoPySpark; Spark on K8s
语种	英语
WOS研究方向	Environmental Sciences & Ecology ; Geology ; Remote Sensing ; Imaging Science & Photographic Technology
WOS类目	Environmental Sciences ; Geosciences, Multidisciplinary ; Remote Sensing ; Imaging Science & Photographic Technology
WOS记录号	WOS:000756548900001
来源期刊	REMOTE SENSING
来源机构	中国科学院西北生态环境资源研究院
文献类型	期刊论文
条目标识符	http://gcip.llas.ac.cn/handle/2XKMVOVA/253790
作者单位	[Guo, Jifu; Huang, Chunlin; Hou, Jinliang] Chinese Acad Sci, Northwest Inst Ecoenvironm & Resources, Key Lab Remote Sensing Gansu Prov, Heihe Remote Sensing Expt Res Stn, Lanzhou 730000, Peoples R China; [Guo, Jifu] Univ Chinese Acad Sci, Beijing 100049, Peoples R China; [Guo, Jifu] Gansu Agr Univ, Coll Informat Sci & Technol, Lanzhou 730070, Peoples R China
推荐引用方式 GB/T 7714	Guo, Jifu,Huang, Chunlin,Hou, Jinliang. A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s[J]. 中国科学院西北生态环境资源研究院,2022,14(3).
APA	Guo, Jifu,Huang, Chunlin,&Hou, Jinliang.(2022).A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s.REMOTE SENSING,14(3).
MLA	Guo, Jifu,et al."A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s".REMOTE SENSING 14.3(2022).