Philippines雷达数据采集与分发开发
合集下载
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
• While highly appealing, raises concerns:
o Cost and difficulty of infrastructure upgrade o Internet connection speed and reliability
FOSS4G 2015 – Seoul, South Korea – September 14th-19th,riculture ▪ Forest ▪ Coastal ▪ Energy ▪ Hydrology
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
LiPAD Web Interface - Tile Selection
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
• File-based storage systems are commonly used to store data • Little setup needed • Pervasive technology • Complexity of directory structure increases with amount of data and processes
Image source: /index.php/about/systemarch2
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - Combined Approach
Working Design -LiPAD
• LiPAD, the customized GeoNode platform, handles the web user interface • GeoServer stores small shapefiles, raster, and vector data • Large files are tiled, named and indexed by the coordinates of each tile and stored
in Ceph • Metadata of tiled files is indexed in LiPAD, represented by a tiled shapefile of the
Philippines
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
• The Philippines is settled along the Pacific Typhoon Belt and the Ring of Fire • It is prone to earthquakes, typhoons, and other hazards • It is abundant in natural resources • There is a need for mapping to assess disaster risk and accounting of natural
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - Combined Approach
• e.g. OpenTopography’s architecture • Comprised of various software and
Working Design
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Working Design - LiDAR Portal for Archiving and Distribution (LiPAD)
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - File Based Storage
LiPAD Web Interface - Data Cart
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Ceph Object Storage System
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
• LiDAR mapping produces high-resolution geospatial data • High resolution data acquisition leads to humongous data sizes • Storage, indexing, retrieval and distribution proves a challenge
hardware resources • Actual data sets are stored as ASPRS
LAS format on a dedicated storage server • Metadata is stored on an IBM DB2 database • Other datasets are stored on the SDSC Cloud platform • Processing and visualization requests are handled by a dedicated large memory, multiprocessor system
o Spatial indexing is on a separate layer o Specialized spatial columns
• Indexing overhead, especially on updates • Scalability and query time issues • Limited support for point cloud data
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
DEVELOPMENT OF DATA ARCHIVING AND DISTRIBUTION SYSTEM FOR THE PHILIPPINES' LIDAR PROGRAM USING OBJECT STORAGE SYSTEMS
Ken Abryl Eleazar Salanio Data Archiving and Distribution Component
2014 Typhoon Tracks
Image sources: https:///wiki/Timeline_of_the_2014_Pacific_typhoon_season
https:///wiki/Ring_of_Fire
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - Combined Approach
• Combines LiDAR flat tiles, RDBMS, and distributed infrastructure o RDBMS manages metadata o LiDAR tiles are stored in dedicated and distributed storage o Data processing is carried out by high-performance compute servers
Introduction – The Philippine Hazard Setting
PHL-LiDAR 1 ▪ Data Acquisition ▪ Data Validation ▪ Data Processing ▪ Training & IEC ▪ Data Archiving ▪ Flood Modeling
PHL-LiDAR 1 kasalanio@.ph
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Outline
• Introduction • Related Work • Working Design • Ceph Object Storage System • Archiving Process Flow • Summary
resources
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
• The Philippines’ Department of Science and Technology (DOST) with Higher Education Institutions (HEIs) organized programs for mapping: PHL-LiDAR 1 and PHL-LiDAR 2
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - GIS-enabled RDBMS
• GIS-enabled Relational Database Management Systems or Geodatabases • Two types of design:
• These programs are an extension of the Disaster Risk and Exposure Assessment for Mitigation (DREAM) LiDAR program
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Working Design
• Added features: o Authentication using Active Directory o Metadata Indexing for Ceph Objects, represented as a grid o Tiled selection of data o Data cart
o Cost and difficulty of infrastructure upgrade o Internet connection speed and reliability
FOSS4G 2015 – Seoul, South Korea – September 14th-19th,riculture ▪ Forest ▪ Coastal ▪ Energy ▪ Hydrology
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
LiPAD Web Interface - Tile Selection
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
• File-based storage systems are commonly used to store data • Little setup needed • Pervasive technology • Complexity of directory structure increases with amount of data and processes
Image source: /index.php/about/systemarch2
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - Combined Approach
Working Design -LiPAD
• LiPAD, the customized GeoNode platform, handles the web user interface • GeoServer stores small shapefiles, raster, and vector data • Large files are tiled, named and indexed by the coordinates of each tile and stored
in Ceph • Metadata of tiled files is indexed in LiPAD, represented by a tiled shapefile of the
Philippines
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
• The Philippines is settled along the Pacific Typhoon Belt and the Ring of Fire • It is prone to earthquakes, typhoons, and other hazards • It is abundant in natural resources • There is a need for mapping to assess disaster risk and accounting of natural
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - Combined Approach
• e.g. OpenTopography’s architecture • Comprised of various software and
Working Design
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Working Design - LiDAR Portal for Archiving and Distribution (LiPAD)
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - File Based Storage
LiPAD Web Interface - Data Cart
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Ceph Object Storage System
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
• LiDAR mapping produces high-resolution geospatial data • High resolution data acquisition leads to humongous data sizes • Storage, indexing, retrieval and distribution proves a challenge
hardware resources • Actual data sets are stored as ASPRS
LAS format on a dedicated storage server • Metadata is stored on an IBM DB2 database • Other datasets are stored on the SDSC Cloud platform • Processing and visualization requests are handled by a dedicated large memory, multiprocessor system
o Spatial indexing is on a separate layer o Specialized spatial columns
• Indexing overhead, especially on updates • Scalability and query time issues • Limited support for point cloud data
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
DEVELOPMENT OF DATA ARCHIVING AND DISTRIBUTION SYSTEM FOR THE PHILIPPINES' LIDAR PROGRAM USING OBJECT STORAGE SYSTEMS
Ken Abryl Eleazar Salanio Data Archiving and Distribution Component
2014 Typhoon Tracks
Image sources: https:///wiki/Timeline_of_the_2014_Pacific_typhoon_season
https:///wiki/Ring_of_Fire
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - Combined Approach
• Combines LiDAR flat tiles, RDBMS, and distributed infrastructure o RDBMS manages metadata o LiDAR tiles are stored in dedicated and distributed storage o Data processing is carried out by high-performance compute servers
Introduction – The Philippine Hazard Setting
PHL-LiDAR 1 ▪ Data Acquisition ▪ Data Validation ▪ Data Processing ▪ Training & IEC ▪ Data Archiving ▪ Flood Modeling
PHL-LiDAR 1 kasalanio@.ph
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Outline
• Introduction • Related Work • Working Design • Ceph Object Storage System • Archiving Process Flow • Summary
resources
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Introduction – The Philippine Hazard Setting
• The Philippines’ Department of Science and Technology (DOST) with Higher Education Institutions (HEIs) organized programs for mapping: PHL-LiDAR 1 and PHL-LiDAR 2
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Related Work - GIS-enabled RDBMS
• GIS-enabled Relational Database Management Systems or Geodatabases • Two types of design:
• These programs are an extension of the Disaster Risk and Exposure Assessment for Mitigation (DREAM) LiDAR program
FOSS4G 2015 – Seoul, South Korea – September 14th-19th, 2015
Working Design
• Added features: o Authentication using Active Directory o Metadata Indexing for Ceph Objects, represented as a grid o Tiled selection of data o Data cart