PCI GXL 2016高级技术与二次开发

Simplified System Flow
Data Server
Client Computer (via web browser)
Quality Control Workstations
Input Data

Outpu t Data
Server 1
Controller
X
Server 2

Server n
Processing Server
Start with 1 Server Add a 2nd Server Add servers as needed
• Easily scalable
(System Stays Online)
• Is a service that runs continuously • Monitors the database for new jobs it can run • Jobs are executables and processes that perform the some task
JPS-WS: example request
Request Get’s job # 11132’s current information http://localhost:8080/jps/job/11134 Response <?xml version="1.0" encoding="UTF-8"?> <Job jobId="11134"> <DefinitionRef name="GXLMasterOrtho"> http://localhost:8080/jps/job/def/GXLMasterOrtho </DefinitionRef> <Priority>50.0</Priority> <StartTime>2014-03-04T10:19:28.253-05:00</StartTime> <LastActivity>2014-03-04T10:21:47.050-05:00</LastActivity> <StatusOverview> Ortho: 10JUL27194518-M2AS-P002_ORTHO_MS.pix </StatusOverview> <CurrentStateRef> http://localhost:8080/jps/job/state/current/11134 </CurrentStateRef> <ParamValuesRef> http://localhost:8080/jps/job/param/values/11134 </ParamValuesRef> <JobChildrenRef>http://localhost:8080/jps/job/children/11134</JobChildrenRef> <LogsRef>http://localhost:8080/jps/job/logs/11134</LogsRef> <Comment>Worldview-2: Vancouver</Comment> </Job>
Controller
• Is made up of three sub-components • JPS database – PostgeSQL • Web server – Apache Tomcat • PCI license server • Sub-components can run on separate machines • Tomcat runs GXL/JPS servlets: • jps.war, jpsadmin.war, footprints.war
2
Private and Confidential
Goals
Geospa tial Data
Fast
Robust
Smart
Traceabl e
Geospatial Data
Process geospatial data, notably imagery
– Massive data sets – Complex algorithms
Responsibilities
Runs continuously as server Queries database for jobs to run Launches ready jobs Monitors running jobs Kills jobs Cleans up temporary folders (new in 2014)
JPS-WS - REST
RESTful architecture XML documents: – Hyperlinked – Validates against: jps_1.0.0.xsd Servlets: jps.war Runs on: Java 6 or higher Tomcat 6 or higher
Robust
Set it and forget it Fail fast principle
– Stop as early as possible – Avoid unnecessary processing – User can react quickly
Smart
Smart and dynamic defaults Access via any machine Minimal user interaction
3rd Party Libraries
Hibernate3 – ORM used to map database schema and data to java classes and instance Log4j – logging infrastructure
JPS Web Service (JPS-WS)
JDBC
JPS - Database
High level design
JPS Processing Server JPS Web service (JPS-WS) JPS Database Jobs
What is a Job?
An entry in the JOBS table A process running on a processing server An executable file such as a python script An instance of a Job Definition
Jobs, Proceses, Threads
Every job is run as separate OS process Each job/process monitored by threads
– Waiting for process to terminate – Capture STDOUT – Capture STDERR
JPS-WS: other requests
URL jobs job/{jobid} jobs/defs job/def/ {jobdefname} job/param/defs {jobdefname} job/param/def/ {jobdefname}/ {paramname} job/logs/{jobid} log/{logid} HTTP Method POST GET PUT GET GET Description Create a new job. Get identified {jobid} job’s information. Modify identified {jobid} job’s information. Get the list of job definitions. Get identified {jobdefname} job definition.
GXL Design
Louis Burry, VP Technology & Delivery Kunming China
Agenda
Topic
Goals Architecture JPS Design JPS Processing Server JPS Web Service (JPS-WS) JPS Database JPS Jobs Java APIs Python APIs
Implementation
Requires Java 6 or later jps-ps-2013.jar – implements main processing server logic
– main() is in JPSManager class
jps-db-2013.jar – provides access to JPS-DB jps-common-2013.jar – common functions needed by other two jars
Quality Control Workstations
JPS
JPS Controller
Server 1 Server 2 Server n
JPS Processing Servers
GXL - Components
• Job Processing System (JPS) Processing Servers Controller Geospatial Data Server QA workstations
Geospatial Data Server
• Jobs access the data server • Pull input data for processing • Use temporary local storage for intermediate processing • Write final output data back • Because geospatial data is so large a high speed network should be used between the data server and processing servers
Traceability
Job history Logs for detailed feedback Asynchronous notification
– Notify users via email – No monitoring needed
Components
Client Computer (via web browser) Geospatial Data Server
Spatially Accurate
– Sub-pixel accuracy
Fast
Distributed processing system – Heterogeneous hardware Parallel processing – Multi-threaded – Multiple processes Load balanced State-of-the-art hardware – GPU – OpenMP
GET
GET
Get list of parameter definitions for a job definition.
Get identified parameter {paramname} definition for a job definition. Add a new log message to identified job. Get the list of logs for identified job. Get log entry details for identified {logid} log entry.
Access JPS information
– Jobs, Job Definitions –…
Another means of accessing JPS-DB
– Forwards requests to JPS-DB – Does not match JPS-DB API capabilities
X
Processing Server(s)
High level design
JPS Processing Server
Launches Monitors GXL Python API
Job
HTTP
X M L
JPS Web Service JPS-WS
JPS Java/ORM es Hibernate