AOSN-III Data System Planning Map

 AOSN III Data System ObjectivesEnsure success of ASAPCreate Observational Data (PIs)Make ObservationsSIO GlidersWHOI GlidersNPS AircraftMBARI AUVMIT AUVShip observationsProcess Raw DataProvide QC flags (optional)Make data available to central serverPush to central server: FTP, HTTP, media ...Central server pullsMaintain PI's FTP or HTTP SiteUse other FTP or HTTP SiteSupport data pipeline creationProvide example data so AOSN-III staff c ...Create own data pipeline metadataMove data to central data serverSet up data serverDefine metadataChoose existing standard (Godin)Modify to fit needs (Godin)Agree on data push/pull interfaces/proto ...Write softwareData transferPush (detect new data on server) (AOSN-I ...Pull (implements metadata settings) (AOS ...Versioning (SSDS programmers)Translator (native -> common format) (Go ...QC (simple -- no AI for now) (AOSN-III H ...Monitor data flow (SSDS programmers)System TestingData transfer (e.g. speed) ( AOSN-III Hi ...Volume testing (storage requirements) (A ...Network testing (Nagios: AOSN-III Hire)Archiving (Use existing software TBD)Create test "Sandbox" region on server ( ...Create a mirror/backup (JPL - Peggy?)Create archive repository (JPL? Godin?)Establish data pipeline (fill out data f ...Determine native format for data of each ...Establish metadata for each platform (PI ...Test on "Sandbox"Data conversion programs (see above)Data transfer (e.g. speed) (see above)Volume requirements) (see above)Real-time operationData transferPushInitiate Push (Data Generators)Detect new data (see above)Pull (see above)Data processingVersioning (see above)Conversion to common format (see above)QC (see above)Monitor data flow (see above)Network testing (see above)Data center <-> data originatorsData center <-> modelersData flow monitorFeed data into models for predictions &  ...Establish data pipeline for modelsNeed list of data to be assimilated (JPL ...Decide on what model output should go to ...(see Establish data pipeline, above)Establish distributed model server for c ...Real-time operationFeed data to model (push or pull) (see b ...Move subset model output to central data ...TestingProvide access to observational data and ...Complete MBARI LAS w MOQuA (Godin)QueriesGraphic productsPlatform-specific productsAUV sectionsGlider sectionsMooring time-seriesData extraction (subject to usage constr ...Establish distributed model output serve ...Enable real-time operational decision ma ...Provide state of system informationLast known position of observational ass ...Data flow/work flow status (SSDS query - ...Observation stateCoverage evaluation (Zhang?)Re-construction evaluations (Zhang, Davi ...Model now-cast and forecastHOPS (physics, chemistry, biology) (Harv ...ROMS (physics, chemistry, biology) (JPL? ...Icon (physics) (NRL, or MOQuA query?)Model-data comparisonComparisons with model interpolated to o ...Comparisons w unassimilated observations(Performed at modeling sites?)Adaptive sampling toolsSensitivity analysisETKF (Sharan, Pierre?)ESSE (Pierre?)OSSEs/OSEs (Yi, Pierre?)Path planning/asset allocation (Zhang, B ...Ancillary data (stored on server like ot ...HF Radar (Paduan & Godin)Wave forecasts (SIO -? & Godin)Atmospheric forecasts (COAMPS) (Doyle &  ...Satellite (Dave Foley & Godin)Mooring (M0, M1, M2) (Godin)other ship observations (?)Integrate with other programsCollect Data from 2006 Field Programs (T ...AESOPPLUSLOCO?Operate as node of larger distributed sy ...OOIMMIGCMDServe the public interestIdentify interest groups Identify partnersIdentify useful productsCreate data flow pipeline to create prod ...
hide AOSN III Data System Objectives
hidefull-1Ensure success of ASAP
hidemessagebox_warningCreate Observational Data (PIs)
hideMake Observations
leafSIO Gliders
leafWHOI Gliders
leafNPS Aircraft
leafShip observations
leafProcess Raw Data
leafProvide QC flags (optional)
hideMake data available to central server
leafPush to central server: FTP, HTTP, media, or e-mail
hideCentral server pulls
leafMaintain PI's FTP or HTTP Site
leafUse other FTP or HTTP Site
hideSupport data pipeline creation
leafProvide example data so AOSN-III staff can create pipeline
leafCreate own data pipeline metadata
hideMove data to central data server
hideSet up data server
hideDefine metadata
leafbutton_okChoose existing standard (Godin)
leafbutton_okModify to fit needs (Godin)
leafbutton_okAgree on data push/pull interfaces/protocols (Godin will propose)
hideWrite software
hideData transfer
leafbutton_okPush (detect new data on server) (AOSN-III Hire)
leafbutton_okPull (implements metadata settings) (AOSN-III Hire)
leafmessagebox_warningVersioning (SSDS programmers)
leafbutton_okTranslator (native -> common format) (Godin)
leafbutton_okQC (simple -- no AI for now) (AOSN-III Hire)
leafmessagebox_warningMonitor data flow (SSDS programmers)
hideSystem Testing
leafbutton_okData transfer (e.g. speed) ( AOSN-III Hire)
leafbutton_okVolume testing (storage requirements) (AOSN-III Hire)
leafbutton_okNetwork testing (Nagios: AOSN-III Hire)
leafpencilArchiving (Use existing software TBD)
leafbutton_okCreate test "Sandbox" region on server (Godin & AOSN-III Hire)
leafhelpCreate a mirror/backup (JPL - Peggy?)
leafhelpCreate archive repository (JPL? Godin?)
hideEstablish data pipeline (fill out data flow chart)
leafbutton_okDetermine native format for data of each platform (PI's and/or AOSN-III Hire)
leafbutton_okEstablish metadata for each platform (PI's and/or AOSN-III Hire)
hideTest on "Sandbox"
leafforwardData conversion programs (see above)
leafforwardData transfer (e.g. speed) (see above)
leafforwardVolume requirements) (see above)
hideReal-time operation
hideData transfer
leafmessagebox_warningInitiate Push (Data Generators)
leafforwardDetect new data (see above)
leafforwardPull (see above)
hideData processing
leafforwardVersioning (see above)
leafforwardConversion to common format (see above)
leafforwardQC (see above)
leafforwardMonitor data flow (see above)
hideforwardNetwork testing (see above)
leafData center <-> data originators
leafData center <-> modelers
leafData flow monitor
hideFeed data into models for predictions & return model output
hideEstablish data pipeline for models
leafhelpNeed list of data to be assimilated (JPL, Harvard, NRL?)
leafhelpDecide on what model output should go to central data server (?)
leafforward(see Establish data pipeline, above)
leafhelpEstablish distributed model server for complete access (JPL?)
hideReal-time operation
leafforwardFeed data to model (push or pull) (see below)
leafforwardMove subset model output to central data center (push or pull) (see above)Internal Link
hideProvide access to observational data and models to ASAP PIs
hidebutton_okComplete MBARI LAS w MOQuA (Godin)
leafGraphic products
hidePlatform-specific products
leafAUV sections
leafGlider sections
leafMooring time-series
leafData extraction (subject to usage constraints)
leafhelpEstablish distributed model output server (JPL?)
hideEnable real-time operational decision making
hideProvide state of system information
leafbutton_okLast known position of observational assets (MOQuA query - Godin)
leafmessagebox_warningData flow/work flow status (SSDS query - ?)
hideObservation state
leafhelpCoverage evaluation (Zhang?)
leafhelpRe-construction evaluations (Zhang, Davis?)
hideModel now-cast and forecast
leafhelpHOPS (physics, chemistry, biology) (Harvard?)
leafhelpROMS (physics, chemistry, biology) (JPL?)
leafhelpIcon (physics) (NRL, or MOQuA query?)
hideModel-data comparison
leafComparisons with model interpolated to observation reference frame
leafComparisons w unassimilated observations
leafhelp(Performed at modeling sites?)
hideAdaptive sampling tools
hideSensitivity analysis
leafhelpETKF (Sharan, Pierre?)
leafhelpESSE (Pierre?)
leafhelpOSSEs/OSEs (Yi, Pierre?)
leafhelpPath planning/asset allocation (Zhang, Bellingham)
hideAncillary data (stored on server like other observations)
leafbutton_okHF Radar (Paduan & Godin)
leafbutton_okWave forecasts (SIO -? & Godin)
leafbutton_okAtmospheric forecasts (COAMPS) (Doyle & Godin)
leafbutton_okSatellite (Dave Foley & Godin)
leafbutton_okMooring (M0, M1, M2) (Godin)
leafhelpother ship observations (?)
hidefull-2Integrate with other programs
hidemessagebox_warningCollect Data from 2006 Field Programs (TBD)
hideksmiletrisOperate as node of larger distributed systems (Godin & AOSN-III Hire)
hidefull-3Serve the public interest
hidehelpIdentify interest groups
leaf Identify partners
leafhelpIdentify useful products
leafhelpCreate data flow pipeline to create products