K:\PAPERS\97papers\97 077.wpd

Transcript

1 7.12 IMPROVED REAL-TIME QUALITY CONTROL OF NDBC MEASUREMENTS David B. Gihousen* National Data Buoy Center, Stennis Space Center, MS 39529-6000 Although these checks have done a poorer job of 1. INTRODUCTION detecting errors due to sensor degradation, the modified algorithms are expected to perform much better. Data quality control is not the most interesting or Furthermore, NDBC supplements the real-time checks popular topic at an Americ an Meteorological Society ct validation effort at NDBC. with an extensive, after-the-fa why the session is always conference. That is probably This effort at NDBC is a “manual-machine” mix that relegated to a Friday morning! Nevertheless, most involves a different set of algorithms and a review of operational meteorologists can recall times when an computer graphics. It is typically accomplished within 24 embarrassingly bad observation was transmitted from an hours of observation time. When degraded data are ation to the public. More serious are automatic weather st detected, the analysts update a control file that instructs easonable observations that erroneous, but apparently r the real-time processing not to encode measurements might lead to an incorrect operat ional decision. So, in that from the offending sensor. NDBC is introducing many e “mishaps” becomes very context, preventing thes improvements to the proce ss, which will be documented in important. a future paper. The National Data Buoy Center (NDBC) has an extensive program to r educe greatly the chances EXISTING DATA VALIDATION TECHNIQUES 3. of transmitting degraded measurements. These measurements are taken from its network of buoys and The validation methods used in 1996 will be Coastal-Marine Automated Network (C-MAN) stations presented before the modifi cations are described. The (Meindl and Hamilton, 1992). This paper discusses validation occurs via software running at the National improvements to the real-tim e software that automatically Weather Service (NWS) Telecommunication Gateway effect, these improvements validates the observations. In (NWSTG) that encodes the observations into World codify rules data analysts—who have years of experience Meteorological Organization (WMO) or NWS-approved with our system—use to detect degraded data. codes. Measurements of sea level pressure, air temperature, sea surfac e temperature, dew point CONTEXT OF REAL-TIME QUALITY CONTROL 2. nd direction, wind gust, wave temperature, wind speed, wi Before delving into the details, the proper place of height, average wave period, and dominant wave period a quality control process needs these algorithms in the dat any measurement fails these are validated, if measured. If he last line of defense in to be understood. They form t checks, it will not be releas ed, and measurements from a preventing degraded data from reaching the public. Many backup sensor, if it exists, will be examined. All NDBC other important measures precede them. New sensors are ers; all buoys have two stations have two anemomet s before field evaluations. tested in environmental chamber barometers; and a few buoys have two air temperature When the onboard software is modified, it is regression sensors. tested to make sure the output agrees with previously Several transmission checks are accomplished before accepted values. Measurement s from new sensors, buoy the data are validated. Any message with a single parity lled payloads, are compared hulls, or onboard systems, ca he wave portion of the message error is not transmitted. T with measurements from standard configurations is transmitted in binary at the end of the transmission. If (Giohousen, 1987). All sensors are calibrated before every this message is shorter than expected, contains checksum deployment, and site surveys are conducted to ensure errors, or has an improper synch character, no waves are ometers at new C-MAN proper exposure of the anem encoded. stations. Servicing technicians remain at the station until The simplest of the data checks, the range check, several hours of acceptable transmissions make it through ensures all measurements fall within established upper the satellite. and lower limits. A different se t of limits is used in each of he real-time data validation The historical role of t 29 climatologically similar areas. algorithms was in removing the large, “gross” errors The second check is the time-continuity check. The (Gilhousen, 1988). These errors are typically caused by formula for performing the time-continuity check is: such things as satellite transmission problems, power en cable connections. What system degradation, and brok these algorithms detect is vi rtually certain to be wrong. (1) , 8 0.5 M is the maximum allowable difference, where M is the F standard deviation of each measurement, and is the time J last acceptable observation. difference in hours since the i J is never greater than 3 hour s desp te the actual time Corresponding Author Address : David B. Gilhousen, * di fference. For information on how this formula was Data Systems Division, Building 1100, SSC, MS derived, see National Data Buoy Center (1996). [email protected] 39520-6000; e-mail: dg ific values of the In practice, using station-spec of measured variables is not standard deviation necessary. The general values in use are listed in Table 1. As wi th the general range limits, departing from the general values of F is necessary for some stations. For

2 either the range or If a measurement fails • ures near stations close to example, since water temperat WTMP time-continuity check for two consecutive the Gulf Stream can change abruptly, for several F observations, the measurement is not transmitted east coast stations was increased to 12.1 °C. until it is manually reviewed at NDBC. Four exemptions to the time-continuity test exist. onsecutive 10­ • A check was installed to ensure that c These exemptions are bas ed on the very rapid changes erages, on stations equipped minute wind direction av TABLE 1. Standard deviations used for the with “continuous winds,” agree with the standard 8­ time-continuity check minute average wind direction. More specifically, the 10-minute average wind direct ion that overlaps the Measurement F standard one must agree within 25° of the standard if the wind speeds exceed 2.5 m/s. 21.0 hPa Sea Level Pressure • A procedure was installed to determine if 11.0 °C Air Temperature measurements from duplicat e sensors in reasonable Water Temperature 8.6 °C agreement. If the measurements fail this Wind Speed 25.0 m/s determination, the software transmits the 6.0 m Wave Height measurement from the sens or that exhibits better 31.0 s Dominant Wave Period time continuity. This sensor is then chosen for all subsequent reports until it is manually reviewed. If the Average Wave Period 31.0 s measurements from duplicate sensors are within the Relative Humidity 20.0% tolerances listed in Table 2, they are judged to be in reasonable agreement. This procedure is designed to failures automatically and detect significant sensor that occur in wind, pressure, temperature, and wave height switch to a backup sensor, if one exists and is during the passage of tropical cyclones and severe transmitting reasonable values. extratropical cyclones. First, air pressure measurements check are released if both the that fail the time-continuity Figure 1 is a time-series plot of a wind speed failure. previous and current pressures are less than 1000 hPa. Under the older procedure, almost 24 hours of degraded Second, wind speed measurem ents are released if both data were transmitted befor e data analysts at NDBC the previous and current pressures are less than 995 hPa. detected a wind speed failure and manually switched to ements are released if either Third, air temperature measur the other sensor. With this new procedure, the software s or the wind direction the wind speed exceeds 7 m/ change is greater than 40°. Finally, wave height ed to validate duplicate TABLE 2. Tolerances us measurements the current wind speed is measurements are released if equal to or greater than 15 m/s. Even with these Tolerance Measurement contingencies in place, analysts can elect to disable the range- and time-continuity checks for limited periods Wind Speed 1.5 m/s during hurricanes. Wind Direction 25° provided speed > 2.5 m/s Sea Level 1.0 hPa Finally, internal-consistency checks are: Pressure If the battery voltage is le • ss than 10.5 volts, pressure would automatically transmi t speeds from the second is not released. This precludes the chance of sensor beginning at 1500 UTC, on March 1, 1996. transmitting bad pressures from a failing station easurements transmitted The number of degraded m operating on minimum power. are estimated to fall from approximately 40 to 15 each • The significant wave height, average, and dominant month. Excluding spectral wave measurements, about wave periods are set to zero if the significant wave 800,000 measurements are transmitted each month. height is less than 0. 15 m. Without this, unrepresentatively large wave periods could be transmitted from an essentially flat, “signal-less” VALIDATION CHANGES TO INCREASE DATA 5. spectrum. If the dew point exceeds t he air temperature by less • Several aspects of the real -time quality control system than 1.1 °C, the dew point is set equal to the air were judged overly restrictive. The following improvements point exceeds the air temperature. If the dew will be made resulting in additional data being transmitted: temperature by more than 1.1 °C, the dew point is not encoded. This approach is taken because a reading The consequences of a parity error will be made less • slightly more than 100 percent (Breaker et. al, 1997) inhibiting. Under the new system, only the is normal for the hygrometers used by NDBC. measurement where the e rror is located is not • ean wind speed is greater than If the ratio of gust-to-m transmitted. Previously, the entire message was not 4 or less than 1, neither the wind speed nor gust is observation that never got transmitted. One famous transmitted. problem was from station transmitted because of this 42020 during the formative stages of the March 1993 VALIDATION CHANGES FOR BETTER QUALITY 4. Superstorm (Gilhousen, 1994). • The ability to transmit wind speeds from one CONTROL anemometer and the wind di rections from another was enabled. This circumstance can result from a Several changes will be made in early 1998 that will cracked propeller or worn bearings on one chance of degraded data being help further reduce the anemometer and a faulty compass associated with transmitted:

3 automated validation techniques have become very the other. At any given time in NDBC’s network of 120 important. stations, this situation happens at one or two stations. • The range check will be made less powerful because on a few, rare instances, it caused valid, but extreme, 7. REFERENCES data to be withheld from transmission. More specifically, a measurement has to fail a time- Breaker, L.C., Gilhousen, D.B., and Burroughs, L.D., continuity check for the previous observation before om long-term measurements of 1997: Preliminary results fr e of a range check. it can be deleted becaus atmospheric moisture in the marine boundary layer, • The real-time processing was installed on two UNIX accepted by J. Atmos. Oceanic Technol. workstations running simultaneously. This provides automatic backup capability without the need for Gilhousen, D.B., 1994: The value of NDBC observations rkstation fails or the manual intervention if a wo during March 1993’s “Storm of the Century”, Weather and processing on it crashes. , 255-264. 9 Forecasting, One other improvement concerns the timeliness of the Gilhousen, D.B., 1988: Quality control of meteorological validation. The validation is now done every 5 minutes Fourth Int. data from automated marine stations, Preprints, instead of every 15 minutes. This means that NWS field ation and Processing Systems Conf. on Interactive Inform offices are more likely to receive the observations before , Anaheim, for Meteorology, Oceanography, and Hydrology ic Administration (NOAA) National Oceanic and Atmospher CA, Amer. Meteor. Soc., 248-253. Weather Radio cut-off times. Gilhousen, D.B., 1987: A fiel d evaluation of NDBC moored 6. CONCLUSION 4 , J. Atmos. Oceanic Technol. buoy winds, , 94-104. Many changes were made recently to the real-time Meindl, E.A., and Hamilton, G.D., 1992: Programs of the processing and quality control software for NDBC stations. , 4 Bull. Amer. Meteor. Soc. , National Data Buoy Center, erall amount of data while These changes increase the ov 984-993. decreasing the chance of degraded measurements being transmitted. In recent y ears, the public began directly National Data Buoy Center, 1996: Handbook of automated accessing NDBC observations from several home pages data quality control checks and procedures of the National to verify marine forecasts and make their own marine Data Buoy Center. hese home pages receives more safety decisions. One of t than three million “hits” a m onth! These Internet home om our encoded messages that pages obtain the reports fr described. Therefore, undergo the quality control Figure 1. Time-series plot of wind speed measurements from two anemometers at buoy station 51003 located west of Hawaii

4

Related documents

TP Link   Guide.book

TP Link Guide.book

® ® Diamante and CaseViewNet WiFi Wireless Communication User Guide

More info »
napp it

napp it

napp- it ZFS Storage Server User‘s Guide Setup First steps published: 2018-Dec-20 (c) napp-it.org Licence: CC-BY-SA see http://creativecommons.org/licenses/by-sa/2.0/

More info »
PUG GRB vol4

PUG GRB vol4

Responsible Organization: GOES- R 416- R-PUG -GRB -0348 Vol 4 2.0 Revision R SERIES PRODUCT DEF INITION AND GOES- USERS’ GUIDE (PUG) VOLUME 1: MAIN VOLUME 2: L0 PRODUCTS VOLUME 3: LEVEL 1B PRODUCTS VO...

More info »
UAD 2 Live Rack Manual 180518

UAD 2 Live Rack Manual 180518

UAD‑2 Live Rack MADI Effects Processor Operation Manual Manual Version 180518 www.uaudio.com

More info »
PostScript Language Reference Manual

PostScript Language Reference Manual

See also cross-references: abcdefghijklmnopqrltuvw PLRM 2nd Edition July 27, 1992 ® PostScript Language Reference Manual SECOND EDITION Adobe Systems Incorporated Addison-Wesley Publishing Company, In...

More info »
Good Vibrations: Can a Digital Nudge Reduce Digital Overload?

Good Vibrations: Can a Digital Nudge Reduce Digital Overload?

Good Vibrations: Can a Digital Nudge Reduce Digital Overload? † † Φ † Ψ † , Michael Sobolev , Deborah Estrin , Nicola Dell Fabian Okeke Ψ Φ † Technion, Cornell Tech, Œe Jacobs Institute [fno2;ms3377;n...

More info »
novaPDF   Paperless office solutions

novaPDF Paperless office solutions

novaPDF Paperless office solutions novaPDF User Manual Copyright © 2019 Softland

More info »
matting.dvi

matting.dvi

R © in Foundations and Trends sample Vol. xx, No xx (xxxx) 1–78 c © xxxx xxxxxxxxx DOI: xxxxxx Image and Video Matting: A Survey 2 1 Jue Wang and Michael F. Cohen 1 Adobe Systems Incorporated, 801 Nor...

More info »
p1305 pedreira

p1305 pedreira

Cubrick: Indexing Millions of Records per Second for Interactive Analytics Pedro Pedreira Chris Croswhite Luis Bona Facebook Inc. Facebook Inc. Federal University of Parana 1 Hacker Way 1 Hacker Way C...

More info »
D:PAPERSSTEREOTAXONOMYTAXONOMY.DVI

D:PAPERSSTEREOTAXONOMYTAXONOMY.DVI

A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms Richard Szeliski Daniel Scharstein Dept. of Math and Computer Science Microsoft Research Middlebury College Microsoft Corp...

More info »
Ten Strategies of a World Class Cybersecurity Operations Center

Ten Strategies of a World Class Cybersecurity Operations Center

Bleed rule--remove from file Bleed rule--remove from file MITRE’s accumulated conveys MITRE’s Ten Strategies of a World-Class Cybersecurity Operations Center expertise on enterprise- accumulated exper...

More info »
Introduction   Google Earth User Guide

Introduction Google Earth User Guide

Google Earth User Guide Table of Contents l Introduction Introduction l This user guide describes Google Earth Version 4 and later. m Getting to Know Google Welcome to Google Earth! Once you download ...

More info »
raval mobisys16

raval mobisys16

What You Mark is What Apps See † , Nisarg Raval, Animesh Srivastava, Ali Razeen, Kiron Lebeck Ashwin Machanavajjhala, Landon P. Cox Duke University † University of Washington ABSTRACT formation abound...

More info »
making the most of the madrid system mm forms

making the most of the madrid system mm forms

MAKING THE MOST OF T HE MADRID SYSTEM INFORMATION AND PRACTICAL TIPS ON HOW TO USE SPECIFIC FORMS (Updated July 201 7)

More info »
Controller Tool Help

Controller Tool Help

Controller Tool Help No. LIT-1201 1147 Code 13.0 Release Software 2018 December Issued for the most up-to-date Exchange of this document. Refer to the Knowledge website version 21 Getting ... Started ...

More info »
Computer Vision: Algorithms and Applications

Computer Vision: Algorithms and Applications

Computer Vision: Algorithms and Applications Richard Szeliski September 3, 2010 draft c © 2010 Springer This electronic draft is for non-commercial personal use only, and may not be posted or re-distr...

More info »
D 05 ParameterGuide eng01 W

D 05 ParameterGuide eng01 W

D-05 Parameter Guide Copyright © 2017 ROLAND CORPORATION 01 Contents 18 Key Mode Alteration 2 Basic Operation Basic Operation for Editing Mono Mode, Poly Mode 18 2 Settings Common to All Screens 2 SEQ...

More info »
Interactive Reconstruction of Monte Carlo Image Sequences using a Recurrent Denoising Autoencoder

Interactive Reconstruction of Monte Carlo Image Sequences using a Recurrent Denoising Autoencoder

Interactive Reconstruction of Monte Carlo Image Sequences using a Recurrent Denoising Autoencoder CHAKRAVARTY R. ALLA CHAITANYA, NVIDIA, University of Montreal and McGill University NVIDIA ANTON S. KA...

More info »
Christl Networks  K .indd

Christl Networks K .indd

Wolfie Christl, Sarah Spiekermann Networks of Control

More info »
GSG 5/6 Series GNSS Simulator User Manual

GSG 5/6 Series GNSS Simulator User Manual

GSG-5/6 Series GNSS Simulator User Manual with SCPI Guide Spectracom Part No.: 4031-600-54001 Revision: 26 Date: 16-Jan-2018 spectracom.com

More info »