 |
|
|
|
This section explains AnDRe, the set of anomaly detection and removal functions of Rack. A quick reference of all the commands is found in Anomaly detection and removal.
Practically, echoes received by a weather radar originate not only from precipitating clouds but also from insects, birds, aircraft, ships, ground, sea clutter and electromagnetic emitters illustrated in the figure below.
These external factors – or anomalies from the meteorological point of view – affect radar data in various ways. Some of the anomalies appear as distinct, localized peaks, some form geometrical patterns and some affect the overall data. Examples of effects are shown in the image set below.
In AnDRe, the processing is divided into two stages: detecting anomalies and removing the anomalies from input data. In principle, these stages modify ODIM data independently so a user may combine Rack with other software performing anomaly detection (see Reading and combining quality data). The independence also means that anomalies can be detected using parallel computation in future versions of Rack (see Anomaly detection).
The following code is an example of applying both detection and removal in Rack :
By default, AnDRe commands apply to the full volume input. Each operator will look for quantities (typically DBZH) it needs as input. To decrease computation time, the processing can be limited using –select command, for example:
For details, see Selecting data .
The design of AnDRe detectors is based on Ropo , the operational anomaly detection C program applied in FMI since 2001.
Each detector has been fully rewritten as a respective C++ class (rack::BiometOp, rack::EmitterOp, rack::ShipOp, rack::SpeckleOp) which internally applies functions of the Drain library, for example segment size analysis, moving window filters and statistics. In development, emphasis has been on code re-usability, speed and stability; see implementation details further below.
Rack provides several detectors that compute detection fields ie. probablity images, explained next. The removal procedures are explained after that.
Currently, AnDRe supports detection the following anomalies:
Anomaly detection and removal start with '–a', and can be listed with
Detects birds and insects. The idea behind bird and insect detector hence "biometeor" detector is very simple. It looks for intensities lower than threshold reflMax and altitudes lower than altitudeMax (metres). The thresholds have fuzzy steepness parameters: intensity steepness reflDev and altitude steepness altitudeDev. The steepness parameters determine "half-widths" of the fuzzy responses (Fuzzy functions ).
--aBiomet .There is also detectors for birds and insects using dual-pol data.
--aBird .
--aInsect .Based on pre-computed clutter map, scales the clutter probability for desired sweeps. The scaling is a heuristic. The loaded map should contain quantity CLUTTER which in each bin represents the probability of contamination. Notice that the quantity does not describe the intensity of contamination, yet might be computed from accumulated dBZ values as described in cluttermaps .
: "sky angle",
--aClutterMap .Faster yet less accurate bird detection is available as aDopplerNoise which computes standard deviation much in the same way the corresponding meteorological motion product Doppler wind variation . It takes three parameters: speed threshold (m/s) as well as window width and height. The obtained deviation is converted to a detection probability by means of a fuzzy bell function.
--aDopplerNoise .AnDRe emitter detector takes four parameters: lengthMin (along the beam, kilometres), widthMax (azimuthal degrees) and sensitivity (between 0.0 and 1.0). The current implementation applies median filtering twice on the data: horizontally and vertically. Then, the vertical result is subtracted from the horizontal, revealing the horizontally elongated line segments. The computed medians are biased; instead of the conventional median position (at 0.5 of the cumulative distribution) one may lighten or darken the results with the sensitivity value. A high value darkens the vertical result and lightens the horizontal result, causing more line segments in the resulting detection field.
--aEmitter .In presense of intensive nearby transmitters or receiver malfunction radar data can be contaminated by intensive overall noise. Rack models the shape of such intensity as a function of distance and applies curve fitting to distinguish between smoothly matching noise areas and pronounced meteorological targets.
--aJamming .
--aJamming .With the polarimetric quantity RhoHV one can easily detect some non-meteorological echoes such as sea clutter. For meteorological targets, RhoHV is close to 1.0 and below otherwise. For example, sea clutter can be detected with the following command:
--aNonMet .The ship detector takes four parameters: minDBZ (minimum reflectance), devDBZ (minimum reflectance difference in a neighborhood), windowWidth (neighbourhood's beam-directional width, metres), windowHeight (neighbourhood's azimuthal width, degrees). First, the detector function computes two separate fields: a fuzzy remapping of minDBZ and field resulting from high-pass filtering with given neighbourhood. Basically, the result is a fuzzy multiplication of those.
In addition, artificial vertical "ship sidelobes" are computed for the latter field, modelling the radar echoes of highly reflective point targets. If the sidelobes also appear in the fuzzy intensity field, they become included in the final result.
--aShip .Speckle detection needs two parameters: threshold (in dBZ) and area (fuzzy maximal speck size). This detector focuses on distinct, small specks, giving maximum value to single-pixel segments, and progressively smaller values for larger segments. The speckle detector applies a semi-recursive flood-fill type function in computing segment sizes: the image is traversed and each segment having intensity of at least threshold becomes analyzed. The size of each segment fuzzily converted to an intensity of equally shaped segment in the result image.
--aSpeckle .The speck areas are optionally scaled to Cartesian coordinate system. Unscaled areas are better for detecting for non-physical, range independent anomalies like noise whereas scaled mode should be used if physical targets are focused.
--aSpeckle without and with polar coordinate inversion.This detector calculates the position of the Sun and marks its with sector of given width and intensity. The detector does not analyse the data itself; sectors will be marked also inside precipitation.
--aSun .Typically, user wants to detect several different types of anomalies. In the ODIM structure, Rack creates new data arrays for storing detection results:
/dataset??/data??/quality1, with what:quantity=QIND –store INTERMEDIATE has been called, all the intermediate detection results /dataset??/data??/quality{N} where {N} is an index starting from 2, with what:quantity=PROB-xxx (non-standard)Universality. Some detectors like –aSpeckle are universal , they apply not only for the quantity they have been computed on (like DBZH) but also other quantities (like VRAD). The resulting quality information is stored respectively to dataset level or data level. This policy can be toggled with –aUniversal .
The resulting cumulative field with what:quantity=QIND is inverted, reflecting the probability of meteorological echoes, ie. quality of the data. In parallel, the classification field with what:quantity=CLASS is updated. The fields are updated automatically after every detection command. They reside on the dataset level or data level, depending on detector universality described above.
In the removal stage, AnDRe combines the QIND and CLASS fields automatically. One may also combine the fields with –aCombine .
After applying preferred detectors, original data in the volume can be enhanced with the help of the overall quality field, /dataset1/quality/data .
There are three techniques available for the removal:
nodata The –aRemover command simply replaces data values with nodata in locations where the quality (anomaly probability) is lower than a threshold:
By default, aRemover applies to practically all quantities (DBZH,TH,VRAD,RHOHV,...). This behaviour can be changed with –select command.
--aRemover with threshold 0.25 (left) and 0.75 (right).A smooth erasure function is implemented as –aDamper which decreases intensity proportionally to the quality index. It applies to reflectivity data (DBZH) only. The lower the quality index, the more intensity will be decreased. An example of the erasing approach:
The parameter defines the threshold quality. The bin values below the quality threshold will be smoothly erased.
--aDamper with threshold 0.25 (left) and 0.5 (right).Gap filling means replacing single values with neighboring, more reliable values. An example of the gap filling approach:
The parameters of aGapFill are window width and height as metres and azimuthal degrees, respectively. Internally, this operator is implemented using 8-directional distance transform.
Another, experimental gap filling operator –aGapFillRec is recursive and produces spline-like correction fields. The result is smoother but slower to compute. Internally, this operator is implemented by repeating a weighted averaging window.
The removal command can be applied together, for example:
Detection based quality fields like the ones explained above describe the probability of signals originating from other than desired targets (other than precipitation, typically). In addition to those, Rack provides two operators that describe the quality.
Radar signal attenuates in heavy rain. The amount of attenuation in liquid rain can be approximated from dBZ data using formula proposed by Ulrich Blahak:
![\[ k(Z) = 1.12\cdot10^{-4} Z_e^{0.62} \]](form_0.png)
--aAttenuation .
--aAttenuation .Note that this field should NOT be used for removing echoes; the radar has already cleaned data in signal processor. However, this operator should be computed after removal operators (if applied). The resulting (total) quality field can be used in further production, like quality based compositing in which input bins of lower quality will be overridden by other, overlapping bins of higher quality.
If both the total reflectivity (TH ) and Doppler filtered reflectivity are available, their difference provides information where the radar has detected static i.e. suspicious targets. The CCor operator computes the diffence between these and fuzzifies the result such that at the given threshold reflecitivy difference the probability of anomaly is 0.5.
--aCCor .Note that this field should NOT be used for removing echoes; the radar has already cleaned data in signal processor. However, this operator should be computed after removal operators (if applied). The resulting (total) quality field can be used in further production, like quality based compositing in which input bins of lower quality will be overridden by other, overlapping bins of higher quality.
Likewise, we can associate timing based quality field that describes the extent in which each measurement bin is representative of the nomimal time of the volume or some other desired time, like the nominal time of a composite.
--aTime .
1.9.1