Extracting metadata

Simple dump of metadata

The basic way to view context in textual format is to write a file with '.txt' extension:

Using –outputFile , the structure of the current data can be written as plain text in a file (*.txt) or to standard input (-):

rack volume.h5 -o volume.txt

rack

Definition: DataSelector.cpp:44

File volume.txt will consist of lines of ODIM entries as follows:

dataset1
dataset1/data1
dataset1/data1/data
dataset1/data1/data:image=[500,360]
dataset1/data1/what
...
    ...
        ...
how:simulated="False"
how:software="IRIS"
how:sw_version="8.13"
how:system="VAISWRM200"
how:wavelength=5.33

Formatting metadata output using templates

There is often need to produce metadata text for:

Text files completing data formats not supporting metadata
Log entries
HTML files applied in monitoring product generation
Graphical layouts (eg. SVG)
Geographic definition files (eg. KML) for adding radar products to GIS applications

One can easily export metadata of current HDF5 structure (latest volume read or product generated). The output is a text file in which variables ${variable} have been expanded to their current values.

The text template can be defined briefly on command line by –format or by reading a file with –formatFile .

For example, from a Cartesian product one may extract the projection and a bounding box with the following command:

rack volume.h5 -c --format 'BBOX=${where:LL_lon},${where:LL_lat},${where:UR_lon},${where:UR_lat}\n' -o formatOut.txt

The result:

BBOX=21.4734,62.4469,32.0369,66.9151

The applicable variables are those the ODIM variables (what:product, where:lat, ...). In addition, what:source fields (NOD, WMO , RAD , PLC , CMT ) are searched for and expanded to convenience variables (${NOD}, etc...). The list of current variables can be examined with –status command.

For example, assume template file radar-conf.tpl with the following contents:

# Metadata for radar '${NOD}'
 
SOURCE="${what:source}"
NOD="${NOD}"
WMO="${WMO}"
PLC="${PLC}"
LAT="${where:lat}"
LON="${where:lon}"
HEIGHT="${where:height}"
PROJDEF="${where:projdef}"
BBOX="${where:LL_lon},${where:LL_lat},${where:UR_lon},${where:UR_lat}"

Then,

rack volume.h5 -c --formatFile radar-conf.tpl --formatOut out.cnf

produces code which can be used by applications supporting Proj4 projection syntax. The result:

# Metadata for radar 'fiuta'
 
SOURCE="WMO:02870,RAD:FI47,PLC:Utajärvi,NOD:fiuta"
NOD="fiuta"
WMO="02870"
PLC="Utajärvi"
LAT="64.7749"
LON="26.3189"
HEIGHT="118"
PROJDEF="+proj=aeqd +lon_0=26.3189 +lat_0=64.7749 +ellps=WGS84"
BBOX="21.4734,62.4469,32.0369,66.9151"

Further, using –script one can quickly create similar conf files with a single command:

rack --formatFile radar-conf.tpl --script '--cReset -c --formatOut site-${NOD}.cnf' <volumes.h5>

New. Latest versions of Rack support variable formatting with following options

bash style substring extraction: :startpos or :startpos:length (as with bash shell variables)
C printf() styled value formatting, like %4.2f or %8d (see https://cplusplus.com/reference/cstdio/printf/ )
date style date and time formatting, like Y/m/d or H:M – for variables with names ending with "date" or "time".

This options are illustrated with examples:

#

rack volume.h5 --format 'time: ${what:startdate|%Y-%m-%d} ${what:starttime|%H:%M} elangle: ${where:elangle|%04.1f} \n' --select dataset:/ -o elangles.out

time: 2014-08-27 09:00 elangle: 00.3 
time: 2014-08-27 09:00 elangle: 00.7 
time: 2014-08-27 09:00 elangle: 01.5 
time: 2014-08-27 09:01 elangle: 03.0 
time: 2014-08-27 09:01 elangle: 05.0 
time: 2014-08-27 09:01 elangle: 09.0 
time: 2014-08-27 09:02 elangle: 02.0 
time: 2014-08-27 09:02 elangle: 07.0 
time: 2014-08-27 09:02 elangle: 11.0 
time: 2014-08-27 09:03 elangle: 15.0 
time: 2014-08-27 09:03 elangle: 25.0 
time: 2014-08-27 09:03 elangle: 45.0 
time: 2014-08-27 09:04 elangle: 00.4 

Sampling data

The input volume as well as the generated polar or Cartesian products can be sampled, producing a text file. The sampling is performed automatically if output file has the extension '.dat'. For example:

rack volume.h5 -o samples.dat

The result:

## input properties 
# bbox=[1507.9672560563260504,3892.9862409188299353,1507.9629643527550797,3892.9862409202742128]
# height=360
# lat=64.77493012323975563
# lon=26.318880077451467514
# width=500
## sampling parameters 
# commentChar="#"
# handleVoid="null"
# i=[-1,1]
# iStep=10
# j=[-1,1]
# jStep=0
# skipVoid=false
# format='${i},${j},${LON},${LAT},${X},${Y},${RANGE},${AZM},${HGHT}'
## resulting geometry 
# rows=50
# rols=36
# samples=1800
5,5,1508.25,3712.74,239.678,2739.54,2750,5,14.844 #void
25,5,1509.31,3717.86,1111.24,12701.5,12750,5,76.3267
65,5,1511.44,3728.09,2854.35,32625.4,32750,5,234.606
75,5,1511.97,3730.65,3290.13,37606.3,37750,5,281.533
85,5,1512.51,3733.21,3725.91,42587.3,42750,5,331.402
95,5,1513.05,3735.77,4161.69,47568.3,47750,5,384.214
...
    ...
        ...
455,355,1482.1,3827.87,-19849.7,226883,227750,355,4244.53
465,355,1481.48,3830.43,-20285.5,231864,232750,355,4406.15
475,355,1480.86,3832.99,-20721.3,236845,237750,355,4570.71
485,355,1480.24,3835.55,-21157.1,241826,242750,355,4738.21
495,355,1479.62,3838.1,-21592.8,246807,247750,355,4908.65

The default sampling step is 10 in both horizontal and vertical direction, and all the quantities are sampled. One may apply –select command to get desired subset of quantities. Further, –format modifies output layout; variables inserted like ${NAME} are converted to actual values in output. (See format in the file header above).

i and j : image coordinates, from top-left corner (bin and azimuth index in polar sampling);
j2: inverted vertical image coordinate ie. from bottom-left corner (in Cartesian sampling only)
LON and LAT : geographical coordinates, in degreees
X and Y : map coordinates, in metres
AZM and RANGE: polar coordinates (in polar sampling only)
the quantities, eg. DBZH , VRAD , and so on, including quality quantities like QIND. The default format consists of all the applicable coordinates and quantities separated by space. Values can be negated by putting a minus sign inside the curly braces of a format string.

Sampling steps, start, and end can be changed with –sample command. One may also skip lines containing undetect or nodata samples. If a comment char is defined some metadata, including applied format, appears as comments in the start of the output.

--sample  <iStep>,<jStep>,<i>,<j>,<commentChar>,<skipVoid>,<handleVoid>  (section: general)
  Extract samples. See --format.
    iStep=10 [horz coord step]
    jStep=0 [vert coord step]
    i=-1:1 [horz index or range]
    j=-1:1 [vert index or range]
    commentChar=# [comment prefix (char or bytevalue)]
    skipVoid=false [skip lines with invalid/missing values]
    handleVoid=null [skip or mark invalid values [skip|null|<number>]]

Example of a formatted sampling:

rack volume.h5 --select quantity=DBZH --sample 10,30,i=50:200,j=0:50 --format '${i},${j} (${LON},${LAT}) ${DBZH}' -o samples-formatted.dat

The result:

## input properties 
# bbox=[1507.9672560563260504,3892.9862409188299353,1507.9629643527550797,3892.9862409202742128]
# height=360
# lat=64.77493012323975563
# lon=26.318880077451467514
# width=500
## sampling parameters 
# commentChar="#"
# handleVoid="null"
# i=[50,200]
# iStep=10
# j=[0,50]
# jStep=30
# skipVoid=false
# format='${i},${j} (${LON},${LAT}) ${DBZH}'
## resulting geometry 
# rows=50
# rols=12
# samples=600
50,0 (1507.96,3724.3) -13.99
90,0 (1507.96,3734.58) -10.79
60,30 (1526.34,3724.77) -11.71
70,30 (1529.41,3726.99) -5.82
80,30 (1532.48,3729.21) -10.65
...
    ...
        ...
160,30 (1557.43,3746.9) null #void
170,30 (1560.59,3749.11) null #void
180,30 (1563.76,3751.31) 6.74
190,30 (1566.93,3753.52) 19.96
200,30 (1570.12,3755.72) null #void

Likewise, one can sample Cartesian data. Example:

rack volume.h5 --select quantity=DBZH --cSize 500,500 -c --sample 50,50 -o samples-cart.dat

The result:

## input properties 
# bbox=[21.473369970467263812,62.446873711034854182,32.036889576527528334,66.915064520295331363]
# epsg=0
# height=500
# proj="+proj=aeqd +lon_0=26.3189 +lat_0=64.7749 +ellps=WGS84"
# width=500
## sampling parameters 
# commentChar="#"
# handleVoid="null"
# i=[-1,1]
# iStep=50
# j=[-1,1]
# jStep=50
# skipVoid=false
# format='${i},${j},${LON},${LAT},${X},${Y},${j2}'
## resulting geometry 
# rows=10
# rols=10
# samples=100
25,25,21.2287,66.7069,-224500,224500,474 #void
225,25,25.7622,66.7873,-24500,224500,474
275,25,26.8983,66.7872,25500,224500,474
225,75,25.7721,66.339,-24500,174500,424
175,125,24.6855,65.8829,-74500,124500,374
225,125,25.7816,65.8906,-24500,124500,374
...
    ...
        ...
275,475,26.8178,62.7511,25500,-225500,24 #void
325,475,27.7958,62.7441,75500,-225500,24 #void
375,475,28.773,62.7303,125500,-225500,24 #void
425,475,29.7492,62.7096,175500,-225500,24 #void
475,475,30.7237,62.682,225500,-225500,24 #void

Note that –cCreate maps a single quantity to Cartesian coordinates; the quantity can be changed with –select .

Does a radar overlap with a given geographical scope?

Table of Contents

Extracting metadata

Simple dump of metadata

Formatting metadata output using templates

Sampling data