Parallel computation
Note
Experimental design – not tested operationally!

Parallel computation – using multiple threads

In processing radar data, some computations can be run independently in separate threads. In Rack, parallel computation can be enabled flexibly on command line. Parallel computing is enabled by putting a desired command sequence inside braces: [...] .

Scheme 1: invoking predefined script for every input

Here, a routine to be run for each input is defined as a script, just as explained in Defining scripts . Then, the triggering commands – input file arguments – are put in braces. In the following example, for each input Pseudo CAPPI and Echo Top products are generated converted to Cartesian coordinates and stored in a file.

rack --script '--pCappi 500 -c -o cappi-${NOD}.h5 --pEchoTop 20 -c -o echoTop-${NOD}.h5' [ data/pvol_fi*.h5 <defunct> ]
Definition: DataSelector.cpp:44

Within braces, [–inputFile] <file> command will trigger the defined script in a separate, parallel thread.

More schematically, consider a command line invocation of Rack

rack --cmd --cmd ... --sript '--cmd --cmd ...' [ --CMD1 --cmd2 --CMD3 --CMD4 -cmd5 ... ] --cmd --cmd ...

marking thread-triggering commands with CMD and other commands with cmd. A schematic decomposition of such a line is shown below:

dot_inline_dotgraph_7.png

During execution, Rack creates and maintains certain resources in memory for commands to read and write: a polar volume, a polar product, Cartesian product and compositing arrays. (See the scheme in Introduction .) A thread has private resources – its so called context – in addition to shared resources called the base context. As a principle in threads, any data loaded or created in the thread are used instead of those of the main thread (base context).
Especially, a composite (accumulation array) initialized in the main thread will serve the threads as a shared resource unless (until) a new composite is initialized or loaded in the thread. Commands use internal logic in selecting data.

The following picture illustrates how the above command line structure is actually computed.

dot_inline_dotgraph_8.png

Scheme 2: direct parallel command sequences

Instead of a predefined script, commands can be run parallel by putting them inside braces and separating independent command sequences to threads with a slash '/' .

rack <cmd> [ <cmd> <cmd> ... / <cmd> <cmd> ... / <cmd> <cmd> ... ] <cmd>

Example:

rack volume.h5 [ --cMethod AVERAGE --pCappi 500 --cSize 200 -c -o 'cappi-${NOD}.h5' / --pEchoTop 20 --cSize 500 -c -o 'echoTop-${NOD}.h5' ]

Future extensions

Accumulation

For accumulating polar data, Rack uses a shared accumulation array, enabling parallelizing the process in straightforward way:

rack --script '--pAdd' [ data-acc/2017*.h5 ] --encoding C --pExtract 'dw' -o accumulated-parallel.h5

However, parallel computing does not provide much advantage because actual computing is simple and file read is more a slowing factor.

Parallel compositing

One-pass scheme

The following example creates a composite as a single command line – as a single yet parallelized computation job:

rack --cMethod WAVG,2,3,-32 --cProj '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs' \
--cSize 2000,2000 --cBBox 0,47,40,73 --cInit --script '--pCappi 1500 --cAdd' \
[ data-20140924/201409241200_radar.polar.*.h5 ] \
--cExtract d -o composite-parallel.h5

Tiled scheme

(Under construction.) In operational environment, one typically has system-level parallel processes in receiving radar data, hence there is less use in designing \i parallel tiled compositing in Rack level. See the single-thread Tiled compositing instead.

Anomaly detection

rack volume.h5 [ --aBird '' --aSpeckle '' ] --aRemover 0.5 -o vol-anom-paral.h5

Product generation

Given a single volume as input, generate products to separate /dataset groups:

rack volume.h5 --append dataset [ --pCappi 500 --pEchoTop 20dBZ ] -o vol-prod-paral.h5

Anomaly detection and product generation combined

As anomaly detectors are independent of each other, they can be computed in parallel, each updating the resulting quality field (QIND quantity) when completed. Finally, the combined detection field can be used for removing the anomalies in the volume.

rack volume.h5 [ --aBird '' --aSpeckle '' ] --aRemover 0.5 --append dataset [ --pCappi 500 --pEchoTop 20dBZ ] -o vol-anom-prod-paral.h5