Radar data processing with Radx

Introduction

Radx is a very useful tool to process radar data, developed by Mike Dixon at NCAR RAL. You can read about it and obtain the source code here.  I will take you through installing and using some aspects of Radx in this tutorial.

Radx is based on a relatively new radar data format called CfRadial, which is based on Unidata’s netCDF binary format.  It is an attempt at developing a cross-platform common data format standard.  The standard is described here.  There are almost as many radar data formats as there are radar meteorologists (not quite, but close), but this effort led by NCAR is an attempt to finally describe a standard that everyone can use for interchange of scanning, profiling, airborne, and possibly spaceborne radar data.

Installing Radx

LINUX: If you have a modern linux distribution (as on our cluster keeling), you can download a 64-bit binary distribution from NCAR EOL from here, and install according to the README_INSTALL_BIN.TXT in the tar file.  Here is what I did to install on keeling, in a directory called software in my home directory:

wget https://www.eol.ucar.edu/system/files/software/radx/64-bit-binary/radx-20140113.x86_64.tgz
tar xvf radx-20140113.x86_64.tgz 
cd radx-20140113.x86_64
./install_bin_release $HOME/software

OS X: I’m going to detail installation on an OS X 10.8 machine using macports as a package manager. Make sure you have the developer tools installed from the Mac App Store (TM) before installing macports.

The required macports libraries are installed with the following

sudo port install hdf5 netcdf udunits2 expat fftw3 netcdf-cxx

Now to compile Radx, you need to tell the installer where the macports libraries are:

export CXXFLAGS="-I/opt/local/include -I/opt/local/include/udunits2 -L/opt/local/lib"

Now, configure the install from the directory where Radx was untarred, specify an install directory where you have write privileges:

./configure --prefix=/install/radx/here

Now compile it

make

If everything went well, then it will finish without error messages.  If it does crash, then read the error message carefully.  The most common problem I find is that you don’t have a library installed or your path to those libraries is not specified correctly.

To install in the path you specified in the configure step:

make install

That should do it.  The executables will be in the path you specified, in the bin subdirectory.  You may want to add this directory to your path to make it easier to run the executables.  Otherwise you can just type the full path each time.

Converting radar data to CfRadial with RadxConvert

Assuming that you have a collection of radar data volumes in some format other than cfradial (uf, sweep, sigmet raw, NEXRAD level II), this procedure will allow you to convert those files to the cfradial format described here.  I’m using Radx version 20130712.  Since Radx is still in development, your mileage may vary as new versions are released.  Please leave a comment if you notice changes.

I’m going to start with converting some NEXRAD level II files.  You can download some example files here from the Cleveland dual-pol 88D.  After untarring/uncompressing, here’s what I have:

[snesbitt@chuva:~/Downloads/KCLE2012122617]$ ls
KCLE20121226_170504_V06 KCLE20121226_172745_V06 KCLE20121226_175027_V06
KCLE20121226_171045_V06 KCLE20121226_173324_V06 KCLE20121226_175607_V06
KCLE20121226_171625_V06 KCLE20121226_173905_V06
KCLE20121226_172205_V06 KCLE20121226_174447_V06

Before we run anything, you can see the full list of options with Radx by typing

RadxConvert -help

or, you can get a verbose help listing as follows (you may want to redirect this to a file for reference using  the UNIX redirect a.k.a. > file )

RadxConvert -print_params > convert_params

Now, let’s convert our level II files.  I want to preserve the volume structure of the data, but with RadxConvert I have the option of separating each sweep if I’d like.  I also want to use netcdf4 compression (rather than zipping and unzipping the files every time – handy!) — at a level 5 out of 10 balancing cpu time in compression versus disk space (if you compress more, there will be more cpu time used in writing and reading later).  The option -const_ngates to keep the number of gates constant for each sweep.  [Note: Some radars including NEXRAD and some research radars that have varying number of gates in sweeps and you may or may not want to deal with that.  The optional flag -const_ngates will make all rays the same length.  I suggest not doing this for NEXRAD for gridding reasons that will become apparent later.]  I also want to see what I am doing so I am invoking verbose mode (-v).

I’m going to run RadxConvert as follows:

RadxConvert -v -compress_level 5 [-const_ngates] -f KCLE20121226_17* -outdir ./cf

I specified the input files with -f and the output path with -outdir.

This will display what it is doing, what fields it is working on, etc.  Note that you can control output fields, concatenation/disaggregation of volumes, and all kinds of other useful things with the many options you can invoke.

This produced cfradial sweep files in my output directory, with each day in its own directory:

[snesbitt@chuva:~/Downloads/KCLE2012122617]$ ls cf/20121226/
cfrad.20121226_170556.948_to_20121226_171035.306_KCLE_v160_SUR.nc
cfrad.20121226_171137.583_to_20121226_171616.192_KCLE_v161_SUR.nc
cfrad.20121226_171717.556_to_20121226_172155.696_KCLE_v162_SUR.nc
cfrad.20121226_172257.236_to_20121226_172735.414_KCLE_v163_SUR.nc
cfrad.20121226_172837.320_to_20121226_173315.195_KCLE_v164_SUR.nc
cfrad.20121226_173417.057_to_20121226_173855.669_KCLE_v165_SUR.nc
cfrad.20121226_173957.147_to_20121226_174437.343_KCLE_v166_SUR.nc
cfrad.20121226_174539.304_to_20121226_175017.319_KCLE_v167_SUR.nc
cfrad.20121226_175119.199_to_20121226_175557.364_KCLE_v168_SUR.nc
cfrad.20121226_175659.276_to_20121226_180137.179_KCLE_v169_SUR.nc

I can view the contents of each file with a utility like ncview, or I can read them into a program using python, MATLAB, etc.

For example, using ncview:

[snesbitt@chuva:~/Downloads/KCLE2012122617]$ ncview cf/20120226/cfrad.20121226_173417.057_to_20121226_173855.669_KCLE_v165_SUR.nc

Screen Shot 2013-07-16 at 9.58.53 AM

Click on a variable, and an image showing all of the sweeps in ray, range coordinates will be shown (see image at left).  This is a PPI scan, so you can see each of the elevation angles ‘stacked’ on top of each other.

You can use RadxConvert to convert many formats into highly customizable outputs.  The best way to learn about the capabilities is to read the documentation online at NCAR and try (and break) it!  Just be sure you’re original data has a duplicate copy someplace.

Gridding with Radx2Grid

Now, one of the very useful components of the Radx software is the ability to grid data to Cartesian coordinates.  Finally getting the radar community out of using SPRINT and REORDER, Radx offers easy to use yet powerful gridding tools that incorporate the power of the legacy programs.

Radx2grid uses CfRadial as the input format.  So be sure to convert the data first as described above.  Also, if you want to perform any polar coordinate QC on the data, or add additional retrievals, you will want to have code (e.g. in python or MATLAB) that reads and writes in cfradial format files before you grid the data.  We will cover this in another tutorial.

Like RadxConvert, Radx2Grid’s documentation can be accessed from the command line using

Radx2Grid -help

or (dumping to a file, which is preferred)

Radx2Grid -print_params > grid_params

I will use the latter method to edit the configuration for gridding.

Now, I will edit the grid_params file to set up my gridding setup using my favorite text editor.  For the example, I will leave everything at the default settings, except for a couple of changes.

I will change the number of compute threads to 16 (as suggested, this value should be nproc x 4):

n_compute_threads = 16;

For NEXRAD data, you should pay attention to the ‘long’ and ‘short’ ray settings as these data are collected with different pulse repetition frequencies (PRFs) and selecting which ones you use may include more/less aliased data (a.k.a. “purple haze”).  Test which is better for your application.

I will select the REORDER Cressman filter technique:

interp_mode = INTERP_MODE_CART_REORDER;

I will change the radius of influence to 3 km (note that the default is to scale this radius with range as is an option in REORDER, with a selectable normalized radius):

reorder_search_radius_km = 4;

I will require one point per octant:

reorder_only_use_one_point_per_octant = TRUE;

Note that there are two ways to set the vertical geometry of the grid.  You can select specific levels with the z_level_array option, or select the grid_z_geom to set regular spacing.  I will select output every 1 km to 20 km starting at 1 km (leave specify_individual_z_levels = FALSE;):

grid_z_geom = { 20, 1, 1 };

For setting the horizontal characteristics of the output grid, there are many options.  Here I want to use a “flat” projection (with equal 1 km horizontal spacing), from -100 to 100 km in x and y.  Note that there are many options including changing the orientation and origin of the grid to be non-radar centric.

grid_xy_geom = { 201, 201, -100, -100, 1, 1 };

There are also sections to change output fields, and specifying data fields that should be “closest point interpolated” — which are called discrete fields — instead of averaged: for fields like hydrometeor ID, this could be preferred.  Also, you can specify folding limits for velocity and differential phase.

In addition to the default output format (which is a netcdf format), you can write other formats such as CEDRIC for dual-doppler synthesis, MDV for viewing in TITAN, or ZEBRA.

You can also set all sorts of metadata in your output grid files, including your radar name, project name, etc.; see the bottom of the grid_params file for all of the options.

There is one bug in the program, or at least a problem with my system.  The NETCDF4 compression option causes the program to crash.  If you turn off compression, however, it works.  I have to set the following:

netcdf_compressed = FALSE;

and

netcdf_style = CLASSIC;

Finally, I set where I would like the files to be created:

output_dir = "./grids";

Here is my final grid_params file for this exercise.

To run Radx2Grid on our level2 CfRadial files, invoke the command:

Radx2Grid -params grid_params -f cf/20121226/cfrad.20121226_17*

Depending on your machine and what you ask it to do, it may take some time to run. Note that on my quad core machine with HyperThreading, the code is nicely parallelized:

Screen Shot 2013-07-16 at 12.06.42 PM

This will produce the following output:

[snesbitt@chuva:~/Downloads/KCLE2012122617]$ ls grids/20121226/*
grids/20121226/ncf_20121226_171035.nc grids/20121226/ncf_20121226_173855.nc
grids/20121226/ncf_20121226_171616.nc grids/20121226/ncf_20121226_174437.nc
grids/20121226/ncf_20121226_172155.nc grids/20121226/ncf_20121226_175017.nc
grids/20121226/ncf_20121226_172735.nc grids/20121226/ncf_20121226_175557.nc
grids/20121226/ncf_20121226_173315.nc grids/20121226/ncf_20121226_180137.nc

These are the output grids, in this case, in NETCDF3 format.  You may want to consider compressing them for storage, as they can be large depending on how many points are in the file.

To quickly view the output, you can use ncview

[snesbitt@chuva:~/Downloads/KCLE2012122617]$ ncview grids/20121226/ncf_20121226_171035.nc

I selected REF (equivalent radar reflectivity and horizontal polarization) and increased the height to 2 km by clicking on the button to the right of z0 under “current”, which changes the height displayed.  You can see the gridding did a decent job at this low level.

Screen Shot 2013-07-16 at 12.09.10 PM

 

However, if you look aloft, say at 6 km, you can see the dreaded “rings” that are the dread of every radar scientist.

Screen Shot 2013-07-16 at 12.15.04 PM

This is due both to the limited number of elevation scans of NEXRAD precip mode aloft, the shallow nature of the precipitation in this case (this is from a synoptic snow event), as well as your gridding strategy.  You may get better results from research radars that have more scans aloft.  As a radar scientist that has dealt with these problems before, my only advise is this: play around with the gridding types and parameters until you get something that is optimized for your scientific goals and hypotheses.  Look at lots of data, do statistics on the results (this is where problems/artifacts can be identified – i.e. make CFADS of your gridded data), and don’t be afraid to break things.  This is where radar data processing can become as much art as it is science.

Other Radx capabilities

Radx is always improving and adding capabilities, so you should periodically obtain new versions once you’re used to installing it on your system.  There are routines to do rain type partitioning (convective/stratiform), filter data, do EVAD analysis, but I haven’t had time to play with these yet.  As I do, I’ll post more information, but you should try these routines out in the mean time.

Please feel free to share corrections, comments, and suggestions on this page.

 

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>