Using Time-series data from CSU-CHILL (TN-009)
This article describes the use of the time-series extraction utility to read binary time series (unprocessed I/Q) data from the CSU-CHILL Radar.
Time Series Data
The radar can be programmed to record unprocessed data (also called "Time-series data") on a per-sweep basis. These files generally have the extension ".dat", and have a format that stores the network stream data used internally by the software at CHILL.
Extraction Tool
To extract just the data portion from the binary files, a simple extraction tool is available. The source code is available upon request to users of time-series binary files. The program has been tested to compile on Linux machines, and will probably work on any POSIX-compliant system. It is delivered as a tar file. To expand it, give the following:
$ tar zxf extract.tar.gz
This will create a directory called "extract" with the source files and a Makefile.
$ ls acq_svr_defs.h extract.c hsk_svr_defs.h readwrite.c server_types.h chill_types.h gate_data.h Makefile readwrite.h SKU_header.h
To compile, give the following:
$ make
This should give a binary.
$ ls acq_svr_defs.h extract.c hsk_svr_defs.h readwrite.h SKU_header.h chill_types.h extract.o Makefile readwrite.o extract gate_data.h readwrite.c server_types.h
You can then run this binary to get a quick usage string:
$ ./extract
Usage: extract [options] input_file Options: -a<left_limit,right_limit> Sets sector limit to be between left and right in deg ... etc
Specifying an input file will convert that file from CHILL binary to an ASCII data format that may be read with a number of different tools such as MATLAB. The various options provided allow the user to control the format of the ASCII output.
ASCII Format
To generate the ASCII file, give the following:
$ ./extract -o test.txt /path/to/datafile.dat
By default, the program will generate an ASCII file with the name specified by "-o" (if "-o" is not given, use the input file name with extension .txt) with four columns.
$ head test.txt
# CSU-CHILL time series extraction generated by $Id: extract.c,v 1.29 2008/04/08 00:00:01 jpfritz Exp $
# Start gate= 0, End gate= 1019, Start pulse= 0, # pulses= 0, channel= 3
# Housekeeping options:
19 10 3 -3
-8 -3 23 0
9 5 -8 -6
7 -1 6 3
13 11 15 -6
0 -5 -1 -7
-12 10 4 -12
The first and second are the vertically received I and Q data, respectively. The third and fourth are for the horizontal receiver channel. Each range gate is available on a separate line of the file. Any line starting with a "#" is a comment line with no data in it.
In order to separate each transmitter pulse, one can give the -h option, followed by a string indicating which data is needed. For example, -hantx will output the antenna position, sample number, UTC time-of-day and the transmitter state for each radar transmit pulse. This information will be output on comment lines.
$ ./extract -hantx -o test.txt /path/to/datafile.dat
$ head test.txt
# CSU-CHILL time series extraction generated by $Id: extract.c,v 1.29 2008/04/08 00:00:01 jpfritz Exp $
# Start gate= 0, End gate= 1019, Start pulse= 0, # pulses= 0, channel= 3
# Housekeeping options: tnax
# 21 17:41:23 UTC Az: 280.942 El: 0.687 Hpow: 87.99 Vpow: 86.81 Prt: 1042.0 Pol: HV
19 10 3 -3
-8 -3 23 0
9 5 -8 -6
7 -1 6 3
13 11 15 -6
0 -5 -1 -7
The line just before the data indicates the pulse number, time (UTC), azimuth, elevation, transmitter power (H and V), the PRT and the polarization of the pulse (HV = both H and V transmitters fired)
This mode is useful for perusing the available data. However, to write a file that is more suited for input to another program, one can use the "lean" mode by adding "l" to the housekeeping option string:
$ ./extract -hlantx -o test.txt /path/to/datafile.dat
$ head test.txt
# CSU-CHILL time series extraction generated by $Id: extract.c,v 1.29 2008/04/08 00:00:01 jpfritz Exp $
# Start gate= 0, End gate= 1019, Start pulse= 0, # pulses= 0, channel= 3
# Housekeeping options: tnalx
21 63683 280.942 0.687 87.989 86.809 1042.000 HV
19 10 3 -3
-8 -3 23 0
9 5 -8 -6
7 -1 6 3
13 11 15 -6
0 -5 -1 -7
There are no text descriptors, and the UTC time is in seconds since midnight. Also note that there is no "#" output in front of the line. This data is fairly easy to ingest into MATLAB or other analysis tools.
Restricting extraction range
Most often, users will want to restrict the range over which the data is extracted, to simplify the downstream analysis. This is achieved with the extract options -s (start range gate, defaults to 0), -e (end range gate, defaults to maximum in file), -p (starting pulse, defaults to first in the input file), -l (number of pulses, defaults to all pulses in input file).
Output data notes
The range gate numbers are "raw", in the sense that they include all the data acquired by the radar, including a few microseconds before the transmitter fired. The offset in this case is 19 range gates.
Each range gate is 150 meters in length.
Output data is in 16-bit integer format, direct from the digital receiver. To convert these to useful power numbers, please refer to Using calibration files with Time Series data (TN-001).
