Here is a revised letter about the weather station by Stephen L. Johnston, Radar Engineer, IEEE Fellow, addressed to Representative Sherwood Boehlert, Chairman, House Science Committee.
- Editor

E-mail: radarslj@ieee.org
February 8, 2001

The Honorable Sherwood Boehlert
Chairman, House Science Committee
2332 Rayburn House Office Building
Washington, DC 20515-4909


Dear Representative Boehlert:

     In Mid December 2000 I sent the first version of this letter to Representative Sensenbrenner and also as an E-mail attachment to Mr. Tom Vanek because of the exigency of the situation. Unfortunately it arrived too late for Representative Sensenbrenner to act on it. Accordingly I am resubmitting my letter to you for your consideration. I have also expanded my letter to include problems with National Weather Service (NWS) flood warnings. I was very pleased that your staff informally sent to the GAO my letter to Representative Sensenbrenner. I was highly impressed by the GAO staff member.

     Please recall my letter of November 15, 2000 to Representative Sensenbrenner. There I expressed concern over the fact that Mr. Donald Burgess of the NWS NEXRAD Operational Support Facility in Norman, Oklahoma had shown that the average tornado Critical Success Index (CSI) for all NWS Weather Forecast Offices (WFOs) was only 0.20 (20%) over the entire period of 1986-1998. Since this period included both pre-NEXRAD and NEXRAD operation, it meant that NEXRAD had not provided any measurable improvement in tornado warning in the nation. This was the first open disclosure of this problem. While that was indeed most disturbing, Mr. Burgess did not pursue this serious matter further before his recent reassignment to the NOAA National Severe Storms Laboratory. Unfortunately I was unable to get the database of Mr. Burgess so I could do further analysis of his results.

     Thanks to the letter from Representative Sensenbrenner to the NWS on this, General Kelly suggested that I contact Mr. William Lerner at the NWS headquarters for NWS tornado data. Mr. Lerner very kindly provided me with tornado warning data from 116 NWS WFOs. That data was surprisingly so compact that he was able to rapidly transmit it to me as an E-mail attachment! I greatly appreciate his full cooperation even though he had been in this work for only two months following the retirement of Mr. Polger. In this letter I will summarize results of our analysis of the new data and make recommendations for further action by your Committee.

     As regards to my methodology, my tornado warning "yardstick" was the CSI. It is the combination of radar Probability of Detection (POD) and False Alarm Rate (FAR) in a well-known simple mathematical equation. I found that the mathematical equation has been known in the NWS for over one hundred years. The first reference to it was in 1884! I sometimes refer to CSI as being the "effective POD" and FAR as being the "Diluter of POD". In that regard at high FAR values, the corresponding CSI value closely approximates the value of (1-FAR), meaning that POD has practically no effect on the CSI value there. We first reduced the number of data sets from 116 to 47 by eliminating the WFOs that had reported no tornadoes in the 14-year period or that had many annual CSI values of zero. This eliminated the Far Western states, New England states, and mid Atlantic states. The eliminated states showed strong correlation to the ranking of states according to Tornado Annual Coverage Fraction (ACF) of Dr. Frank Tatom of VorTeK LLC, Huntsville, Alabama. His top 22 states very closely encompass all the 47 WFOs used in our work. We then ranked the 47 WFO tornado CSI values according to three similar lists, the average of each WFO for the first seven years, the second seven years, and for all 14 years. The two seven year lists approximated dividing the reduced database into two equal portions. We concentrated on the second seven-year list since NEXRAD was installed in the field during that period. The enclosed chart shows the WFO CSI averages values for the last five years in the Great Plain states and east of the Mississippi River. The first part of this letter presents principal findings of our data analysis:

     1. The highest second 7 year period average CSI was only 38%. It was for the Amarillo, Texas WFO. This very low maximum value is much less than the 50% value promised by the NWS during Hearings on the Modernization Plan. Inspection of the database showed that some WFOs reported greater annual values than this BEFORE NEXRAD was installed! Clearly, NEXRAD tornado performance has not lived up to promises.

     2. The region of highest annual CSI values was in the Great Plains states. Inspection of the individual annual CSI histories for those WFOs generally shows little improvement of CSI when NEXRAD was installed. One possible explanation might be due to the nature or tornadoes in the Plains states. I sometimes say facetiously that a blind person can see those tornadoes and a deaf person may hear them at a distance of 100 miles so one does not need radar to detect them! One would think that CSI values for such tornadoes should be much greater without or with NEXRAD. That matter needs further study.

     3. The region of lowest tornado average CSI is east of the Mississippi River. One contributor is certainly the nature of tornadoes in the region. Nature of terrain differences limit low altitude radar coverage. Similarly tornado smaller size restricts required radar distant lateral coverage needed to detect Doppler Tornado Vortex Signatures (TVS). The often-short tornado time on the ground for these tornadoes mandates the most rapid tornado detection and warning. Unfortunately NEXRAD is now locked into a volume scan. Clearly these low values are UNACCEPTABLE! Second 7 year CSI averages of 11 % for Birmingham (that includes Huntsville, Alabama), 10% for Chicago, and only 9% for New Orleans/Baton Rouge are simply pitiful. Strong recommendations will be made on this later in this letter.

     4. Comparison of the second 7-year averages with the first 7-year averages shows that for some 40 %, of them the second seven-year averages were LESS than the first 7 year averages. This means that for them, NEXRAD tornado warning CSI was WORSE than before NEXRAD! This is also unacceptable. Practically all of these were east of the Mississippi River! In the case of the Birmingham, Alabama WFO, before getting its NEXRAD, it did not have a radar of its own. Instead it had only a remote radar monitor from early NWS radar located some 40 miles away at Centerville, Alabama. The Centerville staff analyzed the radar output and coordinated with the Birmingham NWS office. From the data of Mr. Lerner for Birmingham, the first radar lashup actually had better tornado performance than the later NEXRAD!

     5. Inspection of individual annual tornado CSI histories for the second seven years East of the Mississippi River clearly shows sudden very rapid rise in number of warnings issued and number of unverified warnings when NEXRAD was installed. This is because of the NEXRAD policy of warning on EVERYTHING and not to be concerned with false alarms. Is there any wonder why many citizens now ignore NWS tornado warnings? This MUST be corrected.

     6. Although NEXRAD did provide some increase of tornado POD, the excessive NEXRAD FAR values prevented any improvement of CSI. Reduction of NEXRAD tornado FAR would permit NEXRAD CSI improvement. This must be accomplished.

     7. From the above items, the many NEXRAD shortcomings will be recognized. A considerable tornado CSI improvement may be achieved by the following actions. The radar must be a real time radar; i.e., it must not be locked in a volume scan and must be capable of continual antenna azimuth scan at any selected elevation angle or in a vertical antenna scan. It must be capable of being rapidly manually directed to any desired azimuth and elevation angles. The spotter communicator must be seated BESIDE the radar operator instead of in a separate room so the radar operator can quickly investigate spotter reports. Although the NWS will not acknowledge it, the FIRST on line use of Doppler radar techniques in the NWS occurred BEFORE NEXRAD at the Montgomery, Alabama NWS office in 1981, followed by a similar installation in the NWS office in Huntsville, Alabama a decade later. Both Doppler modifications were provided by local government bodies and citizens. Performance of both far exceeded tornado performance of any NEXRAD anywhere in the USA. Unfortunately the Montgomery Doppler was improperly shut down and the excellent radar and Doppler were disposed of. Fortunately the Huntsville Doppler is still in existence although it is now not being properly utilized and its staff is not as skilled as its original staff. This radar is scheduled for disposal and the office permanently closed soon. Clearly, the office should be retained with its Doppler and its warning responsibility should be restored at once.

FLOOD WARNINGS

     While my principal concerns are in tornado warnings, other NWS warnings such as those for floods are of great concern to all residents of the U.S. Radar is the cornerstone of NWS flood warnings under the congressionally approved NWS Modernization Plan. Unfortunately radar derived flood warnings have problems. First, not all radar-detected rain hits the ground. Such rain, called virga by meteorologists, can mislead a radar meteorologist. More importantly, a radar does not directly quantitatively measure either rainfall rate, or total rainfall accumulation. It is necessary to employ an empirical equation that utilizes two parameters. Their values are experimentally derived. These parameters must be changed according to the specific rain conditions present then. Since rainfall conditions are frequently not uniform over an entire radar coverage area it will be appreciated that radar derived rainfall amounts can have considerable in accuracy. They may vary from a 50% under measure to a 200% over measure. Rain gages are the accepted rainfall measurement standard. The NWS modernization Plan originally included rain gages to calibrate NEXRAD rainfall amount measurements, but that was dropped, due to unavailability of funds. Currently NWS is doing limited testing of use of ASOS (Automated Surface Observation Systems) to calibrate NEXRAD rainfall measurements. Unfortunately an ASOS unit provides only one rainfall calibration point for calibration of the entire coverage area of a NEXRAD. As an example in North Alabama there are only some three ASOS units for the entire Tennessee Valley region of North Alabama. The attached paper from Science Magazine entitled "Scarcity of Rain, Stream Gages Threatens Forecasts" discusses the present word wide rain gage situation. Interestingly I have learned that there are several rain gage networks in the U.S. The Department on the Interior U.S. Geological Survey (USGS) has many rain gages around the U.S. as do the TVA and the Corps of Engineers and others. These gage readings are remotely electronically transmitted to central locations. There the data is put on an Internet site that can be accessed by anyone. The USGS rain gage network can be accessed by anyone by going to Internet web URL http://water.usgs.gov/realtime.html and clicking on the desired rain gage to see its readings in real time. Your staff can readily demonstrate that network for you. I would be surprised if any WSO in the U.S monitors them. I have discussed this matter with an Atmospheric scientist, specializing in radar meteorology at the UAH Global Hydrology Climatological Center (GHCC). He said that he could quickly create software that would continually monitor selected rain gages and immediately alert affected NWS WSO. I believe the NWS should immediately implement this concept.

     NWS flood warnings are of great concern to Huntsville citizens. For example, the June 28, 1999 "500 year" flood of Southeast Huntsville especially concerned us. Statements of the Birmingham MIC (Meteorologist In Charge) at the MTC (Modernization Transition Committee) Boise, Idaho meeting indicated, "Birmingham was watching the situation". They did not issue a warning until the Huntsville EMA (Emergency Management Authority) staff who was watching Huntsville area rain gages on the USGS network for two hours called Birmingham and inquired about the situation. To my knowledge, the Birmingham WSO does not monitor these rain gages on the Internet site. Clearly this situation must be corrected immediately by the NWS.

RECOMMENDATIONS

     This letter should be referred to the General Accounting Office (GAO) at once for full study and investigation. I will be glad to assist the GAO in their investigation. The NWS should be directed to retain the Huntsville NWS Doppler in full operation with return of their warning responsibilities.

     May I wish you and your Committee a Happy New Year. I look forward to full success of the Republican Party in both bodies of the Congress and in the White House. I am proud that I voted Republican again.

Sincerely,

Stephen L. Johnston
Radar Engineer
IEEE Fellow

cc Alabama Congressional Delegation