Iowa Environmental Mesonet

IEM Raccoon Released

akrherz@iastate.edu (Daryl Herzmann)

IEM Raccoon generates a Microsoft Powerpoint of storm based warnings and single site NEXRAD imagery for a time period of your choice. The purpose of this app is to provide a quick and easy means for somebody to get an overview of warnings issued during an event and perhaps some imagery that is useful for presentations. It also provides an easy means to investigate the 'storm based' nature of the warnings issued. This idea was based on a technique shown to me by Dan Nietfeld @ NWS Omaha.

The single site NEXRAD imagery backending this application comes from the RIDGE imagery archive, so it is not complete prior to this year. I have filled in a number of large events, but if you do not see NEXRAD imagery in your plots and would really like it, please let me know and I can backfill events without much fuss on my end.

As always, let me know what you think of this and ideas for improvement.

April ASOS Data Uploaded

akrherz@iastate.edu (Daryl Herzmann)

The one minute interval Iowa ASOS data for April has been uploaded to the IEM. Data for Omaha, Sioux Falls, and Moline is available as well. This information is collected monthly from the National Climatic Data Center.

RadMap API Additions

akrherz@iastate.edu (Daryl Herzmann)

The RadMap API generates an image from provided URL parameters (aka a web service). The intent is to allow folks to easily generate time lapses of RADAR information for use in research or presentations.

The archive of GOES and RIDGE imagery is now exposed via this API, so you can have single RADAR overlays or satellite imagery. The archives backending these services continues to be backfilled, so if you find a situation without the RADAR or satellite data you want, please let me know.

Here is an example image of Wichita, KS RADAR and a tornado warning:
http://mesonet.agron.iastate.edu/GIS/radmap.php?vtec=2012.O.NEW.KICT.TO.W.0032&layers[]=sbw&layers[]=uscounties&layers[]=ridge&ridge_radar=ICT&ridge_product=N0Q

Another example of GOES East visible imagery and current convective watches
http://mesonet.agron.iastate.edu/GIS/radmap.php?&layers[]=watches&layers[]=goes&goes_sector=EAST&goes_product=VIS&ts=201204142100&sector=conus

As always, please let me know how this service can be improved.

220 million web requests

akrherz@iastate.edu (Daryl Herzmann)

The IEM webfarm set a website hits milestone on Saturday doing just over 220 million web requests for the day. The good news is that the farm survived the load and didn't have a single known hiccup. The bad news is that the log file from the event is too large for my web statistics package to process (44GB). So the usage stats referenced in this message won't have the data at this time. Previous milestones include:

17 Jun 2001:     1rst hit
25 Jun 2001:       1,000+
19 Sep 2001:      10,000+
19 Mar 2004:     100,000+
25 May 2005:     200,000+
30 May 2006:     500,000+
29 Nov 2006:   1,000,000+
24 Feb 2007:   2,000,000+
17 Oct 2007:   3,000,000+
 5 Feb 2008:   4,000,000+
27 May 2008:   5,000,000+
 5 Jun 2008:   8,000,000+
15 Jul 2009:  10,000,000+
 7 Jan 2010:  15,000,000+
29 Jan 2010:  16,000,000+
 9 Feb 2010:  18,000,000+
23 Apr 2010:  19,000,000+
24 Apr 2010:  22,000,000+
14 May 2010:  23,000,000+
23 Jun 2010:  28,000,000+
23 Jul 2010:  31,000,000+
26 Oct 2010:  34,000,000+
 9 Jan 2011:  39,000,000+
31 Jan 2011:  45,000,000+
 1 Feb 2011:  64,000,000+
26 Apr 2011:  73,000,000+
27 Apr 2011:  78,000,000+
27 Aug 2011:  91,000,000+
19 Feb 2012:  98,000,000+
29 Feb 2012: 106,000,000+
 2 Mar 2012: 178,000,000+
14 Apr 2012: 220,000,000+

Or in plot form with a log-y axis:

March ASOS Data Uploaded

akrherz@iastate.edu (Daryl Herzmann)

The one minute interval Iowa ASOS data for March has been uploaded to the IEM data coffers. Data is also available for Omaha, Sioux Falls, and Moline.

GOES Satellite GIS Update

akrherz@iastate.edu (Daryl Herzmann)

I have made some improvements to the IEM's provided datasets and services of GOES satellite imagery. This imagery is the commonly used imagery for weather surveillance in the United States. These improvements include:

An updated webpage with more details and links to help decipher what is going on with this effort.
All GINI formatted products the IEM receives are now processed and archived. This includes the Hawaii, Alaska, and Puerto Rico regional imagery.
The imagery is available in both "native" AWIPS projection and reprojected EPSG:4326 (simple lat/long).
JSON formatted metadata files are provided along side the raw imagery to help with resolving valid time of the imagery.

As always, enjoy and comments welcome.

At server capacity

akrherz@iastate.edu (Daryl Herzmann)

The IEM web farm is at capacity this fine evening due to all the fun weather. I have to disable the N0R base reflectivity WMS for now to help the web farm survive. Please update to use the Tile Map Service if you can, it has a fighting chance to keep up with the traffic.

Record Warmth Reports

akrherz@iastate.edu (Daryl Herzmann)

I maintain a very simple directory listing of Record Event Reports issued by the National Weather Service. You can browse this directory to see reports of records for sites in and near Iowa.

June COOP Data Uploaded

akrherz@iastate.edu (Daryl Herzmann)

The quality controlled Iowa COOP data for June 2011 has been uploaded to the IEM. These observations are kindly provided by Harry Hillaker, our state climatologist, who passes along these notes on the month:

The month averaged 69.7 degrees or exactly normal while precipitation totaled 6.15 inches or 1.13 inches more than normal. This ranks as the 67th warmest and 29th wettest June among 139 years of records. The normal temperatures broke a string of six consecutive colder than normal months. Precipitation varied widely, ranging from 16.47 inches at Keokuk to only 2.41 inches at Cresco.

The following is the number of new daily records set at COOP sites based on data back to 1951.

                 ____2011__________________________2010 ____
                 JUN  MAY  APR  MAR  FEB  JAN  DEC  NOV  OCT
Maximum High:     75   91  127   15   35    2    2   34   60
Minimum High:    113   69  126    2   75    4   17    0    4
Maximum Low:      81   69   22   15   11    0    0   19    1
Minimum Low:       1   30    5    0   18   13    1    9    1
Maximum Precip:   57   46   51   25   54   42   34   55    4

The following is some bulk statistics on how well IEM's daily COOP data estimator is performing on a monthly basis versus this QC'd dataset. No code changes were made.

                 ___2011____________________________2010_______
                 JUN   MAY   APR   MAR   FEB   JAN   DEC   NOV
High Temp Bias   0.7   0.9   0.9   0.7   0.9   1.0   1.2   0.0
High Temp RMSE   1.3   1.3   1.3   1.3   1.4   1.3   1.5   0.9
Low  Temp Bias   1.3   1.9   1.8   2.4   1.9   1.7   1.5   1.2
Low  Temp RMSE   1.7   2.2   2.1   2.6   2.2   2.0   1.8   1.8
Precip    Bias  -0.23 -0.23 -0.74 -0.25 -0.75 -0.54 -0.52 -0.42
Precip    RMSE   0.80  0.85  0.83  0.44  0.77  0.61  0.58  0.49

ASOS Download Speedup

akrherz@iastate.edu (Daryl Herzmann)

The number one complaint crossing my desk recently is that the ASOS download page is not working. The problem was that the database query to extract the desired data was taking longer than the web server was set to wait for. The page would "time-out" and send an error back to the frustrated user!

I looked into this and implemented a database optimization that should resolve this issue. The request will still take more than 15 seconds, but it should no longer error out after waiting 3 minutes. If you continue to see issues with the page, please let us know.

Internet Outages

akrherz@iastate.edu (Daryl Herzmann)

The Internet has failed for us in the Agronomy building a number of times since 8:53 AM today. We are back again at 9:20 AM, but am not sure if we are stable or not. Sorry for the trouble

Feb ASOS data uploaded

akrherz@iastate.edu (Daryl Herzmann)

The one minute interval Iowa ASOS data for February has been uploaded to the IEM database. This dataset also includes data from Sioux Falls, Moline, and Omaha.

Level II NEXRAD URL Change

akrherz@iastate.edu (Daryl Herzmann)

The IEM provided web directory of Level II NEXRAD data has moved to a new URL. The new URL is http://mesonet- nexrad.agron.iastate.edu/level2/raw/ . The move was necessitated to help improve IEM webfarm stability and speed. The old URLs are being automatically redirected to the new site.

This change was put in place last Friday to help support the load from the big storm, but the DNS was updated this morning.

178 million web requests

akrherz@iastate.edu (Daryl Herzmann)

The IEM web farm attempted to service 178 million requests on Friday thanks to the monster storm. I am not sure this is such a good thing given the trouble the farm had keeping up with the load. It was definitely an educational experience for me trying to figure out how to increase performance. The Web Map Services are definitely the choke point due to the inability to implement caching since the queries coming to the server are adhoc from hundreds of different websites. I am hoping that some of these websites transition to the Tile Map Service, which I have a fighting chance to keep running during high end events.

Here are the previous milestones

17 Jun 2001:     1rst hit
25 Jun 2001:       1,000+
19 Sep 2001:      10,000+
19 Mar 2004:     100,000+
25 May 2005:     200,000+
30 May 2006:     500,000+
29 Nov 2006:   1,000,000+
24 Feb 2007:   2,000,000+
17 Oct 2007:   3,000,000+
 5 Feb 2008:   4,000,000+
27 May 2008:   5,000,000+
 5 Jun 2008:   8,000,000+
15 Jul 2009:  10,000,000+
 7 Jan 2010:  15,000,000+
29 Jan 2010:  16,000,000+
 9 Feb 2010:  18,000,000+
23 Apr 2010:  19,000,000+
24 Apr 2010:  22,000,000+
14 May 2010:  23,000,000+
23 Jun 2010:  28,000,000+
23 Jul 2010:  31,000,000+
26 Oct 2010:  34,000,000+
 9 Jan 2011:  39,000,000+
31 Jan 2011:  45,000,000+
 1 Feb 2011:  64,000,000+
26 Apr 2011:  73,000,000+
27 Apr 2011:  78,000,000+
27 Aug 2011:  91,000,000+
19 Feb 2012:  98,000,000+
29 Feb 2012: 106,000,000+
 2 Mar 2012: 178,000,000+

And the log plot, which has now achieved gravity escape velocity!

At server capacity

akrherz@iastate.edu (Daryl Herzmann)

Sorry folks, out of horsepower to keep up. Gonna have to disable things to get back online. The Web Map Services are the first to go.

106 million hits

akrherz@iastate.edu (Daryl Herzmann)

The IEM website set another personal milestone for web traffic yesterday at just under 106 million hits. This was thanks to the big storm over the eastern US.

Here is a timeline for these milestones:

17 Jun 2001:     1rst hit
25 Jun 2001:       1,000+
19 Sep 2001:      10,000+
19 Mar 2004:     100,000+
25 May 2005:     200,000+
30 May 2006:     500,000+
29 Nov 2006:   1,000,000+
24 Feb 2007:   2,000,000+
17 Oct 2007:   3,000,000+
 5 Feb 2008:   4,000,000+
27 May 2008:   5,000,000+
 5 Jun 2008:   8,000,000+
15 Jul 2009:  10,000,000+
 7 Jan 2010:  15,000,000+
29 Jan 2010:  16,000,000+
 9 Feb 2010:  18,000,000+
23 Apr 2010:  19,000,000+
24 Apr 2010:  22,000,000+
14 May 2010:  23,000,000+
23 Jun 2010:  28,000,000+
23 Jul 2010:  31,000,000+
26 Oct 2010:  34,000,000+
 9 Jan 2011:  39,000,000+
31 Jan 2011:  45,000,000+
 1 Feb 2011:  64,000,000+
26 Apr 2011:  73,000,000+
27 Apr 2011:  78,000,000+
27 Aug 2011:  91,000,000+
19 Feb 2012:  98,000,000+
29 Feb 2012: 106,000,000+

Here is the data in log y-axis format:

Server Outages

akrherz@iastate.edu (Daryl Herzmann)

Had a power circuit blow, attempting to get all the ducks back online and quacking!

May COOP Data Uploaded

akrherz@iastate.edu (Daryl Herzmann)

The quality controlled Iowa COOP data for May 2011 has been uploaded to the IEM. These observations are kindly provided by Harry Hillaker, our state climatologist, who passes along these notes on the month:

May 2011 temperatures averaged 59.1 degrees or 1.0 degrees below normal while precipitation totaled 5.00 inches or 0.44 inch above normal. This ranks as the 52nd coolest and 36th wettest May among 139 years of record. The most noteworthy events in the month were some unusual temperature extremes. Spencer and Sibley reported lows of 20 degrees on the morning of May 3. A lower May temperature has been reported in Iowa only four times in the past 100 years. Just a week later all but six of 133 Iowa reporting points reached or exceeded 90 degrees with Jefferson soaring to 100 degrees. This was Iowa's earliest triple digit reading since April 22, 1980 and the first 100 seen any time in May since 2001. The spring season (March-April-May) averaged 47.0 degrees or 1.3 degrees below normal while precipitation totaled 10.42 inches or 0.20 inch more than normal. This ranks as the 50th coolest and 41st wettest spring among 139 years of records.

The following is the number of new daily records set at COOP sites based on data back to 1951.

                 ____2011_____________________2010 _________
                 MAY  APR  MAR  FEB  JAN  DEC  NOV  OCT  SEP
Maximum High:     91  127   15   35    2    2   34   60   11
Minimum High:     69  126    2   75    4   17    0    4   43
Maximum Low:      69   22   15   11    0    0   19    1   20
Minimum Low:      30    5    0   18   13    1    9    1    1
Maximum Precip:   46   51   25   54   42   34   55    4  106

The following is some bulk statistics on how well IEM's daily COOP data estimator is performing on a monthly basis versus this QC'd dataset. No code changes were made.

                 ___2011_____________________2010_______________
                 MAY   APR   MAR   FEB   JAN   DEC   NOV   OCT
High Temp Bias   0.9   0.9   0.7   0.9   1.0   1.2   0.0   0.1
High Temp RMSE   1.3   1.3   1.3   1.4   1.3   1.5   0.9   1.1
Low  Temp Bias   1.9   1.8   2.4   1.9   1.7   1.5   1.2   0.7
Low  Temp RMSE   2.2   2.1   2.6   2.2   2.0   1.8   1.8   2.0
Precip    Bias  -0.23 -0.74 -0.25 -0.75 -0.54 -0.52 -0.42 -0.05
Precip    RMSE   0.85  0.83  0.44  0.77  0.61  0.58  0.49  0.25

Server Outages

akrherz@iastate.edu (Daryl Herzmann)

The IEM website has been unavailable off and on this Friday morning due to a failure of the main storage array again and capacity problems with the webfarm. Nothing much seems to be going right at the moment, but will get ahead of things sooner than later.

98 million hits

akrherz@iastate.edu (Daryl Herzmann)

The IEM webfarm set another milestone for website traffic servicing just under 99 million hits on Saturday. A vast majority of these hits are coming from automated programs and applications.

17 Jun 2001:    1rst hit
25 Jun 2001:      1,000+
19 Sep 2001:     10,000+
19 Mar 2004:    100,000+
25 May 2005:    200,000+
30 May 2006:    500,000+
29 Nov 2006:  1,000,000+
24 Feb 2007:  2,000,000+
17 Oct 2007:  3,000,000+
 5 Feb 2008:  4,000,000+
27 May 2008:  5,000,000+
 5 Jun 2008:  8,000,000+
15 Jul 2009: 10,000,000+
 7 Jan 2010: 15,000,000+
29 Jan 2010: 16,000,000+
 9 Feb 2010: 18,000,000+
23 Apr 2010: 19,000,000+
24 Apr 2010: 22,000,000+
14 May 2010: 23,000,000+
23 Jun 2010: 28,000,000+
23 Jul 2010: 31,000,000+
26 Oct 2010: 34,000,000+
 9 Jan 2011: 39,000,000+
31 Jan 2011: 45,000,000+
 1 Feb 2011: 64,000,000+
26 Apr 2011: 73,000,000+
27 Apr 2011: 78,000,000+
27 Aug 2011: 91,000,000+
19 Feb 2012: 98,000,000+

Here are those stats plotted on a log plot: