For many moons, the IEM has been processing an archive of one minute interval data from Iowa ASOS (primary airports) sites provided by the National Center for Environmental Information (NCEI, formally NCDC). This news item details an expansion of data processed by the IEM to provide data from all available sites provided by NCEI.
Twice a day, NCEI connects to many of the ASOS weather stations in this country to download its one minute interval data tables. NCEI then collects up this information and provides it in monthly files on their website here. For the lay person and even me, the weather programming nerd, the dataset provided by NCEI is difficult to work with and doesn't really have documentation on what exactly the data format is.
The IEM has made a best attempt effort to ingest this information and provide a simple download of the data found in these files. For some stations, data dates back to the year 2000.
While NCEI updates their available files twice per day, the IEM is just syncing them once every morning (around 8 AM). There is some delay with the most recent observation available for a given site, but generally a days worth of one minute data arrives within the next two days. Rewording, if some big event happens at your local ASOS site and you want to see the one minute data, this service won't have it for a day or two in the future.
I was poking around the Internet on a lazy Saturday morning and noticed that the NOAA MADIS project was now producing High Frequency METAR product. This is a cryptic term to describe observations coming from the primary weather sensors at most airports in the country. We only routinely get hourly data from these sensors, but 5 minute and even 1 minute data is collected but not widely made available. Note: on a once per month basis, the IEM processes a 1 minute interval feed of this information provided by the National Center for Environmental Information (NCEI).
The MADIS project appears to have a connection with the FAA data collection system so as to get a feed of this information! I coded up an ingest of this dataset and it is now augmenting what the IEM provides for ASOS/METAR data.
So this means you will start to see additional observations in the site history and METAR download.
I am also generating "fake" METARs for this data feed to include within the archive. I'd be interested to hear of troubles folks have as they see an increased volume of reports and these auxillary obs. These generated metars contain the string "MADISHF" in the remarks section so to hint where it came from.
Update 9 Jun 2016: So an unfortunate quirk has been found with this dataset and I recently confirmed the quirk with the MADIS Project. The FAA transfers the temperature and dew point data to the NWS in units of whole degree Celsius instead of whole degree Fahrenheit that the ASOS sites actually record the data in. This means that the regular METAR feed, which uses a T-group notation to remove ambiguity when the conversion to Fahrenheit takes place, does not always align well with the 5 minute MADIS data. It also means that sometimes the computed daily extremums can be one degree warmer or cooler than what was actually observed. Oh the fun!
Update 8 Jul 2016: This issue is causing too much confusion on the website. So I have stopped the ingest of the temperature and dewpoint data from this feed. It will simply appear as null values within the various apps. I continue to persue this matter with the FAA and NWS, so hopefully it can be fixed someday.
Update 24 Jul 2017: The issue of temperature precision is known to be a limitation of a MS-DOS based program that runs at the ASOS sites. The ASOS program managers are in the process of updating the software and hope to fix this issue in the process of software upgrades. The completion date is 2023.
The ASOS download interface has been improved to support requesting multiple stations (a long requested feature) and improvements have been made to how time zones are handled. Please let me know of any issues you encounter downloading data or suggestions for improvements you may have.