Search Archive:
526 Views 17 July NWS Data Outage [resolved]


The data flow from the NWS stopped at about 11:30 AM 17 July 2017. No ETA on a fix. This impacts lots of IEM services. Will update the news item once it is repaired.\n

Resolved as of 12:07 PM.

1101 Views 13 Feb NWS Data Outage [Resolved]


The flow of National Weather Service data has been down for the past few hours. This is a nationwide outage of their satellite system, so there are lots of folks in this same boat. Will update this message once it is resolved upstream.

Update 3:30 PM: Data is flowing again and the extent of the outage was from approximately 12 to 3 PM. I suspect some data is lost forever.

595 Views 13 Dec METAR Data Outage

There has been a fire at some location important to the relay of METARs to the world. This relay is down until further notice, so it appears many METAR / Airport / ASOS+AWOS sites will be unavailable in the interim! Will update this NEWS item with any further details I get.

Here's a plot of my monitoring showing the downturn in available sites:

Updated 9 PM: Most sites are back now, but am unsure of when the full restoration will happen. There was a problem with a telco location in Omaha that caused this outage.

445 Views ISU Internet Outage [resolved]


Iowa State University lost Internet access Sunday, 20 Nov 2016, between 6:10 and 8:21 PM CST. An outage of this duration does cause data loss for the IEM project, but I will make an attempt to repair some of the holes caused. No word on why the border routers failed.

960 Views Sept 25th Internet Outages


ISU Network folks will be doing maintenance on Sunday, 25 September 2016. They expect a number of outages during the day as they replace routers, etc. So availability to the IEM will be up and down during the day with actual outages not expected to last for too long.

Update 10 PM 26 September: So firstly, an appology for the prolonged issues that occured. It did not help the situation that I was traveling and could not connect myself to the IEM servers during the outage. Full network service was restored around about noon today. On Sunday, the 1 hour initial outage in the morning was known and expected. What happened after about 11 AM was not. There was some router config issue that was preventing returning traffic from the IEM to reach the clients during this time. When these types of issues happen, I post updates to my @akrherz Twitter account.

1140 Views Network Outage [resolved]


There was an ISU Internet outage this morning that prevented most folks from accessing the IEM website between 5:00 and 5:27 AM this morning (5 April 2016).

1129 Views Network Outage [resolved]


The IEM was unavailable this morning (3 March 2016) between 4:30 and 5:03 AM due to scheduled network maintenance.

1712 Views 17 Jun METAR Outage [resolved]


There was a rather large METAR data outage last night due to issues at the Federal Aviation Administration (FAA). The FAA collects the airport weather station data (in METAR format) and disseminates it to the National Weather Service, from whom the IEM collects the data from. It is doubtful that I'll be able to repair the hole in the archive from this outage.

The outage lasted from just before midnight this morning to about 5:30 AM. Here's a plot from my monitoring showing global METAR station counts with the number of stations reporting within the past hour plotted.

2350 Views 27 April Outage [resolved]


The IEM website was very slow or unavailable for a period between about 3:20 and 4:30 PM on Monday, 27 April 2015. This was due to a cascade failure as a backup database server flooded a file server with IO requests and that slowed down another process that reads data from that server. Oye. The primary database server is about an order of magnitude faster than the backup server, so write loads the primary server generates sometimes slows down the backup server.

I am moving the backup database instance to a different disk system to prevent this from happening in the future. Thanks for your patience.

2496 Views 16 April Outage [resolved]


The IEM website was very slow or unresponsive between 1-2 PM on Thursday, 16 April 2015. I was sitting in a meeting at the time and did not immediately notice that bad things were happening. The issue was a cascading failure of my tilecache service due to a failure to generate N0R RADAR Composites during this time. The lack of current radar data was causing most of the incoming requests to bypass a caching layer and instead hit the mapserver backend, which was quickly overwhelmed with work.

I have made some changes to hopefully prevent this from happening again, thanks for your patience.

3443 Views ISU Network Issues [resolved]


There are unknown issues with ISU's network this morning causing trouble with various IEM services. I will update this news item as I find things out. The most noticable impact was for users being unable to connect to the Level II radar server.

Update 8 PM: It took a while to get everything back on the network after ISU had DHCP server issues over night. We should be back at full strength now. Thanks for your patience.

3518 Views 13 Nov Internet Outage [resolved]


ISU lost Internet access for a number of minutes a bit after 2 PM today. The local IT folks say a soon to be replaced router failed.

2842 Views 10 Oct Internet Outage [resolved]


Internet for the entire ISU campus was out between 2:47 and 3:22 PM today (10 October). No word on what the issue was or if we are stable now.

4:15 PM update A network misconfiguration was made resulting in the campus wide outage. We should be stable now though.

2682 Views Sep 22 Internet Outage [resolved]


Our local network/Internet was down between 10:05 and 10:12 AM this morning. Unsure what is going on other than a local building router is sick.

2370 Views 9/18 RADAR Data Outage [resolved]


Our upstream source of NWS RADAR data is currently off the Internets, so we are having an outage of RADAR products. Have not heard any details on what the issue is other than network outage!

Update 2:30 PM: The upstream source returned at 2:26 PM and I have mostly repaired the missed data during the outage.

1111 Views 17 Aug - Internet Outages


There were two internet outages this morning as ISU continues work on the network backbone upgrades. There is a light at the end of the tunnel and the hope is that the work is mostly done now.

1075 Views 13 Aug - Internet Outage


There was a brief Internet outage between 12:20 and 12:30 PM today as some local network issue occurred. I assume this is related to continued work on local upgrades.

1201 Views 11 Aug - Internet Outage


We lost Internet connectivity around 10:10 AM. It is not clear what is currently broken, but I have hacked around it by hard coding a network routing path. Unsure how stable this config is, but things are working again at the moment (10:45 AM).

1481 Views 7 AM - 9 June Network Outage


A routine ISU network maintenance event this morning for the Agronomy building did not go as hoped and resulted in a prolonged outage between about 6:50 and 7:30 AM this morning.