Search Archive:
398 Views 17 July NWS Data Outage [resolved]

Link: https://mesonet.agron.iastate.edu/

The data flow from the NWS stopped at about 11:30 AM 17 July 2017. No ETA on a fix. This impacts lots of IEM services. Will update the news item once it is repaired.\n

Resolved as of 12:07 PM.

802 Views 13 Feb NWS Data Outage [Resolved]

Link: https://www.washingtonpost.com/news/capital-weather-gang/wp/2017/02/13/weather-service-suffers-catastrophic-outage-website-stops-sending-forecasts-warnings

The flow of National Weather Service data has been down for the past few hours. This is a nationwide outage of their satellite system, so there are lots of folks in this same boat. Will update this message once it is resolved upstream.

Update 3:30 PM: Data is flowing again and the extent of the outage was from approximately 12 to 3 PM. I suspect some data is lost forever.

536 Views 13 Dec METAR Data Outage

There has been a fire at some location important to the relay of METARs to the world. This relay is down until further notice, so it appears many METAR / Airport / ASOS+AWOS sites will be unavailable in the interim! Will update this NEWS item with any further details I get.

Here's a plot of my monitoring showing the downturn in available sites:

Updated 9 PM: Most sites are back now, but am unsure of when the full restoration will happen. There was a problem with a telco location in Omaha that caused this outage.

392 Views ISU Internet Outage [resolved]

Link: https://www.it.iastate.edu/cel/view/2242

Iowa State University lost Internet access Sunday, 20 Nov 2016, between 6:10 and 8:21 PM CST. An outage of this duration does cause data loss for the IEM project, but I will make an attempt to repair some of the holes caused. No word on why the border routers failed.

902 Views Sept 25th Internet Outages

Link: http://www.inside.iastate.edu/article/2016/09/15/upgrades

ISU Network folks will be doing maintenance on Sunday, 25 September 2016. They expect a number of outages during the day as they replace routers, etc. So availability to the IEM will be up and down during the day with actual outages not expected to last for too long.

Update 10 PM 26 September: So firstly, an appology for the prolonged issues that occured. It did not help the situation that I was traveling and could not connect myself to the IEM servers during the outage. Full network service was restored around about noon today. On Sunday, the 1 hour initial outage in the morning was known and expected. What happened after about 11 AM was not. There was some router config issue that was preventing returning traffic from the IEM to reach the clients during this time. When these types of issues happen, I post updates to my @akrherz Twitter account.

1096 Views Network Outage [resolved]

Link: https://www.it.iastate.edu/cel/view/2141

There was an ISU Internet outage this morning that prevented most folks from accessing the IEM website between 5:00 and 5:27 AM this morning (5 April 2016).

1113 Views Network Outage [resolved]

Link: https://www.it.iastate.edu/cel/view/2115

The IEM was unavailable this morning (3 March 2016) between 4:30 and 5:03 AM due to scheduled network maintenance.

1690 Views 17 Jun METAR Outage [resolved]

Link: http://mesonet.agron.iastate.edu/ASOS/current.phtml

There was a rather large METAR data outage last night due to issues at the Federal Aviation Administration (FAA). The FAA collects the airport weather station data (in METAR format) and disseminates it to the National Weather Service, from whom the IEM collects the data from. It is doubtful that I'll be able to repair the hole in the archive from this outage.

The outage lasted from just before midnight this morning to about 5:30 AM. Here's a plot from my monitoring showing global METAR station counts with the number of stations reporting within the past hour plotted.

2345 Views 27 April Outage [resolved]

Link: http://mesonet.agron.iastate.edu/

The IEM website was very slow or unavailable for a period between about 3:20 and 4:30 PM on Monday, 27 April 2015. This was due to a cascade failure as a backup database server flooded a file server with IO requests and that slowed down another process that reads data from that server. Oye. The primary database server is about an order of magnitude faster than the backup server, so write loads the primary server generates sometimes slows down the backup server.

I am moving the backup database instance to a different disk system to prevent this from happening in the future. Thanks for your patience.

2492 Views 16 April Outage [resolved]

Link: http://mesonet.agron.iastate.edu/

The IEM website was very slow or unresponsive between 1-2 PM on Thursday, 16 April 2015. I was sitting in a meeting at the time and did not immediately notice that bad things were happening. The issue was a cascading failure of my tilecache service due to a failure to generate N0R RADAR Composites during this time. The lack of current radar data was causing most of the incoming requests to bypass a caching layer and instead hit the mapserver backend, which was quickly overwhelmed with work.

I have made some changes to hopefully prevent this from happening again, thanks for your patience.

3433 Views ISU Network Issues [resolved]

Link: https://www.it.iastate.edu/cel/view/1934

There are unknown issues with ISU's network this morning causing trouble with various IEM services. I will update this news item as I find things out. The most noticable impact was for users being unable to connect to the Level II radar server.

Update 8 PM: It took a while to get everything back on the network after ISU had DHCP server issues over night. We should be back at full strength now. Thanks for your patience.

3507 Views 13 Nov Internet Outage [resolved]

Link: https://www-it.sws.iastate.edu/cel/view/1891

ISU lost Internet access for a number of minutes a bit after 2 PM today. The local IT folks say a soon to be replaced router failed.

2835 Views 10 Oct Internet Outage [resolved]

Link: https://www-it.sws.iastate.edu/cel/view/1875

Internet for the entire ISU campus was out between 2:47 and 3:22 PM today (10 October). No word on what the issue was or if we are stable now.

4:15 PM update A network misconfiguration was made resulting in the campus wide outage. We should be stable now though.

2675 Views Sep 22 Internet Outage [resolved]

Link: http://mesonet.agron.iastate.edu/

Our local network/Internet was down between 10:05 and 10:12 AM this morning. Unsure what is going on other than a local building router is sick.

2366 Views 9/18 RADAR Data Outage [resolved]

Link: http://mesonet.agron.iastate.edu/

Our upstream source of NWS RADAR data is currently off the Internets, so we are having an outage of RADAR products. Have not heard any details on what the issue is other than network outage!

Update 2:30 PM: The upstream source returned at 2:26 PM and I have mostly repaired the missed data during the outage.

1099 Views 17 Aug - Internet Outages

Link: https://www-it.sws.iastate.edu/cel/view/1850

There were two internet outages this morning as ISU continues work on the network backbone upgrades. There is a light at the end of the tunnel and the hope is that the work is mostly done now.

1070 Views 13 Aug - Internet Outage

Link: https://www-it.sws.iastate.edu/cel/

There was a brief Internet outage between 12:20 and 12:30 PM today as some local network issue occurred. I assume this is related to continued work on local upgrades.

1200 Views 11 Aug - Internet Outage

Link: http://mesonet.agron.iastate.edu/

We lost Internet connectivity around 10:10 AM. It is not clear what is currently broken, but I have hacked around it by hard coding a network routing path. Unsure how stable this config is, but things are working again at the moment (10:45 AM).

1479 Views 7 AM - 9 June Network Outage

Link: https://www-it.sws.iastate.edu/cel/view/1777

A routine ISU network maintenance event this morning for the Agronomy building did not go as hoped and resulted in a prolonged outage between about 6:50 and 7:30 AM this morning.