Showing posts with label disaster planning. Show all posts
Showing posts with label disaster planning. Show all posts

09 August 2016

Single Router Failure Blamed on Southwest Outage

Screenshot of Computerworld story on Southwest Airlines Outage in July 2016
Computerworld Story on July 2016 Outage at Southwest Airlkines

Single point of failure.  You know the concept.  Those single points are easier to see in the rear view mirror. Can a single router cause an entire data center to go down?

In a Dallas Morning News story, CEO Gary Kelly said it could -- and did. There was a "backup system" in place to address a single router failure. 

But, Kelly insisted, because of the router's unusual "partial failure," the backup procedures weren't triggered, and the problem became a massive one.

No, this isn't a very good technical explanation .

Kelly took pains to insist that the legacy technology in their data center wasn't involved in the failure. Nor was the cause a hack of some sort (probably true, though at the time of this report it might be premature to completely rule that out).

Whether they're justified in saying so or not, Southwest Airlines unions want a change of leadership, arguing that the CEO and others are delaying technology improvements in order to maintain profits for investors. (True, Southwest employees hold plenty of Southwest stock, too.) What do the IT folks at Southwest say? I didn't see any comment on that.

CNBC listed the publicly reported airline technology failures since 2015.



Single Router Failure Blamed on Southwest Outage

Screenshot of Computerworld story on Southwest Airlines Outage in July 2016
Computerworld Story on July 2016 Outage at Southwest Airlkines

Single point of failure.  You know the concept.  Those single points are easier to see in the rear view mirror. Can a single router cause an entire data center to go down?

In a Dallas Morning News story, CEO Gary Kelly said it could -- and did. There was a "backup system" in place to address a single router failure. 

But, Kelly insisted, because of the router's unusual "partial failure," the backup procedures weren't triggered, and the problem became a massive one.

No, this isn't a very good technical explanation .

Kelly took pains to insist that the legacy technology in their data center wasn't involved in the failure. Nor was the cause a hack of some sort (probably true, though at the time of this report it might be premature to completely rule that out).

Whether they're justified in saying so or not, Southwest Airlines unions want a change of leadership, arguing that the CEO and others are delaying technology improvements in order to maintain profits for investors. (True, Southwest employees hold plenty of Southwest stock, too.) What do the IT folks at Southwest say? I didn't see any comment on that.

CNBC listed the publicly reported airline technology failures since 2015.



18 April 2015

Chat, Big UK Transaction, Chinese Rule Changes: Cascading Impacts of Bloomberg Outage

Bloomberg IoS App Screenshot 2011 | Clive Darra via Flickr 
Michael Bloomberg's namesake company is a de facto standard software/network application that is widely used in finance. On April 17, 2015, as reported by media outlets including Fortune, that outage experienced a worldwide failure.

Should we care? It depends. The British government UK Debt Management Office cared enough to delay the sale of $4.5 billion of treasury notes.  Another cascading effect mentioned by the Wall Street Journal was the inability of traders to react to regulatory changes made by the Chinese Government around the time that the outage began. The WSJ said Bloomberg identified the "root cause" as a hardware failure.

How many were affected? Bloomberg claims to have around 320,000 customers worldwide according to the Fortune story, and all were affected by the outage which began at 3PM Singapore time. 

On a WSJ blogJosie Cox wrote about the importance of this information source, quoting this senior trader:
The communication chat has become vital to the sharing of information across regions and counterparties. So a global outage like this is systemically important to markets all around the world,” said Louis Gargour, the chief investment officer at London-based asset manager LNG Capital, adding that this shows just how vulnerable the market has become.
Bloomberg representatives acknowledged that redundant systems had failed and an investigation was underway. They did not believe this to be a cyberattack. 

Reuters offers a somewhat similar services, but traders were reportedly uncomfortable using it, and many simply wait for service to be restored, or resorted to phone calls and other resources.  

Media outlets reported that the loss of Bloomberg's chat service was the most difficult to overcome during the outage. 

19 August 2013

"Faceback" -- Analyzing the Facebook Backup Plan after NY Times Outage


On August 14 of this year, the New York Times web site experienced an outage that affected its web and mobile services. The outage lasted for around an hour and a half. The explanation was similar to others the GlitchReporter has listed on this site: the problem emerged when a maintenance update had been released.

Tentative Root Cause Analysis

Software testing failure.

Other Worries

The outage was significant for a number of reasons, but two of them are worthy of additional analysis.
  1. Hacker Scare There was an abundance of tweets suggesting that the site had been hacked by Chinese adversaries, or had been the object of a DoS attack. This misinformation is likely to become a recurring issue and not only for the NYT. While it can be subsequently "disproven," some customers will continue to believe that site had been hacked and is thus less reliable than previously believed.
  2. Failover and Backup While the New York Times technology folks were sorting out what their representative called "an internal problem," important news was happening. Computerworld said that at around 12:45 p.m.on the day of the outage, "the Times tweeted that it was posting major breaking stories on its Facebook page." As the Guardian's Dan Gilmore worried, this is a flawed, and perhaps simply a spur of the moment decision. Social network platforms should provide channels for announcements of alternate NYT digital publication channels -- not host them. 

23 July 2013

NYC Legacy 911 System Fails Again - Intergraph System Still on Deck

911 Call Center via NBC News
As the GlitchReporter noted a few weeks ago, the NYC 911 system is scheduled to be replaced by a new system costing an estimated $2B. The new system, originally to be designed by HP, is now being developed by Intergraph, but will not be ready until 2015. The FDNY says the current 911 system is around 30 years old. It is the same system that notoriously ran into trouble on 9/11.

In the meantime, glitches continue to be experienced in some combination of the legacy system -- as we noted in that post -- and ICAD, a dispatch system used by the NYPD.  A previous outage was blamed on someone turning off a server, which caused other servers to crash.

Yesterday, as reported by the NYT, there were six outages, resulting in a total of around 90 minutes of downtime.

Details as to which systems (Dispatch? 911? NYPD? FDNY?) are failing, and the causes of such failures, seem to go generally unexplained. According to reporters, Intergraph refers questions to City workers.

13 May 2013

Software Errors Further Weakened Nuclear Utility's Credibility

How bad is it? That's what Japan -- and worriers worldwide -- asked of its utility, Tokyo Electric Power (TEP) after the disastrous quake and tsunami.  TEP employees no doubt lost family members, and some of them braved unprecedented risks to their own health and safety. No doubt working under pressure and minimum rest, TEP nevertheless had a responsibility to provide accurate reports of groundwater radioactivity, and were forced to rely upon software to deliver those reports.

A NY Times report provided this narrative.
"Tokyo Electric Power said late Thursday that the groundwater near the plant had also been contaminated. The company initially said the radiation was 10,000 times the normal level, then later questioned its own data. It did the same with a report that the level of iodine 131 in seawater near the plant had fallen below 2,000 times the statutory limit, compared with a level of more than 4,000 times the limit on Thursday. On Friday, Hidehiko Nishiyama, deputy head of the Japanese Nuclear and Industrial Safety Agency, said at a news conference that the government had questions about the company’s figures and had asked it to review the data. Tokyo Electric officials said they would recalculate the figures after an error was discovered in a computer program (italics mine). The company has several times issued radioactivity reports only to retract them after experts questioned their validity."

30 August 2010

VA DMV Closes Due to Statewide Glitch


It's 2010 and yet the entire state of Virginia's DMV system is being taken down for an entire day by a hardware failure. Here's the state's message to its citizens: "DMV will not be able to process driver's licenses or ID cards in its 74 customer service centers on Monday, August 30 due to a statewide computer system problem affecting those transactions only."  As reported by ZDNet, the outage is being blamed by contractor Northrop Grumman on the failure of two circuit boards manufactured by EMC -- i.e. a SAN failure.  I must agree with ZDNet's Dignan that something just doesn't seem right about the explanation being given by project managers.  Disaster planning?  Redundancy?  Single point of the failure?  And that's just the beginning. What if there'd been a real disaster?  After all, this is Virginia, home of many federal agencies and much of the federal workforce.