Showing posts with label airlines. Show all posts
Showing posts with label airlines. Show all posts

29 May 2017

British Airways Glitch Echoes 2016 Southwest Outage

GlitchReporter.com: Guardian story on British Airlines Outage 2017
While the postmortem is still ongoing, a May 2017 British Airways computer system glitch appears to have had much in common with a 2016 Southwest Airlines outage. Both involved seemingly well-understood hardware failures that produced a cascade of problems that delayed an orderly recovery. 

Planning for such outages is nontrivial. As the scale of data and network connectivity increases, models for post-failure recovery processing are difficult to model. That said, public perception that the airlines should be doing better to mitigate these outages is understandable. Second-guessing has already begun. Was it a lapse in cybersecurity?  Massive outsourcing?  Loss of talent?  Cost-cutting?

It may have been a failure to properly model British Airways systems and the processes required to recover from a hardware outage.

H. Herodotou, B. Ding, S. Balakrishnan, G. Outhred, and P. Fitter, "Scalable near real-time failure localization of data center networks," in Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD '14.    New York, NY, USA: ACM, 2014, pp. 1689-1698. [Online]. Available: http://doi.acm.org/10.1145/2623330.2623365 

09 August 2016

Single Router Failure Blamed on Southwest Outage

Screenshot of Computerworld story on Southwest Airlines Outage in July 2016
Computerworld Story on July 2016 Outage at Southwest Airlkines

Single point of failure.  You know the concept.  Those single points are easier to see in the rear view mirror. Can a single router cause an entire data center to go down?

In a Dallas Morning News story, CEO Gary Kelly said it could -- and did. There was a "backup system" in place to address a single router failure. 

But, Kelly insisted, because of the router's unusual "partial failure," the backup procedures weren't triggered, and the problem became a massive one.

No, this isn't a very good technical explanation .

Kelly took pains to insist that the legacy technology in their data center wasn't involved in the failure. Nor was the cause a hack of some sort (probably true, though at the time of this report it might be premature to completely rule that out).

Whether they're justified in saying so or not, Southwest Airlines unions want a change of leadership, arguing that the CEO and others are delaying technology improvements in order to maintain profits for investors. (True, Southwest employees hold plenty of Southwest stock, too.) What do the IT folks at Southwest say? I didn't see any comment on that.

CNBC listed the publicly reported airline technology failures since 2015.



Single Router Failure Blamed on Southwest Outage

Screenshot of Computerworld story on Southwest Airlines Outage in July 2016
Computerworld Story on July 2016 Outage at Southwest Airlkines

Single point of failure.  You know the concept.  Those single points are easier to see in the rear view mirror. Can a single router cause an entire data center to go down?

In a Dallas Morning News story, CEO Gary Kelly said it could -- and did. There was a "backup system" in place to address a single router failure. 

But, Kelly insisted, because of the router's unusual "partial failure," the backup procedures weren't triggered, and the problem became a massive one.

No, this isn't a very good technical explanation .

Kelly took pains to insist that the legacy technology in their data center wasn't involved in the failure. Nor was the cause a hack of some sort (probably true, though at the time of this report it might be premature to completely rule that out).

Whether they're justified in saying so or not, Southwest Airlines unions want a change of leadership, arguing that the CEO and others are delaying technology improvements in order to maintain profits for investors. (True, Southwest employees hold plenty of Southwest stock, too.) What do the IT folks at Southwest say? I didn't see any comment on that.

CNBC listed the publicly reported airline technology failures since 2015.



08 August 2016

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

UPDATE (Via Reuters) A former industry worker suggested this possible explanation: "The carrier was probably running a routine test of its backup power supplies when the switch gear failed and locked Delta out of its reserve generators as well as from Georgia Power, industry analyst and former airline executive Robert Mann said. That would result in a shutdown of Delta's data center, which controls bookings, flight operations and other critical systems, he said."

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

UPDATE (Via Reuters) A former industry worker suggested this possible explanation: "The carrier was probably running a routine test of its backup power supplies when the switch gear failed and locked Delta out of its reserve generators as well as from Georgia Power, industry analyst and former airline executive Robert Mann said. That would result in a shutdown of Delta's data center, which controls bookings, flight operations and other critical systems, he said."

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

UPDATE (Via Reuters) A former industry worker suggested this possible explanation: "The carrier was probably running a routine test of its backup power supplies when the switch gear failed and locked Delta out of its reserve generators as well as from Georgia Power, industry analyst and former airline executive Robert Mann said. That would result in a shutdown of Delta's data center, which controls bookings, flight operations and other critical systems, he said."

14 January 2016

JetBlue's Turn for a DataCenter Outage


There's a pattern of airline glitches which illustrate the risks of centralized systems. Whether these are simply the ones most often publicized, or whether they are the most stressed is unclear, but there's far too much to report on from that one sector.

One among multiple media reports of the outage was this one provided by CNBC. The airline reported that the cause was a power outage at a Verizon data center.  The explanation, so far, stops there -- as though planning for a power outage wasn't part of operating a data center?

►Read previous GlitchReporter messages about airline glitches.

15 August 2015

Today Must Be Saturday, Time for an Air Traffic Control Glitch


Newport News - Williamsburg VA Airport (credit: Wikipedia)
In a long-familiar aviation pattern, there was another glitch in the US air traffic control system. CNBC reported late in the afternoon that an "automation" problem had been resolved. Airlines said the impact was significant; for example, Southwest Airlines said that "fewer than 200" of its flights were affected. The meaning of "automation" was characteristically unclear at the time of this glitch report.

08 July 2015

"Network Connectivity" Failure Hits United Airlines

Fox31 In Denver Reporting United Airlines Computer Failure
It's "paper time," but younger travelers at United Airlines counters around an undetermined part of the United States may never have seen a paper ticket.

The reason is a software or hardware failure described as "network connectivity."  As is usually the case, the carrier doesn't refer to it as "failure."

CNN reported that the problem occurred at "multiple airports," but an on-air reporter at WNYC indicated that local airports may not have extensive delays as of this report (10A July 8, 2015).

UPDATE The problem was rectified about 90 minutes later.

United Airlines System Failure History


In summer of 2011, United Airlines also experienced a system-wide outage. And there were dispatch system outages in November 2012 and again just last month.

Image credit: @KDVR  

03 April 2015

JetBlue Joins Commercial Aviation Glitch Airshow

JetBlue Outage Reported on ABC News http://abcn.ws/1y5HhHE
JetBlue is known for its quasi-independent "budget airline" business model, but on March 30, 2015 it looked like just another of the lumbering US airline giants. As reported by ABC News, the airline experienced a nationwide computer system outage. The cause was not immediately reported, but this (extended) quote from the airline's 2013 10K report to investors is instructive: Cause? Computerworld reported the outage but did not appear to report on the cause. (But you can read about a 2007 JetBlue outage caused by capacity problems there.)

We rely heavily on automated systems to operate our business; any failure of these systems could harm our business.We are dependent on automated systems and technology to operate our business, enhance customer service and achieve low operating costs. The performance and reliability of our automated systems and data center is critical to our ability to operate our business and compete effectively. These systems include our computerized airline reservation system, flight operations system, telecommunications systems, website, maintenance systems, check-in kiosks, in-flight entertainment systems and our primary and redundant data centers. Our website and reservation system must be able to accommodate a high volume of traffic and deliver important flight information. These systems require upgrades or replacement periodically, which involve implementation and other operational risks. Our business may be harmed if we fail to operate, replace or upgrade our systems or data center infrastructure successfully.
We rely on the third party providers of our current automated systems and data center infrastructure for technical support. If the current provider were to fail to adequately provide technical support for any one of our key existing systems or if new or updated components were not integrated smoothly, we could experience service disruptions, which, if they were to occur, could result in the loss of important data, increase our expenses, decrease our revenues and generally harm our business and reputation. Furthermore, our automated systems cannot be completely protected against events beyond our control, including natural disasters, computer viruses, other security breaches, or telecommunications failures. Substantial or sustained system failures could impact customer service and result in our customers purchasing tickets from other airlines. We have implemented security measures and change control procedures and have disaster recovery plans as well as requiring our third party providers to have disaster recovery plans; however, we cannot assure you these measures are adequate to prevent disruptions, which, if they were to occur, could result in the loss of important data, increase our expenses, decrease our revenues and generally harm our business and reputation.

26 March 2011

Mere Outage or Outrage? Ask Alaska/Horizon Air Ticketing Agents

Via AP - ABC News / Twitter @BreakingNews Alaska / Horizon Airlines reported that a computer outage initially stated to be of unknown origin with its "central computers" has delayed a number of flights.  Later in the day, the same ABC News / AP report offered these additional details:
Alaska Airlines and its Horizon Air affiliate canceled 95 flights Saturday because a computer system used for flight planning failed. The outage lasted intermittently for about seven hours and resulted in the two airlines scrapping about 12 percent of their combined schedule before technicians fixed the system, which returned at 10 a.m. Pacific time.

30 October 2007

United Airlines

[21 June 2007] The United Airlines computer system responsible for dispatching flights for departure went down, forcing the airline to temporarily ground all flights for about two hours. Ciose to 300 flights were delayed or canceled. But the day after the glitch, United's system, called Unimatic, was not blamed for the error. COO Peter McDonald said the error, which took place during routine testing, was caused by employee error. The error affected both the primary system and the backup. Apparently the employee accidentally disabled both systems during a test run.

According to a report by Baseline, "Michael Boyd, an airline consultant based in Evergreen, Colo., told the Chicago Tribune that United's computer glitch will cost it more than $10 million in lost revenue due to refunds and re-bookings."

No additional details were given.