Showing posts with label aeronautics. Show all posts
Showing posts with label aeronautics. Show all posts

08 August 2016

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

UPDATE (Via Reuters) A former industry worker suggested this possible explanation: "The carrier was probably running a routine test of its backup power supplies when the switch gear failed and locked Delta out of its reserve generators as well as from Georgia Power, industry analyst and former airline executive Robert Mann said. That would result in a shutdown of Delta's data center, which controls bookings, flight operations and other critical systems, he said."

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

UPDATE (Via Reuters) A former industry worker suggested this possible explanation: "The carrier was probably running a routine test of its backup power supplies when the switch gear failed and locked Delta out of its reserve generators as well as from Georgia Power, industry analyst and former airline executive Robert Mann said. That would result in a shutdown of Delta's data center, which controls bookings, flight operations and other critical systems, he said."

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

Delta Airlines' Turn for Resilience and DR Systems Fail

ComputerWorld Story on Delta Airlines August System-wide Outage
Power failure is one of the most planned-for disaster recovery modes, yet DR plans to address it appear to fail on a regular basis. As reported in this Computerworld story, a power outage struck the company's headquarters around 2:30A. 

The result was not only a system-wide outage, but also side-effects seemingly designed to maximize already frayed traveler nerves. According to the report, "airport screens and other flight status systems were incorrectly showing flights as being on time."

Testing for power failure is one of most obvious elements of the DR canon, and has been for decades. Such testing is anything but trivial, but it appears from this and other system-wide outages that resilience planning is not given a high enough priority.

Announcements about the outage and related customer service support are key, as shown in the recent Southwest Airlines outage. That airport monitors are showing incorrect status suggests that secondary systems designed to degrade graceful are not working properly, either. 

Engineers have suggested a number of remedies. SDN has been suggested as one way to making communication systems more resilient. In their study "The Data Center as a Computer," Barroso, Clidaras and Hölzle suggest an approach that involves"tolerating faults, not hiding them."

UPDATE (Via Reuters) A former industry worker suggested this possible explanation: "The carrier was probably running a routine test of its backup power supplies when the switch gear failed and locked Delta out of its reserve generators as well as from Georgia Power, industry analyst and former airline executive Robert Mann said. That would result in a shutdown of Delta's data center, which controls bookings, flight operations and other critical systems, he said."

14 January 2016

JetBlue's Turn for a DataCenter Outage


There's a pattern of airline glitches which illustrate the risks of centralized systems. Whether these are simply the ones most often publicized, or whether they are the most stressed is unclear, but there's far too much to report on from that one sector.

One among multiple media reports of the outage was this one provided by CNBC. The airline reported that the cause was a power outage at a Verizon data center.  The explanation, so far, stops there -- as though planning for a power outage wasn't part of operating a data center?

►Read previous GlitchReporter messages about airline glitches.

15 October 2015

Glitch Again Targets Travelers in US Customs Watch List Outage

UPI Report on Customs Outages in US Airports
If the major media outlets are the only ones you read, you'd think glitches disproportionately affect travelers. As reported yesterday by UPI, travelers were once again the "target" of an outage at a system operated by DHS that checks incoming passengers against a Customs and Border Protection watch list. The glitch was relatively shortlived (less than an hour ot two), several airports reported long lines and delays.

In July 2014, the State Department's global travel system, called the Consular Consolidated Database, suffered a glitch that resulted in delays for passport and visa applications.

photo: Huff Post Story on Database Outage July 2014
State Department Glitch in July 2014

03 April 2015

JetBlue Joins Commercial Aviation Glitch Airshow

JetBlue Outage Reported on ABC News http://abcn.ws/1y5HhHE
JetBlue is known for its quasi-independent "budget airline" business model, but on March 30, 2015 it looked like just another of the lumbering US airline giants. As reported by ABC News, the airline experienced a nationwide computer system outage. The cause was not immediately reported, but this (extended) quote from the airline's 2013 10K report to investors is instructive: Cause? Computerworld reported the outage but did not appear to report on the cause. (But you can read about a 2007 JetBlue outage caused by capacity problems there.)

We rely heavily on automated systems to operate our business; any failure of these systems could harm our business.We are dependent on automated systems and technology to operate our business, enhance customer service and achieve low operating costs. The performance and reliability of our automated systems and data center is critical to our ability to operate our business and compete effectively. These systems include our computerized airline reservation system, flight operations system, telecommunications systems, website, maintenance systems, check-in kiosks, in-flight entertainment systems and our primary and redundant data centers. Our website and reservation system must be able to accommodate a high volume of traffic and deliver important flight information. These systems require upgrades or replacement periodically, which involve implementation and other operational risks. Our business may be harmed if we fail to operate, replace or upgrade our systems or data center infrastructure successfully.
We rely on the third party providers of our current automated systems and data center infrastructure for technical support. If the current provider were to fail to adequately provide technical support for any one of our key existing systems or if new or updated components were not integrated smoothly, we could experience service disruptions, which, if they were to occur, could result in the loss of important data, increase our expenses, decrease our revenues and generally harm our business and reputation. Furthermore, our automated systems cannot be completely protected against events beyond our control, including natural disasters, computer viruses, other security breaches, or telecommunications failures. Substantial or sustained system failures could impact customer service and result in our customers purchasing tickets from other airlines. We have implemented security measures and change control procedures and have disaster recovery plans as well as requiring our third party providers to have disaster recovery plans; however, we cannot assure you these measures are adequate to prevent disruptions, which, if they were to occur, could result in the loss of important data, increase our expenses, decrease our revenues and generally harm our business and reputation.

16 December 2014

Heathrow, Gatwick Airports Affected by ATC Outage

A brief report of the air traffic control software failure in an operations center in the south of England appears on softwareqc.com. Around 40 flights were cancelled at Heathrow alone.




09 July 2013

UK Air Control System Remediates Outage Affecting Southerly Routes



Nick Heath at ZDNet reported that earlier this morning the UK National Air Traffic Service suffered an outage that affected many southerly routes across the country. Heath was tipped off to the problem by a tweet from David Kaminski-Morrow. As Heath puts it,
The technical issues were triggered by a "problem with a flight plan" entered into a system according to a spokeswoman for NATS.
The problem was resolved after a short time, but a backlog of flights resulted. GlitchReporter suspects either software test failure, or a related lapse in data quality.

05 July 2013

Ground-based Midcourse Defense (GMD) Intercept Fails

Boeing GMD Description
The U.S. Department of Defense announced that a test of the Ground-based Midcourse Defense (GMD) system, part of the Missile Defense System, failed to intercept as planned. The target was a long range ballistic missile fired from the Marshall Islands. The interceptor was launched from Vandenberg AFB. 

According to the Wikipedia program summary, recent estimates of inception-to-date program costs through 2017 are $40.9 billion.

Boeing is the prime contractor for GMD, and describes the program on its web site:
The Ground-based Midcourse Defense (GMD) System is the first and only operationally deployed missile defense program to defend the homeland against long-range ballistic missile attacks. The system provides early detection and tracking during the boost phase, midcourse target discrimination, precision intercept and destruction of inbound ICBMs through force of hit-to-kill technology.

21 October 2007

Predator UAV Failure

An April 2006 crash of a Predator UAV near Nogales was indirectly caused by repeated software failures. As reported by USA Today:
No one was injured, but the plane slammed into a hillside near homes in a sparsely populated area north of Nogales, Ariz. A 'pilot' flying the single-engine turboprop plane from a trailer at a military base accidentally shut down the plane's engine while attempting to deal with a locked up computer control console, according to the NTSB.
The NTSB found several disturbing problems in their report:

  • Computers used to control the Predator had locked up at least 16 times in the four months before the accident. The failures were not well documented and no attempt was made to determine what caused the problems.
  • The drone's pilot was supposed to be monitored by a flight instructor, but none were around at the time of the outage.
  • When the Predator's engine stopped, it cut electrical power to key instruments, including the radio transponder that made the plane visible to air-traffic controllers. This meant that nobody knew where the plane went, including other aircraft.
  • The Predator's engine cannot be restarted without a radio link to the ground. By the time the operators realized the drone had lost power, it had descended behind mountains and radio contact was impossible.
  • A report in C4ISR Journal mentions that the operator had attempted to transfer control to a secondary workstation. However, this procedure calls for the secondary workstation to be "identically configured" at the time of the transfer. The secondary workstation was configured in "fuel off" mode, so the drone obeyed this command.

The Predator B destroyed in the crash cost $6.5M.