Showing posts with label simulation. Show all posts
Showing posts with label simulation. Show all posts

29 May 2017

British Airways Glitch Echoes 2016 Southwest Outage

GlitchReporter.com: Guardian story on British Airlines Outage 2017
While the postmortem is still ongoing, a May 2017 British Airways computer system glitch appears to have had much in common with a 2016 Southwest Airlines outage. Both involved seemingly well-understood hardware failures that produced a cascade of problems that delayed an orderly recovery. 

Planning for such outages is nontrivial. As the scale of data and network connectivity increases, models for post-failure recovery processing are difficult to model. That said, public perception that the airlines should be doing better to mitigate these outages is understandable. Second-guessing has already begun. Was it a lapse in cybersecurity?  Massive outsourcing?  Loss of talent?  Cost-cutting?

It may have been a failure to properly model British Airways systems and the processes required to recover from a hardware outage.

H. Herodotou, B. Ding, S. Balakrishnan, G. Outhred, and P. Fitter, "Scalable near real-time failure localization of data center networks," in Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD '14.    New York, NY, USA: ACM, 2014, pp. 1689-1698. [Online]. Available: http://doi.acm.org/10.1145/2623330.2623365 

04 October 2013

I.T. Press Asleep During Health Exchange Debacle

There is no illustration for this blog post, because the news is about the absence of news coverage.

Remember that IBM TV ad from the 90's with the sales curve spiking as a nameless tech enterprise hit the ENTER key and opened its web store for business on day one? Apparently architects and resource planners for health exchanges never saw it. 

On October 1, 2013 health insurance exchanges for the federal and state governments opened for business. The law, the Patient Protection and Affordable Care Act, was enacted in March 2010. Allowing for delays in launching the supporting bureaucracy and awarding contracts, it seems fair to say that technologists had at least a year to prepare for October 1, 2013. So when health exchange web resources fell short of meeting demand, the failure appears to be technical rather than organizational.

The failure should have been big news in the information technology trade press.

Home page for InfoWorld? No mention. ComputerWorld didn't mention it. Their lead story was the FBI busting the operator of Silk Road. CNET covered the government web sites that were down due to the budget dispute, but does not mention performance issues with the exchanges. (The screenshot displayed with their story was of the Library of Congress web site, not a health exchange site.)  CNN Tech? Nothing. Slashdot ran an unattributed four-sentence story.

So we must leave it to the traditional lay journalists? A Reuters story that ran on Yahoo News included a sensible survey of experiences at various state-run sites as well as the federal site. The New York Times covered it on October 2, pointing out that exchanges in California, Vermont, Minnesota, Nevada and Connecticut were functioning fairly well despite occasional glitches.

Here are a few reasons why this absence of coverage is troubling. This systemic pattern of glitches may reflect:
  • Complacency toward failed software quality 
  • Complacency about IT project management 
  • Failure to take responsibility for success or failure of IT initiatives
  • Possible weakness in professional standards for system capacity planning, risk management and resilience 
  • Design failures in separation of static vs. dynamic page handling under load 
  • Failures to perform adequate load and scalability simulation and test
  • Inability to connect the public impact of systems with stories with technology storytelling
  • Inability to recognize and implement the "we're overloaded, come back later" UI design pattern 
Who's to blame? A GAO report on the status of preparations for the exchanges, issued in June of 2013, identified some of the major vendors in an eWeek story.

Regardless, I am embarrassed for the profession.

UPDATE 8-OCT-2013 The New York Times reports that part of the account creation code was at fault. This story also links to more detailed speculation at a Reddit thread, and a Reuters story quoting some sensible guesswork by Appdynamics.

UPDATE 13-OCT-013 The New York Times is running this IT failure as a home page article. Summary at IEEE Spectrum.



16 June 2010

Twitter Outages: Network Simulation, Anyone?

Whether caused by increased use over the World Cup, or other causes, Twitter is having service level management and possibly other issues. In their June 11 post acknowledging the problems and blaming it on a "perfect storm" of issues, their engineers attributed the outages to capacity planning and errors in configuration management.  While some, such as Forrester's Gualtieri, quoted by Computerworld, accept this level of transparency, it's minimal at best. My recent TechRepublic post on capacity planning for backup has some suggestions that Twitter might want to consider; the methods currently in use by the firm, one guesses, haven't involved careful simulation or other prudent measures.