Facebook Outage - NorthEast (FIOS)

Email Received about 40 Minutes after Outage

Twitter Messaging about Facebook Outage

Some users are seeing Facebook outages as the time of this posting. The Facebook Twitter account is silent on the matter.  Posted at 12:40 Eastern.

UPDATE Service was restored by 12:50 in this locale.

UPDATE The email message reproduced above was received at 12:59.

Facebook (Regional?) Outage

Currently seeing a timeout on Facebook, thought it is responding to pings. (NYC Greater Metro, FIOS). The API is also nonresponsive (fails through Hootsuite).

I.T. Press Asleep During Health Exchange Debacle

There is no illustration for this blog post, because the news is about the absence of news coverage.

Remember that IBM TV ad from the 90's with the sales curve spiking as a nameless tech enterprise hit the ENTER key and opened its web store for business on day one? Apparently architects and resource planners for health exchanges never saw it. 

On October 1, 2013 health insurance exchanges for the federal and state governments opened for business. The law, the Patient Protection and Affordable Care Act, was enacted in March 2010. Allowing for delays in launching the supporting bureaucracy and awarding contracts, it seems fair to say that technologists had at least a year to prepare for October 1, 2013. So when health exchange web resources fell short of meeting demand, the failure appears to be technical rather than organizational.

The failure should have been big news in the information technology trade press.

Home page for InfoWorld? No mention. ComputerWorld didn't mention it. Their lead story was the FBI busting the operator of Silk Road. CNET covered the government web sites that were down due to the budget dispute, but does not mention performance issues with the exchanges. (The screenshot displayed with their story was of the Library of Congress web site, not a health exchange site.)  CNN Tech? Nothing. Slashdot ran an unattributed four-sentence story.

So we must leave it to the traditional lay journalists? A Reuters story that ran on Yahoo News included a sensible survey of experiences at various state-run sites as well as the federal site. The New York Times covered it on October 2, pointing out that exchanges in California, Vermont, Minnesota, Nevada and Connecticut were functioning fairly well despite occasional glitches.

Here are a few reasons why this absence of coverage is troubling. This systemic pattern of glitches may reflect:

• Complacency toward failed software quality 
• Complacency about IT project management 
• Failure to take responsibility for success or failure of IT initiatives
• Possible weakness in professional standards for system capacity planning, risk management and resilience 
• Design failures in separation of static vs. dynamic page handling under load 
• Failures to perform adequate load and scalability simulation and test
• Inability to connect the public impact of systems with stories with technology storytelling
• Inability to recognize and implement the "we're overloaded, come back later" UI design pattern 

Who's to blame? A GAO report on the status of preparations for the exchanges, issued in June of 2013, identified some of the major vendors in an eWeek story.

Regardless, I am embarrassed for the profession.

UPDATE 8-OCT-2013 The New York Times reports that part of the account creation code was at fault. This story also links to more detailed speculation at a Reddit thread, and a Reuters story quoting some sensible guesswork by Appdynamics.

UPDATE 13-OCT-013 The New York Times is running this IT failure as a home page article. Summary at IEEE Spectrum.

Glitch vs. Glitch: Microsoft Live vs. Google Gmail

Since no one has figured out how to write error-free software, cloud glitches, sooner or later, are likely to affect more and more people. As Woody Leonhard notes, it's instructive to see how cloud providers react. Leonhard suggests a comparison between the recent Gmail incident and Microsoft's 30 December 2010 outage at Hotmail. In the latter case, Leonhard's research suggests that Microsoft didn't provide an official response until four days after the seemingly not dissimilar scripting problem that caused the Hotmail outage. Not that Google has given much of an explanation so far, either.

As anyone whose flight has been cancelled will attest, providing timely information eases the pain a little -- even if it's only to say "We don't have a new departure time yet, but here's what we know so far . . ." IT must make a similar commitment, or be content with user-unfriendly, unsympathetic software and services that take a cue from indifference writ large. The challenges are both technical and organizational. Support staff need to be empowered to escalate and route problems to the appropriate teams, since existing problem resolution workflow is likely to be unsuitable for major outages. 

Support services must scale along with cloud migration, and the indications so far are unimpressive.