Showing posts with label software quality. Show all posts
Showing posts with label software quality. Show all posts

27 March 2016

F-35 Mission Software Glitch May Impact Deployment Schedule


Jane's Report on F-35 Software
In a content preview, OHS Jane's 360 headlines their story with a claim that the F-35's mission software has serious stability concerns that are a risk to DoD operators.

In March 2016, Jane's called this problem a "software glitch":
A software glitch that interferes with the ability of the Lockheed Martin F-35 Lightning II Joint Strike Fighter's AN/APG-81 AESA radar working in flight poses the greatest threat to delaying US Air Force (USAF) plans to declare its jets operationally deployable, a top service official told IHS Jane's on 4 March.
Read the Guardian's analysis of the same announcement.

15 February 2016

"Oops, Something Went Wrong" Intuit Quickbooks Online Glitch

Users Respond to Quickbooks Online Glitch
Intuit's Quickbooks Online experienced a glitch that prevented users from logging in.  The cryptic error message: "Oops, something went wrong."

Oh yes, do visit errorprocessing.com or visit the Pinterest page for more fun -- and annoyingly unhelpful -- screenshots.



Event: 2016-02-15 Noonish eastern.

23 December 2015

Washington State Glitch A Get-Out-Early Pass for Prisoners


Software is increasingly involved in enforcing public policies. The recent Volkswagen emissions testing scandal demonstrated this -- both from the deception perspective, and the testing that uncovered the deception. 

In Washington state, according to an NPR report, software was employed to comply with a state Supreme Court order  that required state corrections officials to add good-behavior credits earned in county jail venues to their state-administered sentences.  The error reportedly affected only around 3% of all releases, but the error had been in place for more than 13 years. 

 Audit?  Sort of. Although the problem had been detected by a victim's family which questioned the early release of a convict, the state, according to the NPR report, said that "a coding fix was scheduled, repeatedly delayed and then was never implemented."

Washington state taxpayer-supported hearings? Count on it.

Image credit: Kate Ter Haar | Flickr
Mark Underwood (@knowlengr) is a software quality engineer at Krypton Brothers based in greater NYC. 


24 September 2015

Facebook Outage - NorthEast (FIOS)

Email Received about 40 Minutes after Outage
Twitter Messaging about Facebook Outage

Some users are seeing Facebook outages as the time of this posting. The Facebook Twitter account is silent on the matter.  Posted at 12:40 Eastern.

UPDATE Service was restored by 12:50 in this locale.

UPDATE The email message reproduced above was received at 12:59.





04 November 2014

NYC EMS System Outage (Again)


As the Glitch Reporter has previously noted, outages seem to be an unavoidable, if irregular aspect of systems development for the HP-managed NYC emergency management system. 

The outage was reported by the local ABC TV station, WNYC radio and covered in more detail by Progress Queens. 

NYC 911 emergency call system goes down, as problems with ECTP continue — Progress Queens

25 September 2014

Glitch: WSJ Says Apple Pulled iOS Software Update

Image by Apple via WSJ Online

The Wall Street Journal reports that Apple has withdrawn an issued update to IOS 8
Apple Pulls iOS Software Update - WSJ. Some owners reported that the patch interfered with call placement and/or the fingerprint sensor.


26 August 2014

CME Group resumes electronic trading after another glitch delays open | Reuters

In reporting this glitch yesterday, Reuters reported that:

The delay marks another headache for CME, which shut electronic trade for leading agricultural contracts on April 8 in the worst-ever trading outage for those markets. CME Executive Chairman Terrence Duffy has said that outage was triggered when sophisticated technology tripped over a trading halt in a single market.
CME Group resumes electronic trading after glitch delays open | Reuters

16 October 2013

Outage Crippling Rogers Wireless Network Was Software Failure

Around 6.5 million cellphone users were affected by an outage caused by an October 10 software glitch, Toronto TV reported last week. 

Rogers' CTO gave the following explanation.

"Due to a software glitch, the mobile network switches couldn't handle a sudden spike in all of the network signals they were receiving that contain instructions needed to put through cellphone calls and text messages." 

While data services continued without interruption, the glitch was sufficiently severe and prolonged that Rogers will credit its cellular customers for one day of service.

19 August 2013

"Faceback" -- Analyzing the Facebook Backup Plan after NY Times Outage


On August 14 of this year, the New York Times web site experienced an outage that affected its web and mobile services. The outage lasted for around an hour and a half. The explanation was similar to others the GlitchReporter has listed on this site: the problem emerged when a maintenance update had been released.

Tentative Root Cause Analysis

Software testing failure.

Other Worries

The outage was significant for a number of reasons, but two of them are worthy of additional analysis.
  1. Hacker Scare There was an abundance of tweets suggesting that the site had been hacked by Chinese adversaries, or had been the object of a DoS attack. This misinformation is likely to become a recurring issue and not only for the NYT. While it can be subsequently "disproven," some customers will continue to believe that site had been hacked and is thus less reliable than previously believed.
  2. Failover and Backup While the New York Times technology folks were sorting out what their representative called "an internal problem," important news was happening. Computerworld said that at around 12:45 p.m.on the day of the outage, "the Times tweeted that it was posting major breaking stories on its Facebook page." As the Guardian's Dan Gilmore worried, this is a flawed, and perhaps simply a spur of the moment decision. Social network platforms should provide channels for announcements of alternate NYT digital publication channels -- not host them. 

30 July 2013

Failed DMV Software Update Shakes New York Confidence in Public Infrastructure

NY Daily News Story on DMV Outage
On the heels of recent New York City 911 dispatch outages, public confidence in New York state information technology infrastructure was dealt another blow yesterday. 

As reported by the NY Daily News and others, the statewide Department of Motor Vehicles system ("a mainframe") had been updated on Sunday. Something unspecified went wrong, and the public DMV site as well as -- apparently -- computers in DMV offices, went down. The outage lasted at least until sometime the following day.

Rant

The Glitch Reporter has the usual obvious questions: (1) what happened with pre-rollout testing? (2) what failed in the software implementation that was released; (3) why aren't the public's systems more resilient when failures of this sort occur?

11 July 2013

Smokers Benefit From Software Glitch by Affordable Care Contractor(s)


The glitch is noteworthy, but more so because HHS is reporting that it will a year to fix.

Read more about this flaw in a Yahoo Finance report highlighted by Soulskill on Slashdot. This glitch will, for what is estimated to be a year, prevent insurers from charging more for smokers. Such surcharges are allowed under the law.  The analysis suggested by Yahoo Finance points out that this is likely to have an impact on rates for older smokers. The law would have permitted, for example, a rate surcharge of as much as $4,500/yr additional above the "silver" plan for a 64 year old customer. That would bring such a person's annual cost to $13,600, according to the Kaiser Health Reform Subsidy Calculator, cited by Yahoo.

Instead, insurers would not be able to add the surcharge.

Anyone know which contractor is responsible for this system? No doubt there are ripple effects throughout the design, but a year's delay to implement this business rule is mind-numbing.




09 July 2013

UK Air Control System Remediates Outage Affecting Southerly Routes



Nick Heath at ZDNet reported that earlier this morning the UK National Air Traffic Service suffered an outage that affected many southerly routes across the country. Heath was tipped off to the problem by a tweet from David Kaminski-Morrow. As Heath puts it,
The technical issues were triggered by a "problem with a flight plan" entered into a system according to a spokeswoman for NATS.
The problem was resolved after a short time, but a backlog of flights resulted. GlitchReporter suspects either software test failure, or a related lapse in data quality.

22 June 2013

Southwest Airlines System-wide Software Outage

Southwest Airlines home page screenshot
Southwest Airlines grounded 250 flights.
Late in the evening of 21 June 2013, a software failure of unreported origin caused a system-wide outage at Southwest Airlines. The outage lasted for three hours, according to news reports,  until 2A Eastern. During this period, 250 flights were grounded. As reported by USA Today, "the glitch impaired the airline's ability to do such things as conduct check-ins, print boarding passes and monitor the weight of each aircraft." A backup system was deployed, according to this source, around 8 pm Pacific time, but it was running much more slowly.

31 May 2013

Three Outages Experienced by I/CAD -- NYC's Dispatch System

WNYC, NBC News and others reported that the NYC Dispatch system, revamped at considerable cost, has experienced more than one interruption since its latest release. In the latest incident, there was a period of 30 minutes during which CSR's had to manually write down caller information and hand-carry it to dispatchers. NBC reported that Wednesday was the first day for the software in its current release to go live.
NYC 911 System Room via NBC News

13 May 2013

Software Errors Further Weakened Nuclear Utility's Credibility

How bad is it? That's what Japan -- and worriers worldwide -- asked of its utility, Tokyo Electric Power (TEP) after the disastrous quake and tsunami.  TEP employees no doubt lost family members, and some of them braved unprecedented risks to their own health and safety. No doubt working under pressure and minimum rest, TEP nevertheless had a responsibility to provide accurate reports of groundwater radioactivity, and were forced to rely upon software to deliver those reports.

A NY Times report provided this narrative.
"Tokyo Electric Power said late Thursday that the groundwater near the plant had also been contaminated. The company initially said the radiation was 10,000 times the normal level, then later questioned its own data. It did the same with a report that the level of iodine 131 in seawater near the plant had fallen below 2,000 times the statutory limit, compared with a level of more than 4,000 times the limit on Thursday. On Friday, Hidehiko Nishiyama, deputy head of the Japanese Nuclear and Industrial Safety Agency, said at a news conference that the government had questions about the company’s figures and had asked it to review the data. Tokyo Electric officials said they would recalculate the figures after an error was discovered in a computer program (italics mine). The company has several times issued radioactivity reports only to retract them after experts questioned their validity."

20 March 2011

Glitch or Sabotage? China had Google Guessing

Another reason to favor glitch reduction emerges from a recent investigation Google made into Gmail service disruptions in China. According to a NY Times story, Google concluded that:
“There is no issue on our side; we have checked extensively. This is a government blockage, carefully designed to look like the problem is with Gmail.”
China did not confirm or deny adding to its immodest wholesale censorship of Facebook, Twitter and other sites. Politics aside, the point is that lapses in software quality can become a convenient cover for mischief -- covert or otherwise. 

01 March 2011

Cloud Glitch: Gmail Outage Highlights Big Benefits / Big Risk Proposition

Automation. It simultaneously extends benefits and risks. On the risk side of the equation, small problems can be dramatically magnified. A steady migration of some applications to the cloud has exposed many to this long-understood computing gamble. The latest instance is yesterday's Google Gmail outage, caused by a yet-to-be-specified software failure in Google's storage infrastructure.

Commentary about this outage has focused on the reliance of tape technology to restore Google's backups. I'm guessing that's because consumer backups may have drifted toward disk backups instead of tape (out of economic desperation, rather than a well-reasoned decision). The bigger story isn't about tape. It's not about the flaw itself, though that would be an interesting sidebar. It's about how one of the world's largest technology firms with an essentially unlimited budget for its software development life cycle (SDLC) allowed flawed software to enter production systems.

23 December 2010

Skype Supernodes Self-Sabotaged

From Skype's Website as of 10:26A EST 23-DEC-2010
ZDNet's Larry Dignan reports that some versions of Skype's own software were responsible for taking down what it terms "supernodes" in its version of P2P networks. In Skype's Peter Parkes account of the outage,which ZDNet says affected service for most of December 22, their plan is to create new supernodes to offset the problem. No further explanation of the software cause was offered. 

30 November 2009

Black Friday for eCommerce Hyperlinks at hp.com?

It was to be a busy weekend for HP.com, but perhaps it was too busy for some of its web content managers.  Over the weekend I received several HP small business promotions by email.  When I clicked into the links, selected "laptops" and then "ultra-portables," I got the dreaded "404."  So remarkable was this that I assumed this had to be a problem with me, not the website.  I tried it again several times between Friday and Sunday evening -- to no avail.  None of the category groupings offered up by the landing page from the push email from HP resulted in anything but 404.


If I'd known this was going to last an entire weekend and become "a story," I would have grabbed a screenshot and featured it on errorprocessing.com. But I had no such prescience, and as a result I have no hard evidence to back up this finding.

Instead the only proven result is a statistically insignificant decline in an HP small business push email conversion sales total -- one less ultra-portable purchased online.