29 October 2015

Double-Barreled Glitch Reflecting Redundancy, Resilience Failures Rocks 2015 World Series Broadcast

A extended power failure killed video for game 1 of the 2015 World Series

It wasn't the only glitch  that night, but the one with the widest impact by far was the power failure that reportedly took the Fox Sports U.S. video feed offline during game one of the 2015 World Series.

As noted by a Wired post, for Kansas City area residents, where the game originated, Google Fiber suffered a glitch during the game before the Fox Sports outage, leaving local fans without video for the game, the timing of which Google itself tweeted "couldn't have happened at a worse time." 

But that was overshadowed by the Fox Sports fourth inning glitch, which Fox later explained as "a rare electronics failure caused both the primary and backup generators inside the Fox Sports production compound to lose power." The outage lasted around seven minutes, according to the Wired report, after which Fox switched to the international video feed. It also failed briefly, but was restored quickly. The problem was resolved in around 20 minutes and Fox resumed normal operations.

How can an experienced network broadcaster, which has broadcast hundreds or perhaps thousands of games experience a failure of its redundant systems? Especially on one of its most-watched games of the year? Ratings for the game were the strongest for a World Series game since 2009

The need for regular testing -- as well as questions about the reliability of switch-over technologies -- arose during Superstorm Sandy and previous extended power outages in the New York area, As Computerworld later reported, 
Devastation caused by Hurricane Sandy forced at least two major hospitals and a data center in lower Manhattan to resort to backup generators fueled by diesel for power.In these cases the backup processes failed.

Image credit: Fox News, Wikipedia (BBC truck shown) 

No comments: