Lots of people have heard of the “Three Mile Island” nuclear power plant accident of 1979.  Usually because they’ve seen, or heard of, the 1979 disaster movie, The China Syndrome, which actually preceded TMI but benefitted mightily from the TMI event happening twelve days after the film opened. It’s a reference to how if a nuclear reactor’s core ever experienced meltdown, it would get so hot it would melt through the earth, and keep sinking until it went through core and out the other side – in China. A fanciful concept, indeed, as it’s not true in several respects…but it helps illustrate the horror of how bad a nuclear meltdown could be.

Why’s this relevant to the Engineer’s Corner? Because TMI is famous among engineers for being a “normal accident“. 

What’s a “normal accident”?  It’s an event in which there wasn’t any one thing that nearly caused a meltdown of catastrophic proportions. Instead, it was an intricate chain of little events inside a highly complex system. Each event was the byproduct of “normal” operations, and none of the events, by themselves, was terribly problematic. The catastrophe arises because they all happened in a particular order within a particular timeline. The idea is that even though the likelihood of all those events happening…in that order, and in that timeline…is very, very small? It becomes more and more inevitable the more complex the overall system is.

Well, today RIPR had its own little version of a “normal accident” as well.  As listeners to Fresh Air from 3 to 4pm noticed, we had to substitute a rebroadcast of WBUR’s Radio Boston and, later, the BBC World Service. 

To begin, we use a 12ft diameter satellite dish to get all our NPR, BBC, et al programming. It’s located out in North Providence mostly because there is physically no room to locate it at our studios in 1 Union Station…it’s a pretty big dish!  To get the audio from the satellite decoders back to our studios, we lease a T-1 line from Windstream/Verizon. A T-1 is a special data service; think of it as 24 telephone land lines bonded together into one big line. It provides 1.5Mbps (1500kbps) of bandwidth, and although it’s based on technology dating to 1962, it’s generally quite reliable. Often moreso than modern IP-based systems.

But recently, we’ve had problems with the T-1, and today we think we have finally tracked them down to a bad “C Card” in the T-1 circuit that Verizon replaced at 4:06pm.  But at random times for the past few weeks, that T-1 would just mysteriously fail for a few minutes, sometimes an hour, and then “fix itself.”  Whenever it failed, we’d lose all our satellite feeds…live and pre-recorded…for that period.

So even though the technology is usually reliable, that T-1 is so important to our operations that we naturally have a backup system, just in case. It allows us to get programming from WBUR’s satellite dish system up in Boston. But today, for an unknown reason, the subscription to Fresh Air on the backup system was unsubscribed…so dead air on the backup.

Because we believe in the “belt/suspenders/unitard” approach, there’s actually a third way we can get programming, and that’s by manually playing a webcast off NPR.org.  But for an inexplicable reason, every time I tried that…after a few minutes an audio feedback loop would kick in and the webcast would crash.

With literally no other option on the table, we switched to Radio Boston, which – fortunately – was airing a special with Bill Littlefield, host of Only a Game, which is celebrating their 20th anniversary today. While it’s not Fresh Air, at least it was something that many of our listeners probably found interesting.  We later managed to get our backup feed functioning enough with the BBC World Service, and changed to that.

The good news is that we’re hopeful that the problem with our main T-1 has been found and fixed. The bad news is that these events have, in part, demonstrated that despite the fact that a T-1’s “robust legacy” technology means it doesn’t break much…it also means that when it does break, it can be very difficult to get it fixed.  Accordingly, I have been researching modern solutions to supplement the T-1 for months: MPLS dedicated internet paths, and/or fiberoptic point-to-point solutions. Now I’m looking to them to replace the T-1 rather than simply supplement it.

I also was spending a lot of time trying to find a viable wireless Ethernet bridge solution, but unfortunately there’s no way to achieve line-of-sight paths from our studios to the North Providence site…not with an unacceptable number of towers needed to see over/around certain hills. Despite many strong efforts to find one!  That final attempt was just last week, in fact.

In the interim, we appreciate your patience with us and we’ll keep doing our best to avoid issues like this in the future!