Your Intrepid Engineer is the sole full-time tech person for The Public’s Radio, meaning I have responsibility for a lot of physical locations with a lot of hardware in them: a main studio, five AM/FM broadcast locations (at the time of writing, Dec 2021, it’s also soon to add another three, possibly four), two STL relay sites, and three remote news bureaus.

Standardization isn’t just a good idea, it’s a hard necessity here. Fortunately, tools have come to the forefront that make life a lot easier in that regard. Here’s an overview of what I do to keep things organized. I wouldn’t necessarily recommend you copy it directly, but instead use it as a springboard to adapt to your own situation.

Google Docs

The savior of many in this pandemic, I’ve been using the heck out of Google Docs (technically Google Workspace) for several years before that:

• Google Docs. Technically my least-used part of the Google architecture, there are a few things I keep here mostly because I’ll have to update/edit them on a very irregular basis and it’s easier to start from the last-known-version than restarting from scratch: studio door labels, transmitter site visit checklists, parking signs, keybox legends, breaker box legends, Chief Operator notification signs, etc.
• Google Sheets. The big one! I have spreadsheets that organize each local IP address of every site (and all the exterior IP’s too), spreadsheets for organizing the remote control setup at each location, and for tracking the 15 BRIC Links we have at fixed locations across the system. I’ll expound more on this in a minute, but for now the key is to set things up so it’s easy to standardize across sites. To make it so every site’s primary STL is 192.168.xxx.30, and you can see that just by switching from worksheet to worksheet. I open these Sheets never less than weekly to refer to and/or update things. There’s other things that’re handy too; I use Sheets extensively to document equipment rack topography and to plan changes. Also to keep a current version of the program schedule handy.
• Google Drive. Perfect for uploading PDF’s/DOC’s of all your equipment manuals. Software drivers. MIB files for SNMP, HD Radio Importer Keys, etc. Keep copies of images that’re handy here, like pics of various things at your site (think annual inspections), and of your station logos, too! Also some audio files, like legal ID’s.
• Google Chrome. Specifically the bookmarks. Every location gets every network device bookmarked, with user/pass info included in the bookmark. Those bookmarks are sync’d across all the browsers logged into that account, making accessing any remote site from another remote site a heckuva lot easier.

You don’t need to use Google Docs specifically. I imagine Microsoft OneDrive can work just as well, along with a bunch of other cloud services. I use Google Docs because it’s:

• Free, although if you need a lot of drive space (maybe for archiving skimmer audio?) you can easily pay a little each month to get a lot more.
• Nicely integrated with Chrome, our browser of choice.
• I can “sign in” to the Account in Chrome on each of the remote site’s desktop computers, meaning it’ll save & sync passwords, links and bookmarks across every machine.
• Pretty ubiquitous these days; even if someone isn’t a regular Google Docs user, they’re at least familiar with it.

Two Factor Authentication: By the way, I shouldn’t have to say this but I will: if you’re going to store a lot of sensitive data in a Google Account, you damn well better turn Two-Factor Authentication (or Two-Step Verification / 2SV as Google calls it) on for that account. And don’t use a “text me a code” for 2SV; that’s not very secure, either…use the Google Authenticator app.

Or alternatively, consider using LastPass or any one of a dozen different password management systems.

Standardizing Your Hardware Makes & Models

While a lot easier to say it than to do it, it really does help to standardize as much possible on what hardware you’re using. You don’t have to keep multiple troubleshooting lists in your head for a bunch of different equipment manufacturers.

Transmitters. By and large, TPR is a Nautel shop. We do have a few exceptions; but Nautel is the rule: WNPN Main = Nautel GV15 & NV20 (89.3 Main & Aux), V7.5 & VS2.5 (102.7 Main & Bkup), and P400 (WPVD Main). I expect for our new FM sites we’ll be buying more GV and VS series transmitters, too. This isn’t meant to be a commercial for Nautel, but rather that it’s best to pick a particular manufacturer and stick with them. The simpler your networking will be. The simpler your remote control & telemetry will be. The simpler your tech support issues will be. And the more you can command the OEM’s attention when there’s a problem because you’ve given them more money.

Remote Controls. I cannot emphasize enough how useful it is to standardize on your remote controls. We mostly use Burk ARC Plus Touch systems (leaning heavily on their SNMP-PLUS package) and trying to integrate the handful of locations that don’t have a Burk remote control system is a huge pain in the @$$.

Studio/Transmitter Links (STL’s). This is tricky, because despite my best efforts it has proven incredibly difficult to truly standardize on things here. Mostly because we decided to go the route of putting as much hardware as we could at the transmitter instead of at the studio. A lot of engineers believe in the opposite, but I don’t like the lack of flexibility (esp in COVID19 times). Also, since Providence is a PPM market, there are distinct advantages to keeping all your audio processing gear at the transmitter. Even so, all our STL’s are IP-based and work on one of three technologies:

• LiveWire: the rarest; we only use it for our main 89.3FM & HD2 signals’ primary STL, and I guess for the satellite feeds from the satdish at WPVD, back to the studios at 1 Union Station. Requires a very robust network, but has very low latency and high audio fidelity…and is very flexible.
• Worldcast IP Codec (aka Horizon NextGen): for a while these were my primary STL’s for most of my sites, but as they age I’ve seen them get increasingly flaky so I’m starting to phase them out. The SureStream “bit-splicing” technology to spread the connection across two ISP’s is really nice, though.
• Comrex BRIC Link-II: I’ve been a big, big fan of Comrex’s gear for many years now. It doesn’t hurt that their factory is less than two hours’ drive from me. We’ve used them extensively in both primary and, especially, backup audio paths for over a decade and they work great…especially since they introduced the “bit-splicing” multi-ISP functionality they call “CrossLock“.

Even so, I’ve still managed to, a bit loosely, “standardize” on how the STL’s work:

• All sites use AES3 digital audio unless and until it must be switched to analog (if ever).
• Most sites have a Broadcast Tools AES Silence Sentinel to automatically switch between main and backup STL audio. For 89.3 this is a bit complicated since the main and aux sites are two different locations. At the main site, I have LiveWire as STL1, and a Worldcast IP Codec as STL2, for the FM/HD1. The HD2 gets LiveWire as STL1 and a BRIC Link for STL2. But at the aux site? I just used BRIC Link-II’s on two ISP’s; there isn’t a redundant audio path. I figure it’s already a third-level STL backup at that point.
• Lately for my other sites (besides 89.3) I’ve been moving more towards a model of having an Inovonics 655 as the main STL switcher (it’s a little clunky, but you can manually control it with SNMP, too).
• STL1 is a Worldcast IP Codec or BRIC Link, via the AES input on the 655.
• STL2 is the http webcast player built into the 655; it connects to a separate BRIC Link back at the studio (or if that’s unavailable, it tries the BRIC Link at the satellite dish facility, which is fed via a Backup Enco DAD in the same building).
• STL3 is a radio tuned to our main 89.3FM signal; a good thing to have if things really go to hell in a disaster and local telco/ISP’s because unviable.

This isn’t to brag on how much redundancy we’ve got (well, maybe a little!) but to point out how important it is to think every aspect of this process through from start to finish. Getting into the details matters. You need a system that can be (as much as possible) applied in the same way to every location. And in my case, I also needed something that could and would work automatically without any human intervention. So I also needed to standardize on rules/procedures for how those things would happen automatically….lest I get an avalanche of automated calls/emails from multiple monitoring devices that’re all because of one point of failure.

Organizing Your Network

This is the big one. Each location gets a TP Link gateway/router (mostly TL-R600VPN‘s, but those are being phased out in favor of the ER605 that, on the one I’ve used so far, seems pretty much the same). Each gets a SFF desktop computer (with UltraVNC for remote control, since only UltraVNC natively supports encrypted passwords). And that computer gets a Chrome browser set up with access to the same Google Account that all my Sheets, Drive, Docs, etc so they all can access it as needed, quick and easy. Each gets a L2TP VPN set up that I can connect my iPhone to for VNC purposes. Each gets a Foscam R2E ptz webcam so I can look around if needed.

But most importantly, each site gets a worksheet in my Google Sheet “IP Address Schema”.

All the transmitter sites are 192.168.xx0.yyy where the yyy octet is consistent from site to site but the xx0 octet is unique to that site. Similarly, the remote news bureaus are 10.1.xx.yy. Also, FM translators are 192.168.xx5.yyy where “xx” is the same as the primary station. Livewire is 192.168.087.yyy, and Livewire-adjacent is 192.168.088.yyy

Then I break “yyy” down by group:

• 1 – 9 is network. 1 is the gateway, 9 is wifi access point.
• 10 – 19 is transmitters. 10 is main tx, 14 is backup tx.
• 20 – 29 is remote control. 20 is the R/C itself.
• 30 – 49 is STL’s. 30-36 is the main STL’s for FM, HD2, HD3, HD4, SCA1, SCA2. 40-46 follows the same pattern for the backup STL’s.
• 50 – 69 is audio. 50-54 are processors for FM, HD2, HD3, HD4. 60 is a Justin 808 (if used). 68 is EAS. 69 is RDS.
• It’s worth noting that if you have a Livewire-based processor with a single NIC, like an Omnia VOLT? The pattern still holds, even if the third octet is “87” for Livewire.
• In our case, though, even though our Livewire network spans multiple locations we treat it as one local LAN across them all. Your network may or may not do that, so YMMV.
• 70 – 79 is HD Radio. 70 is Exporter (or Exporter/Importer). 71 is Importer. 72 is Exgine.
• 80 – 89 is monitoring. 81-85 are radios for FM, HD1, HD2, HD3 and HD4. 89 is RDS monitor.
• 90 – 99 is webcams. 90 is the NVR/DVR, 91-95 is any local NVR camera. 96-98 is for the Foscam PTZ camera(s).
• 100 – 129 is “Utility”. Usually this is for web-controlled power switches and UPS/battery backups. Also for a generator, if needed. And also for KVM switches.
• 130 – 139 is Computers. 130 is the main desktop computer, 135 is a second desktop computer. 139 is usually a printer.
• 140 – 149 is WiFi. We make extensive use of UBNT wireless ethernet bridges, though.
• 150 – 199 is reserved for local DHCP.
• 200 – 225 is usually VPN-related DHCP addresses.
• 226 – 255 is “reserved for special cases”

The Google Sheet itself has many columns of information, from left to right.

• IP address
• Group (whatever you want to call it)
• Name (whatever you want to call it, or use the NetBIOS name if there is one)
• Make
• Model
• MAC address
• Serial Number
• External IP address 1 (useful when dealing with devices that must be “visible” from outside the firewall)
• External IP address 2
• External Port (used for port forwarding/translation, if you do that sort of thing)
• Internal Port
• Location 1 (can be a street address, but more just a reference so you can find it if it’s not in a physically obvious location)
• Location 2
• Username
• Password
• DNS1
• DNS2
• Notes (often handy when the web GUI requires a specific URL)

The worksheet for external IP’s is somewhat different:

• Location (shorthand for the facility in question; call letters are fine here)
• ISP (company name)
• IP address
• Subdomain (I’ll come back to this)
• Description (whatever you want)
• Address (make sure this matches whatever address the ISP has on file)
• Gateway
• Subnet Mask
• DNS1
• DNS2
• ISP helpdesk number
• Account Number
• Username
• Password / PIN

I cannot possibly overstate the importance of having the tech support number, account number, and username/password (or PIN or whatever) for each ISP connection. When you’re in the field, you’ll never have the paper bill handy to refer to that stuff, and for many ISP’s they require it. At least with Google Sheets you can call it up on your smartphone while on the call to tech support.

The subdomain thing is this: I registered a specific, short domain name just for engineer’s usage, and I create A NAME records for it so each IP address has an easy-to-remember subdomain. If nothing else, I can quickly “ping” the subdomain to get the IP address.

Organizing Your BRIC Links

Obviously this one is a bit more hardware-specific, but I suppose the principle can apply to a lot of audio codec deployments. The point for me is, really, is that it’s very useful to have BRIC Links “names” that tell you what the codec’s purpose is; that way you can glance at the “connections” page and know immediately what’s going on. That’s handy when you’ve got over a dozen BRIC’s in play.

Side note: sign up for Comrex’s Switchboard service. It is worth every penny in helping keep track of everything. Doubly so if, like TPR, you have several portable Access-NX codecs out in the field, too.

Achieving a naming convention like the above is difficult enough, but you also want something that’s simple and easy to remember for non-technical staff. So I came up with the following “formula” for naming my BRIC’s:

• Common name. I googled “four letter baby names“, and scrolled through it to pick names so that there’s one for each letter of the alphabet: Alba, Bryn, Cleo, Dora, Edna, Faye, etc, with a focus on names easy to say, easy to spell, and easy to remember. Those criteria seem to favor traditional women’s names, oddly enough, but your mileage may vary.
• Wherever possible, I create a subdomain of “commonname.domain.org” for easy reference of the IP address via ping.
• Location. A three or four letter abbreviation for a description name of where the codec is located. Call letters can work, although I don’t like them because call letters could imply the studio end or the transmitter end of the connection. I tended to use things like “WSTY” for “Westerly” or “1US” for “1 Union Station”.
• Purpose. I decided to limit this to three categories, but modify to suit your tastes.
• STL = Studio/Transmitter Link. 24/7 nailed up operation.
• PGM = Programming Source. Similar to STL, but doesn’t involve a transmitter site. Usually an offsite backup studio or something like that.
• P2P = Point to Point temporary connection. Interviews, pledge drive shifts. Refers mostly to the two Access rack units at the studios and the BRIC’s out at the remote news bureaus.
• Main/Bkup. Just what is says: is this a primary (“main”) connection, or a backup (“bkup”) aux connection. That can be important because you’ll want to make sure you don’t accidentally kill off both the main and backup connections at the same time.
• Feed. Mostly for distinguishing STL feeds. We use FM, HD2, HD3 and AUDIO (general purpose).
• TX/RX. Is this a connection that’s one-way transmitting (TX), one-way receiving (RX), or bidirectional (TX/RX)? It’s a powerful feature of BRIC Links that you can have multiple RX-only connections to a single source BRIC Link, even with a single TX/RX connection!

So that’s how we got names like ALBA-1290-PGM-BKUP-AUDIO-TX, for example.

The spreadsheet that tracks all these has the columns:

• Class. Usually just “engineering” and “production”. Is this a codec used only by engineering or more general usage by the staff?
• Location. A more detailed physical address.
• Model. Self-explanatory. If you have more than just Comrex gear, you should add a column for “Make” as well.
• Username / Password. I usually use the same password for all of these for simplicity’s sake, but if you’re not going to do that, you definitely want to track all the passwords here.
• IP address 1. Your external IP for ISP1.
• IP address 2. If you’re using CrossLock (or any multi-ISP bit-splicing tech) here’s the second external IP.
• MAC address. Always track the MAC! Also BRIC Links use MAC addresses in regards to both CrossLock and Switchboard.
• Purpose. This technically is covered by the Name you create (see above), but sometimes a few extra words for clarity can help.

The rest of the Sheet is columns from “Common Name” onward, as listed above.

Organizing Your Remote Controls

This part is highly organization-specific, but I’ll just say that I learned that I needed to standardize on this stuff the hard way. We had an old Burk ARC16 that was retired and I got the upgrade adapter to make the IP8 panels work with a new Burk ARC Plus Touch (aka ARC-PT). I set up that site first and I mistakenly tried to set it up so it more-or-less matched the arrangement under the old ARC16. Pretty soon I realized I would’ve been better off starting from scratch, because the ARC-PT is just so different (and more powerful) than the ARC16 is.

Then I tried setting up another site (we got a grant so I got four ARC-PT’s at once) and quickly realized I had created a giant mess. If I wanted these ARC-PT’s to all talk to each other, I needed to start over from scratch….this time with a plan!

So back to the drawing board (actually Google Sheet) I went and spent a lot of time dreaming up every possible meter, status and relay input/output that might go into an ARC-PT. First in terms of physical I/O, then virtually using SNMP.

Then I went back and started organizing and editing things. I decided, somewhat arbitrarily, that it would be a good idea to keep all my “display” I/O in blocks 1 thru 99, preserving 100 thru 199 as “actual” measurements. That’s handy when certain real measurements are provided by the device in metric (or Celsius), but you want to display them in imperial (or Fahrenheit). So you take your reading in the 1xx slot, then the 0xx slot is a mathematical formula derived from the 1xx slot. Blocks 200 – 250 are reserved for items that are unique to a specific site, and 251 – 256 are just “reserved”, period.

FWIW, I’ve noticed that many engineers like their remote controls to display information I just don’t think is terribly important anymore. Getting into the current and voltage of all the PA’s on a modern FM transmitter just feels…pointless. If you really need to, you can always log into the transmitter’s own web GUI to get those readings. But how often do they really provide a useful diagnostic tool these days? On older transmitters? Sure, that makes sense. But not on anything made in the last ten years. Similarly, I don’t include relay controls to raise/lower the power setting; I just have on/off. Modern TX’s don’t drift in their power settings, so nobody should be messing with that remotely anyways.

Beyond that, obviously other makes/models of remote control will have different numbers of available slots. So you’ll need to adjust your Sheets to accommodate that.

Trying to write out all the various items in our Sheets would be ridiculous, so here’s a link to a generic Google Sheet based on our system, that anyone can view. Take a look and adapt as needed to your own situation.