4:49am on the Wednesday before Thanksgiving.
As the three or four of you know, I'm not doing my morning radio show this week. I was in New York visiting my sister on Monday... and I don't usually do the Friday after Thanksgiving... so in the middle I decided, at the urging of my wife and staff, to take off Tuesday and Wednesday. I figured I'd use the couple of days to work on creating a couple of streaming audio channels and setting up a projector screen TV on one of the walls of the studio.
As you know from the previous two blog posts, from the moment I stepped off the plane on Monday afternoon, I've been working to solve a problem that dates back 18 months. And that is to establish a non-interrupted, web-based connection between our new swanky studio at the Purdue Calumet Commercialization and Manufacturing Excellence Center and our old, certifiably non-swanky transmitter and tower site in the shadows of a junkyard.
I've been lying in bed here for a couple of hours thinking about it. It's probably a fairly unproductive use of my time as an innovator and creative force (haha) to be lying here on my day off thinking of something as fundamental as solid internet, but that's the reality of the situation. And as I lay here... I'm starting to realize that our most recent problems with Comcast's Metro E service could possibly be related to the difficulties that we've had from the day we started to move our operation seven blocks down Indianapolis Boulevard.t
What does that mean? That means from day one we have had trouble connecting over Comcast internet to our old studio, transmitter and ultimately tower. The problems started when we started at the new studio in the Spring of 2014. And now that I think about it, the entire time there's been drops and machine-sounding audio. With Comcast's Metro E, which we're now trying, the drops are more frequent than they were when we tried, in various forms, to go over Comcast internet, but there have always been challenges.
Now I know what the three or four of you are thinking - how can you run an innovative, world-changing radio station and media company when you don't even have reliable internet? Good point. And it's one of the reasons we moved seven blocks down Indianapolis Boulevard in the first place - to get some reliable internet. Let me explain.
Sometime around 2010, we put a little video camera in the old WJOB studio and started streaming video. We ostensibly did it so that the operations guy at the time, Stew, and I could watch where the many hosts left the dials when they were done broadcasting. When you have a ton of local people come in and do radio shows - as you should if you're gonna carry your weight as a local radio station - then from time to time hosts will leave dials in the wrong spot and you'll wind up with dead air.
Yes, we still use dials. It's a 1970s era Autogram board at the old 6405 Olcott studio... and it's beautiful. It has the sound of listening to late-night AM radio before Ronald Reagan, before the Beatles, before we even made it to the moon.
Anyways, we put the stream of our studio on the WJOB website and zoom - a ton of people started to watch. So we improvised. To stream video in 2010, as the three or four of you probably know, you originate the video at your studio, then you turn it into an encoded digital form, then you send that encoded video over the internet to a streaming company. If someone wants to watch you and your ragged beard talk in the studio, then he or she logs in to the servers at the streaming company and they make a connection that way. It's a drawn-out process, but basically if you want to stream video, even now five years later, you send the video to a big company and they take care of it.
We started streaming with Ustream... then we used Livestream. This was actually before youtube allowed live streaming, so imagine how far back that goes. But back in 2010-12, you had to stream at one of two levels. You could stream for free on Ustream or Livestream, but the quality was poor (low bit rates) and they'd play pre-rolls each time someone logged on.
Ustream and Livestream offered another method, and this is the one we eventually settled on. You could purchase high-quality streaming with no pre-roll commercials for about $1200 per month. That would mean that a viewer could click to watch us in the WJOB studio and what then (2010-11) was a high-quality stream and watch us - via the Livestream servers.
Now the three or four of you might think - why in the Sam Hill would I want to watch me or anyone else sit there and talk on the radio? Good question. But then again you have to understand this area of Indiana and Illinois just to the south of Chicago. We have no television here. Zippo. All of our TV comes from Chicago... and our government comes from downstate. We are lost in the land of no TV and no control of our government.
So for some reason a ton of people started logging on to WJOB1230.com to watch us do radio. The problem became that the $1200 fee to stream to Livestream did not give us unlimited streaming to our growing video fan base. There was this thing called "overages" that nearly took us under.
The problem wasn't that not enough people were watching our video. The problem was that too many people watched. I don't remember the exact money it cost us to stream, but I seem to recall that we were spending $3500 and rising on streaming video - with no corresponding revenue stream. Yikes.
So we decided to build our own digital vertical streaming system and stream right there from our studio, transmitter, tower site at 6405 Olcott St. behind a Chevy dealership and next to a big railroad yard. It worked. It took forever but we built a Wowza-based system in which we could generate video and audio from our studio or a basketball game or even a community event, bring it back to our servers at WJOB, and then create a one-to-one connection with anyone who wanted to watch our stuff.
It was a great technological achievement for a local radio station to be able to do this in 2011. We checked around and we may have been the only local radio station in America who could do this at that time. Everyone else had to send their video to Ustream or Livestream and they would create the one-to-one connection with the viewer.
So here's how we circle back to the problems that we're now having with the Comcast Metro E service in trying to connect from our new studios - where we had moved to partly to get clean internet.
To stream video from our own system, we designed and purchased, at great expense, specially-equipped servers. Three of them. Alpha, Bravo and Charlie. They were beautiful machines. We also purchased several instances of Wirecast software and Wowza software, which ran into the thousands. It was a beautiful, well-oiled system. We were prepared to be a leader in streaming video of radio.
But there was one glaring problem, and if you're a tech head, you see it already. Internet. You have to buy a boatload of internet. You see, every time someone in, say, Crown Point, Ind., would click on the WJOB app (yes, we had designed our own app also) to watch the morning show, they created a direct connection to our servers at 6405 Olcott St. in Hammond. And that used internet. Times that by 10, by 100, and you can see that we could use a lot of internet, even streaming at low SD rates.
So we contracted with a company called Globalcomm, which at some point became known as First Communications. And what a freaking mistake that was. We had to build a second little tower on the WJOB building, run wire to the NIPSCO pole, and then wire all the way to some AT&T hub... just to connect to this high-speed internet service by First Communications. It cost tens of thousands to set up the whole system, and then the internet cost about $1500 a month.
The only problem was, as with Comcast now, the internet didn't work right. We were supposed to get a steady upload and download of 10 gigs. As the three or four of you may know, in sending a video stream to the internet it is the upload speed that counts. And it never reached 10 gigs. We had First Com techs out a zillion times... and they couldn't figure it out. But as with Comcast, they kept billing us. And we kept paying it.
What that low gig upload speed did was limit the number of people who could watch our video. We did the math and we should have been able to take more than 1000 viewers at a time using standard definition of about 750kb... but we could never take more than 250 or 300. When we'd have a big guest come in the studio, or we'd video a big high school basketball game, our servers would crash and no one could watch video.
If you look at the serving up of video from WJOB, back in 2011 to early 2014... again, it wasn't that not not enough people were watching. It was that too many people watched. They crashed our servers.
It might have been different if First Communications could have delivered the 10 gig uploads that they contracted to. We fought them hard and got some financial consideration... but in the end they're part of a huge company and we really couldn't do anything but sue them. And who wants to do that? I suspect that the problem was with First Communications then... and may be the problem with Comcast now... is that our old WJOB studio is located in one of the oldest part of industrial Hammond. That means the infrastructure is all old. The phone lines are old. The cable lines and switching equipment are old. As a matter of fact, the guy who owned WJOB back in the day - Julian Colby - started cable in Hammond. That means that the cable infrastructure that goes in to our studio building may be the first cable lines installed in the city.
Trying to do innovative, creative things amidst smokestacks and rail yards... it just doesn't work. The infrastructure doesn't support it. We have even had huge challenges dealing with the local power company, NIPSCO. They lose power all the time in our area. I believe it's because the transmission lines and switching equipment are just flat out old, just like the cable lines and the phone lines.
So here I am laying in bed in the middle of the night thinking of these things. Instead of running through my mind ways to innovate on radio, I'm trying to figure out what could be the problem in connecting two studios via internet that are seven blocks apart. And I come to this. Comcast, from the beginning, no matter what system we at WJOB have used to enter the internet, has given us drops and degraded audio from the day we started at the new WJOB studios at Purdue. Perhaps it's the same problems that First Communications ran into in trying to give us steady, reliable internet connection to a studio amidst delivery trucks and semis, railroad tracks and smokestacks. There is just too much heavy industry - and too old of communications infrastructure - to provide that reliable internet.
And it may be, because the stuff is so old and there aren't even maps available of some of the lines, impossible to find the exact breakdown in the system. I've got to accept something here. I don't know what it is, but I probably have to accept that if I'm going to continue on this course of innovating in radio... and media in general... then I may not be able to do it in the gut of industrial America.
This is a hard realization to swallow. I have held tight to a dream of innovating amidst the smokestacks, of developing new web-based ways of broadcasting direct from the industrial heartland. I grew up looking to the north at smokestacks along Lake Michigan. It comforts me to see the steam and soot rise from them. I don't mind waiting for trains, which I've done for approximately one-quarter of my life. And the sight of an 18-wheeled tanker hauling oil is music to my soul.
But maybe it's just asking too much to mix industry with web-based innovation. It just doesn't mix. I can't even, after 18 months, get a steady, reliable Comcast connection to go seven blocks down Indianapolis Boulevard. Accept it. Move on.
8:49am - Wednesday before Thanksgiving
So, as you can see from yesterday's blog post that using the Comcast Metro E service to connect our new studio with our transmitter/tower site, we drop to dead air every few minutes. That's how bad it's gotten. It takes about 15 seconds for the Bridge-It Tieline encoding equipment to reboot, then we go back on the air. It's beyond annoying.
As a temporary fix last night, after we did the testing to determine the frequency of the drops (every few minutes), I hooked up a temporary connection using phone lines. This, too, is a bit risky in that the phone line on the Purdue side of the equation is a Comcast phone line... and the Comcast lines have already been down for a few days over the weekend. But at least on the receiving end at the old studio it's a standard AT&T POTS line.
This is a bad way to do radio. It doesn't sound good, as is evidenced by a text I got at 1:23am today from Rick Kubic, who hosts the late-night show on WJOB, "Region After Dark."
FYI I received a lot of texts and emails that the station sounds a bit tinny.
Oh well. At least for now we're on the air. Overnight, I was hoping that somehow Rachel at our third-party vendor could get the attention of someone at Comcast today to analyze the data that Mark Smith and I produced last night... and that Comcast will address the drops in the connection of the Metro E. But here's the email I got from Rachel this morning:
Can we schedule a vendor meet
I will scheduled it for Next week and let you know what they come up with on Monday
The team that schedules this out of office until Monday Morning!
Nice. You'd think I'd again be pissed - Comcast is a zillion-dollar company and there's no one there who can help a local AM radio ensure that they stay on the air over the weekend? For some reason, however, I'm not that pissed. It's sunny out and I'm putzing around the house listening to an old BoDeans album and not much is gonna rile me up... not even Comcast. Here's what I answered.
I don't have any answers for you. As the data from the testing last night shows, we're dropping to dead air every few minutes. It's difficult for us to continue running as a radio station... as it has been for nearly 18 months. It is difficult to understand, however, that if the service does not work - and I have attempted to verify this now by two reliable sources - that it would take a week or so to address it. You would think that Comcast could at some point consider this an emergency repair and take care of it.
This is as calmly as I can put my disappointment. I'm embarrassed that my wife and I put our name behind a local heritage radio station that's gonna sound "tinny" all weekend long... if it stays on the air at all. How's that for a Happy Thanksgiving.
Alexis and I did some final Thanksgiving shopping at Target just now. A gravy bowl, a turkey serving plate, a few plastic serving spoons, and a set of 600 thread count sheets for the bed. Don't ask.
So on this my day off the day before Thanksgiving, I really didn't want to deal with any Comcast people. Just not today. As I stepped off the treadmill at Planet Fitness, however, Ray from Comcast asked if I could come down to the old studio and let him and his guys in the building. I was a little surprised at Ray showing up on the scene with a crew in that this was my last communication with Rachel.
I can certainly try to have my IT guy available, and myself... but after thinking about it, unless there's a concerted and serious effort on Comcast's part that comes from the top - preferably Kevin Connolly - then I don't have a lot of faith that any haphazard group of Comcast techs is 1. going to believe us that there is a Comcast problem and 2. going to be able to fix it. We've met with many techs already.
What I'm most concerned about is that we are broadcasting a Whiting High School state finals football game on Saturday night, and for various reasons, we have to accept the feed from Indianapolis at the new studio and then send it to our old studio. I'm working now to set up an over-the-air system to accomplish that. If you think it's worth setting up a meeting with Comcast while I'm working this other method, let me know and I will try to make Mark and me available.
In a word - what I didn't want to happen happened. I wound up spending a good portion of the afternoon walking around our tower and transmitter site with several linemen from Comcast. Nice enough guys, but once again I had to explain the problem to a whole new cast of characters. To resolve this problem, I suspect, it will take more than a group of linemen, however astute they may be.
After that, I got on the line with Bill from Tieline. Now this is a company that has its act together. Bill and I once again worked through several scenarios, taking a good hour to try a workaround through what we diagnose as a bad internet connection. Really what it boils down to is you can put something called a "jitter fix "... a kind of delay on the signal of five or more seconds that tries to compensate for low quality internet. We tried to get it to work, but I have to update the firmware on the Tieline Bridge-It box first. I'll do that on Thanksgiving morning.
On another front... I talked today with several techies about how to set up a wireless connection between our new studios at the Purdue center... and our old studios and transmitter site next to the fence company. There's Marti, Ubiquity, microwave dishes... a whole assortment of methods to send wireless audio seven blocks. I prefer not to send data packets, as we're now doing on Comcast. You gotta break the analog audio down to digital... send it on the internet (or wireless)... and then decode the stream back to analog on the receiving end. I don't like that. I want to send the audio directly, without changing its form.
But in my infinite wisdom, I gave away a key piece of equipment that would allow us to do this for next to nothing. As a matter of fact, whenever someone over the years has asked me for a piece of equipment lying around the old building... I pretty much let them cart it away. "Less junk," I say. And several times that has come back to bite me in the ass. So it goes with the case of the Marti receiver.
The Marti system has worked great for WJOB for generations. The station owns a few frequencies on which we could go to a game, set up a mini-transmitter (Marti unit), connect it to a small antenna, and send the audio back to the station and up the tower. We have the transmitting Marti unit, a few of them actually, and we still own the frequencies... but I gave away the receiving unit. Someone was doing some work in the back room - I believe it was one of the FBI techs that used to come service their units that were on the tower - and the guy asked if he could take the old Marti receiving unit off the wall.
"Sure, go ahead." And the guy took it. He might have even given me 50 bucks for it. Either way, the FBI tech knew the value of what looked like a piece of junk to me. I'm trying to buy a Marti receiving unit now and they start around $2500. Oh well. The same FBI guy bought a weather vane off of me from the top of the building for, like, 50 bucks. A year later an old weatherman - I forget his name - stopped by the station and asked about the weather vane.
"That was worth some bucks, you know.," the guy said. "A real collectors item."
Needless to say, I try not to just give stuff away anymore. Some of the stuff like record players and old mixing boards, filaments and tubes, an old transmitter, an antenna unit the FBI left behind... those have value. I'm not sure what that is, but as I've learned the hard way, it's more than 50 bucks. Good night to the three or four of you. Have a great Thanksgiving.