Round and round the testing goes...
As new services make their way into the cable pipeline, the demand is building to develop comprehensive, effective and efficient (in both time and money) testing and measurement equipment, procedures and protocols.
While management keeps looking over their collective shoulders at the competition forming fast on the horizon, they're putting increasing pressure on their engineering professionals to get cracking on rolling out data communications (with or without cable modems), advanced analog or digital set-top deployments, new premium services, or God forbid, IP telephony!
As a result, engineers in the trenches, in all different sized systems, are struggling to stop leaks, block ingress, maintain demanding signal levels and deal with a whole host of related problems to keep their systems competitive.
CED queried a randomly-selected cross-section of engineering talent in the industry, from very small to very big systems, to see what common concerns, solutions and crystal ball predictions they might have when it comes to improving their systems' performance in this newly competitive environment. One operator, in fact, has taken the advanced testing task to heart and embarked on an aggressive effort to develop his own testing software/hardware product.
Respondents to this informal survey included:
Bill Bauer, owner, WinDBreak Cable, Harrison, Neb.
Saconna Blair, director of network management, Jones Intercable Inc., Englewood, Colo.
Jim Dupler, Midwest regional engineer, Rifkin & Associates, Mt. Vernon, Ill.
Hung Nguyen, senior staff engineer, Time Warner Cable, Denver, Colo.
Oleh Sniezko, director of transmission engineering, TCI Communications Inc., Denver, Colo.
Larry Warren, senior staff engineer, InterMedia Partners, Nashville, Tenn.
What follows is an edited transcript of interviews conducted individually.Testing concerns
CED: As far as testing goes, especially with all the new services being introduced into the cable pipeline, what are your biggest testing hurdlesshort-term and long-term?
Blair: As far as short-term, we're really most concerned about the basics. When I say basics, we just like to make sure that every technician has equipment that can measure signals accurately and quickly and make assumptions. We still see the RF analog signals we're sending down as our primary concern. Long-term, there are probably two things. One is the return (path), and we just really don't know exactly where we're at with that. It's fuzzier, and it's more of a concern in the long-term. Second would be digital signals.
Dupler: Right now, we're doing a pretty good job on our forward plant. We've already got DMX and Sega and so forth out there. We've found a few little problems, but we were able to correct them relatively easily. Data brings its own problems.
But, I think if you just keep to the basics, keep on your leakage, keep an eye on your FCC proofs and everything, it's going to fall in line. I know we're going to start in our reverse path once we activate it. When we find the new business, we're going to go ahead and put the filters on the homes (that are) not taking any kind of reverse product.
Nguyen: Short-term and long-term, I think they're both the same. The biggest one is the return path testing. We're trying to develop a test procedure and trying to understand the effect of the lasers. Those are the big things right now. Both in the plan and the electronics. We really don't have any problem with the forward (path). We're doing OK with that.
Sniezko: Short-term is having inexpensive equipment to replace the link analyzer from HP. And the way we are doing this is trying to use sweep systems, but they don't give you full answers.
Long-term is to have a non-disruptive way of testing all the parameters we are interested in in the reverse path, including how much dynamic range or how far from distortion levels we are.
Warren: One (concern) is just getting our people, our technicians, up to speed on all the testing procedures they're going to have to learn. They're all used to using signal level meters, and some of them use spectrum analyzers and those types of things. It's going to take a different mentality I think to test for these new services. It's just a matter of training.
I don't see any transition problems. It's going to vary from system to system obviously. Some of the larger systems have people who are stronger technically than others. And it will be easier to bring those people up to speed. And we'll have to spend a little more time in those systems where we don't have a real strong technical staff.
Bauer: Short-term, my biggest hurdle is in ingress testing and figuring out what is causing my cable modems to slow down. The problem was that I couldn't come up with any way of finding out what was causing it. And, the only pieces of testing equipment that were available to me were $25,000 spectrum analyzers.
I assumed I had nice, clean plant. And the reality was that from a spectrum analyzer standpoint, I had a nice, clean plant. But that's not what was causing the problem. What was causing the problem was ingress that was very short duration, broadband, with a lot of energy in it. They were hits of several microseconds in length, but they would wipe out everything. Data would go through, but if you had these hits on a regular basis, you'd just be taking out a whole chunk of the data that was coming through.
The hits were frequent enough that it was probably slowing down my network by 20 to 30 percent. So, my short-term need was a piece of inexpensive equipment that told me what the problem was.
And that's what we've been working on to develop. (Currently) it's not everything I want because it's not in a nice handheld package, but it is inexpensive enough that I can afford it. It uses an off-the-shelf 60 MHz digital storage oscilloscope, my laptop computer, and the software we're developing. But it shows me what my problems are.
Once I've figured out what the problem is, I want to know where it's coming from. We have things that we're working on that will tell us that, but they're pretty preliminary.
Beyond that, I would like to understand that if I send out a packet of information, how many retransmissions did it incur before it got back to me.
That's the ultimate thing that I'm after here. I want to send out a packet of information, and I want it to bounce back to me, and I want to know whether it took some hits on the way. And I want it in a $2,500 handheld package that I can take anyplace.
I'd say we're 80 percent there. With this piece of equipment we've got and the other part that we're working on that identifies it, the last part would be to incorporate the cable modem analysis statistics into this.Is the equipment there?
CED: How are you fixed for test equipment? Do you have what you need? If not, why not?
Dupler: Yeah, I think the equipment is available. I think the one thing we're going to have to invest in is reverse sweep equipment. But that's coming on the market hot and heavy now.
It's pretty pricey. We're actually supposed to demo one vendor's unit here soon to see how well it works and everything. There are only two that I've actually looked at. One is in the $40,000 range, and one is in the $18,000 to $20,000 range. So there's a big jump there. One unit I believe is specifically for the reverse, and the other has both the forward and reverse built into it. And I think that's why it is a little bit more pricey.
Sniezko: It's available. But it's not affordable to put in every place that you would like to have it. It's available from a number of vendors, and it's becoming more affordable. And I think the more people work on this one, and the more standardized processes become, somebody will come up with a good piece of equipment.
Warren: In some cases, it's available. In other cases, it's not. We don't have all the equipment we need to do adequate testing. And that's primarily due to a lack of competitive pricing on some equipment. However, most of the equipment we have in house will do the tests we need to do.
Bauer: With this new piece of software, I have what I need today, except for the problem of where it (ingress) is coming from in the plant. And because it's not in a nice handheld package, it gets very difficult. We've been doing some testing with what we call a 'cable clothespin.' So, at the pole, we can actually do testing on each of the individual lines. We go to each drop and send a signal down to the house. If there is a broken cable, we get very high ingress coming back. If it's a good, clean signal, we get very low (ingress) coming back.
The other thing that we're doing is that we can send pulses down the hardline and see if we're getting reflections back. In one of those cases we did and we found out that we had a seizure screw in the blind terminator that had never been tightened.
Once we saw what it was, and we did the calculations as to how many feet it was from where we were, it was easy. But, there is no way we would have ever found it without this software.
Now, a TDR (time domain reflectometer) might have been able to do it, but you have to physically disconnect the system to do that. And this was a non-intrusive thing we ran right over the top of everything else and saw the reflection come back.
The interesting thing is that our oldest, most reliable standby test equipment, i.e., cable customers, are not doing their jobs anymore. We've always relied on them when something wasn't working. The cable customer called us. Well, that won't do in today's market. What's happening is, if they're displeased, they're not calling inthey're changing service. And so, we've got to be much more proactive on our testing.Lessons learned from testing
CED: Do you have any guidelines or rules-of-thumb that repeated testing over the past few years has taught you when it comes to establishing better performance in your system?
Nguyen: We have not arrived at one solution. We're trying to right now. Our Road Runner cable modem group has a different test procedure to do the data, and then on the cable side, they have another procedure. We're trying to tie them together. I think by the end of this year we should have it.
Dupler: It's (testing) actually made us better. I know I wasn't very popular when I put DMX in the FM band. Especially if you've got a strong FM station around, it will knock out a block. My philosophy behind that was it will make us go fix the problem. And 99 percent of the time, it's a drop problem or an in-the-house problem.
There are an awful lot of people out there, especially in smaller systems, that if they have capacity, will stay away from channel 19 and 20 because of pagers. They just don't want to deal with it. But it makes you clean the problem up. And you can fix multiple problems when you do that. It just makes your system much, much tighter, and we've noticed our CLI, each year, has gone down. The FCC tests are a pain. But it has made us get much better.
Bauer: The rule-of-thumb we've discovered is that everything is terminated. You leave no open ports. No unterminated taps. No unterminated anything, any place! And that covers everything in the headend.
You have to take those extra steps. That means you use headend cable. You use all the right fittings. You terminate everything, every unused port on every splitter, on every piece of equipment. Even to the test ports on the trunk amps and the feeder amps. You throw away the caps that they have, and put terminators on it. Get it closed up tight. You do anything to drive down any radiated RF. That's probably what's done the most for a clean system for me.
Use good fittings. Don't go cheap. Don't rely on the weather-proofing of a vendor-supplied fitting. They're great and do a fantastic job, but put the heat shrink on. Seal that up. It's not that the fitting is going to deteriorate in a short period of time. It's the long-term (conditions) that cause the problems. Go the extra measure, put the heat shrink on and seal everything up.
Blair: Really, first and foremost, it still comes down to looking at just the levels alone. One of the first things to do is look at end of line. What does a customer see? Carrier-to-noise would be the second thing.
We're looking at what is non-interfering to the customer. Sweep response is no longer the same critical thing that it was six years ago. It has a different purpose now, and it doesn't have to have the same resolution over the entire bandwidth. You still have some issues up at the roll-off part.
And then, I can tell you the one thing that comes in over and over, and that is proof of the picture.
An FCC proof of performance looks at seven channels spread across the band. In one swipe through a television, you can see if there are any distortions there. And the one nice thing about carrier-to-noise is that noise is visible. But noise is more subjective than beats are. So, a measurement of noise tells you what the system is performing at, if you're at the high end or low end. Those RF levels tell us everything about the system and proof of the picture.Is remote testing viable?
CED: What are your thoughts on remote or automated testing? When do you see a big degree of it happening in your system?
Warren: We would love to incorporate that into our systems. But, the biggest problem with it today is that it's still pricey. There are different companies who manufacture equipment you can place at the end of the line that is strand-mountable. It will measure carrier-to-noise performance and distortion, QAM modulation and those types of things. It's possible that we're going to purchase some of that type of equipment down the road. We don't have any of it today. But, I can see a very useful function there. If it were easily movable from test point to test point, and if the costs were reasonable, it would be very useful too.
I think we'll evaluate it, and when we do, I think most seriously it will be along the lines of automated reverse monitoring equipment. There are three or four companies today who manufacture equipment that will allow you to take the output of an optical return receiver, route it into a switch and it will monitor ingress levels, carrier-to-noise ratios and so forth off of that receiver. And, it will give you an alarm and let you know when ingress levels reach unacceptable levels.
I think that's going to be a very valuable piece of equipment on systems that are launching high-speed cable modems like we are.
Sniezko: It's too expensive. Any monitoring system is still very limited in what it does. I don't think they make sense at all.
Our services are our main monitoring system. As soon as you have problems reported by your system, which is comprised of modems at customer premises and the headend, this should monitor service quality. And, if there is any indication that service quality is getting degraded, for whatever reason, then you go into a manual mode of monitoring, which is much less expensive.
Nguyen: We are looking at it. We're looking at some type of status monitor at the end of the line, middle of the line, the amps, the power supply, etc. We're looking at deploying something like that on a very limited basis. We need it as we roll out new services.
Our thought is that we can use the existing equipment, such as our digital boxes. We can poll those boxes and correlate that information (to determine our condition). That's a short-term solution.
Long-term, yes, I think remote testing makes sense. Right now we can't afford it. And also, we don't need it at this moment, because we don't have that much (advanced) service deployed yet. When you have hundreds of thousands of customers out there using data, digital boxes and so on, I think eventually we might need more of that stuff.
Bauer: I guess it depends on the level of remote testing. We currently have full remote capabilities of the Internet server. And we've done remote monitoring and control of that from day one. That's just because the systems we have been running are in a lights-out condition.
I don't necessarily see in my systems widely fluctuating levels on the cable plant. I think the manufacturers of the trunk and feeder amps have done a fantastic job. They are very good, strong and reliable pieces of equipment. I have amps I haven't opened in years, because they don't need it. When I do go to those, we made the extra effort to make sure they had good, solid seals on them and everything was done correctly. When I open one now, you hear the air escaping. So, you know that there was a good, solid seal on there.
As far as monitoring levels, I think that in the smaller systems you're not going to get a significant bang for the buck to do end-of-line system monitoring. I think that being able to have remote access to the signal levels coming out of the headend would be fine. Most of my problems tend to be in the headend, not in the plant.I wish I may, I wish I might...
CED: If you could wave your magic broadband wand and create the most comprehensive, useful piece of testing equipment available, what would it be? What would it do?
Sniezko: It doesn't exist. It would do my frequency response and group delay in a non-disruptive way. And it would measure the dynamic range in the reverse path. And for the other pieces that exist, the one that monitors just ingress or all the interference that's out there without signal. Spectrum analyzers are doing a very good job on this one and can actually monitor signals underneath the service, as well as the discrete signal.
The other issue would be impulse noise. We don't have anything to monitor impulse noise. You can build pieces to capture impulse noise, but they are not single-unit pieces of equipment. You have to put different gating devices together.
Warren: If I could design a piece of equipment that would do it all, it would weigh five pounds. It would have batteries that last 24 hours. And it would combine the functions of an OTDR, an optical power meter, a spectrum analyzer and a digital power meter, all in the same package.
Nguyen: I think if we could integrate both the forward and reverse path testing, as well as analog and digital testing, that would be my wish right now. And right now, we have separate pieces of equipment for each of those.
And also, I think more importantly, at the headend, we're going to need some kind of equipment that integrates all those tests as well. Data, analog, digital, video, forward, return and different types of modulation schemes, FSK, QPSK and QAM. We run all those things now, and we really struggle with testing all those things.
And operationally, the technician needs a cheaper set of equipment to service all those things, too.
Dupler: Well, I know one vendor has made some pretty good strides in their equipment. They're making a meter that does leakage detection and level recording. It tells you whether you've passed or failed the FCC requirements. That's a pretty useful piece of equipment. It has a couple of pieces of equipment all wrapped up into one.
Now there are pros and cons to that. Say your signal level meter goes on the blink. Well, you've just lost two pieces of equipment. So it's a Catch-22. But I like having one piece of equipment when the installer/tech goes in the house. I like that. They carry enough equipment as it is.
Blair: We still wait for our customers to call us if there is a service problem. We're trying to take a stab at establishing a standard for network monitoring. It's a piece of what's needed overall in testing. I think you could say our service and install technicians turn out to be our network monitors.
Say we're running around two percent service calls a month. That means we're seeing 24 percent of our customers over a year's time, and that's an opportunity to make sure that network is capable of (providing) existing services, as well as future services.
So, when we go to that house, if they (the technicians) are taking a very small meter that gives them just two carriers, with a go or no-go, it doesn't tell you anything. What we really want to see is that we upgrade everyone to a meter that is capable of looking at everything, and then analyzing it and giving you feedback.
So, I think our goal would be, if you're talking about a magic wish, since you're already running people to homes, to use the information (the technicians get on those calls) more accurately to maintain (the network). That, alongside of your network monitoring eventually gets you to a very reliable network.
Bauer: I want a handheld device that has the ability to look at very short duration, high-energy broadband pulses coming through that are trashing the data. And, in addition, it would tell me down to even the node or house, where that ingress was coming from.
It would also have the full capabilities of being able to transport or send data out and get it back and know whether there's still a problem. That's one of the pieces that is missing even in anything we're doing with the cable modems. How do you know when you're having problems? And where are the problems occurring?
I've got it (a solution) on the drawing board right now. So, I would say we're making good progress. We have the concept for doing it, but as to putting it into the field at an affordable cost, we're not there yet. We're several months away from actually doing a field test that would identify where it's coming from. The concept is there, (laughs) we just need more hours in the day.