Finding and Fixing Out-of-Spec Modems
If you are in technical operations I’ll bet you were given a goal at one point in your career to reduce truck-roll trouble calls. Signal level meters played a major role in providing the needed visibility to direct our work efforts at every job. And now, with the advent of DOCSIS modems in the late ’90s, there is an ongoing effort amongst cable operators to prevent new, and reduce existing, out-of-spec (OOS) modems. It’s a big deal! Technical performance goals are tied to these numbers and tech ops managers and supervisors require their technicians to not leave the customer premises until the modem meets minimum specifications.
Nodes and hubs are given health scores based on these OOS metrics and are ranked across large boundaries, and preventive maintenance programs shifted to include DOCSIS-based preventive mindset tools.
So, who controls the “in-spec” specification? Much of the ability to assure a modem will operate within engineering targets is in the hands of the broadband technician, however, there are a significant number of reasons outside of the customer premises network that could influence why a modem will not transmit or receive within design parameters. The signal health can be degraded by other headend combining, CMTS configuration, HFC level alignment-linearity, and loading.
I think you would agree that the broadband technician must have a path to escalate OOS modems that are not the result of the customer premises.
Performance Bar: What is an affordable “out-of-spec” definition?
Entire teams of paid technicians and engineers may get caught up spending significant time troubleshooting for what ails only a handful of modems. If parameters are too tight, achieving them becomes a never-ending expense involving in-premises amplifiers, cable simulators, forward-only attenuators, reverse-only attenuators, and more.
Too loose, and the extra margin is lost and the demand maintenance is never ending. Poor habits can grow from loose specifications and this leads to hard work.
Heaven forbid when a corporate escalation occurs! Work smart. Be sure your in-spec target is attainable before troubleshooting.
Identify the magnitude of the impairment and characterize it for what level of improvement to expect. For example, mitigating ingress down by only 2dB to achieve an in-spec carrier-to-interference isn’t enough. The engineer in the room will ask you for 6dB of performance margin, or breathing room, knowing that noise floors in the plant breathe and change especially when our customers arrive home in the afternoon.
What is 1 dB amongst friends?
You need to understand your written out-of-spec standard before applying any significant effort.
Four common DOCSIS metrics are based on RF levels and health. Specifically:
• Downstream RF receive level (to the modem) – Rx level is too high or low.
• Downstream health SNR/RxMER (to the modem) – RxMER is too low.
• Upstream RF transmit level (from the modem) – Tx level is too high or low.
• Upstream health SNR/RxMER (to the CMTS) – RxMER is too low at CMTS.
Every DB counts, and achieving inspec compliance at each end of line begins by maintaining the original engineering plant and premises design. Accurately building plant with the appropriate line equalizers and tap values is as important as assembling the correct headend combining network which accounts for every bit of attenuation.
When a technician invests quality time to travel out to the deepest areas of the plant it only makes sense to spend adequate time to grow both upstream and downstream performance margin while they’re there. Few proud managers believe that a modem that is slightly out of spec is acceptable. In fact, many of these same managers are uncomfortable with a slightly in-spec modem at the end of the day.
Headend, plant, and premises teams must work cohesively to identify design problems, align optics and amplifiers for the lowest noise floors, determine when end-of-line equalizers are missing and identify RF level variance at any point in the HFC network. The broadband technician must be trained and trusted to consistently account for losses at every job.
All technicians must communicate across departments to identify network shortfalls and each must understand their part in assuring proper operation and compliance to the engineering standards.
Proven techniques for reducing OOS modems
Building on basic RF fundamentals is our best chance to foster a lifetime install. You need to size coaxial cables for the distance, account for losses, understand premises wiring limitations and know what to do in one-off scenarios.
Also, avoid poor decisions such as installing unneeded house amplifiers, running a second drop, upsizing a drop unnecessarily and lowering the value of a faceplate in order to pass specifications.
Once RF downstream levels are achieved, target SNR/RxMER. If its lower than expected, check for downstream ingress by isolating the drop or by leveraging a feature to see ingress below the QAM.
Technicians need to address RF level variances in both the forward and reverse paths as RF level swings are not normal operation.
Upstream SNR is everyone’s responsibility.
Limitations begin when two node upstream paths are combined reducing C/N and SNR by 3dB. Future bonding of more than two upstream carriers will lose another 3dB. The signal ingress must be identified and mitigated at every install and trouble call prior to leaving.
You also need to manage the perceptions of those who take a unique situation and turn it in to a “common and chronic problem everywhere.” I’m pretty sure you just cracked a smile, right? This means that you already know that high-transmit modems often captures the attention of devil's advocates for a variety of reasons. A vast majority of high-transmit modems can be cured by calibrating headend/ hub combining networks, achieving unity gain in the return path and requiring home-run outlets at all premises networks.
Don’t overlook high-value taps and high-value DCs in your distribution network that were missed during the upgrade.
It helps to create an incentive program to reduce the first layer of “grossly OOS modems,” which are known to impact customers more. Implement a “Node Health Report" to identify the nodes that contain the highest percentage of OOS modems.
It also helps to educate employees on the company’s written standard and build a guide for OOS modem reduction to include each group’s role and responsibility in accomplishing these goals.
Include answers to FAQs and detail escalation requirements.
Next, develop simple reporting to show comparative views across service areas and unfavorable level variances and be sure to communicate consistent and accurate information to your entire workforce.
Advanced DOCSIS tools – a new OOS metric for improvement
In addition to the four common metrics outlined above, DOCSIS pre-equalizer MIBs can be leveraged to gain visibility to upstream linear distortions in the coaxial network such as microreflections, equalizer stress, in-channel frequency response, and group delay.
Ever since Charter integrated this new highly-predictive maintenance tool our techs have a new proactive OOS view to help identify and correct before-unseen issues at every install and trouble call that would have most certainly degraded services over time. Damaged cable and connectors that cause poor RF levels or MER can now be seen. Charter’s Node Slayer tool has become the technicians’ first choice tool when they are able to combat drop and hardline network issues quickly and reduce the pain of intermittent issues often before the customer experiences signal degradation.
It’s tough to argue that the use of “block and tackle” techniques used in the early days is still the brute force method used daily.
Integrating the use of intelligent DOCSIS tools to assist in our grand effort to reduce out-of-spec modems is now paramount.
It’s important to be aware that high upstream equalizer tap stress can cause a modem to transmit a little higher to support error correction. Add this one to the high-transmit points mentioned earlier. Enabling pre-equalization on a CMTS line card helps reduce SNR performance hits, however, enabling this feature should be treated as a privilege and not a right. Node upstream paths that already exhibit decent SNR performance will appear to stabilize.
There is a certain percentage of OOS modems that will exhibit this “lightly OOS” trait; these should be treated the same in every market. A “successful connect” or a “first call resolution” is achievable through the use of traditional test equipment and by leveraging DOCSIS CPE to certify each job “in spec.”
Smart techs will seek help when things get tricky, and they will bring 100 percent of all cable modems in to compliance, else they are prepared to articulate and escalate OOS modems not within their control. To quote one of our top installation/service technicians: “I know I’ve gained the respect of the senior tech when they have my name and number saved into their mobile phone, and I’m not about to lose that by crying wolf in a controllable OOS situation.”
I remember the days of chasing set top box non-responders with limited success. Today we leverage far superior methods using advanced DOCSIS modem tools in conjunction with achievable target specifications. This is now my first choice standard and I’m craving the migration to DSG and DOCSIS modems in set-top boxes for primary two-way communications. ■
There is an ongoing effort amongst cable operators to prevent new, and reduce existing, out-of-spec (OOS) modems. It’s a big deal! Technical performance goals are tied to these numbers and tech ops managers and supervisors require their technicians to not leave the customer premises until the modem meets minimum specifications.