Test & Measurement - A Preponderance of Evidence
One challenge in testing a network is too much data.
A requisite of keeping service networks up and running efficiently, even as they become increasingly complicated, is the ability to get data out of the network in real time. The problem is not a lack of information, however.
The problem is a surfeit of data so overwhelming that frequently it is literally impossible to extract meaningful information from the overflow in a timely manner. Oddly enough, the other half of the problem is that this volume of data is often hard to get to.
Consequently, accessing, sifting and analyzing network data for actionable intelligence is becoming an enormous priority. Test and measurement (T&M) equipment vendors, which by nature concentrate on the data extraction side of the equation, are now coming up with increasingly clever means of making sense of raw data for their customers.
Institutional resources are being brought to bear, as well. A few weeks ago, in mid-June, the SCTE approved a new standard that addresses the issues of verifying, monitoring and evaluating video transport streams. Meanwhile, CableLabs is engaged in a project in which it is creating a set of best practices for extracting data from DOCSIS cable modems and turning it into actionable information.
CableLabs is exploring how to tap the vast amount of operational data collected by cable modems and make proactive use of it, so that this data can be used not only to diagnose failures or service glitches after they’re reported, but potentially also to evaluate connections for incipient problems and potentially fix them before they manifest as something detectable by customers.
CableLabs declined to comment on the effort.
The subject of the CableLabs program did not come up in conversation with JDSU product marketing manager Kevin Oliver, but the subject of getting access to HFC network monitoring data did.
SNMP – the Simple Network Management Protocol – is specified in the Internet Protocol Suite as a standard, common means of monitoring network-attached devices. T&M companies aren’t getting full access to SNMP information out of DOCSIS systems, according to Oliver.
“SNMP queries are part of the spec,” Oliver said, “and we need to have access to that. Operators tend to be stingy providing that access.”
There might be a reason for that, he allows. “My understanding is that it bogs the equipment down if everyone’s constantly pinging it all the time. But maybe operators can pull that info and put it into a database, and then provide access to the database,” Oliver said.
|Reference analysis point in a digital TV system|
|The SCTE has just approved a new standard that specifies what video transport problems to test for and how to prioritize problems in terms of likely severity. The standard (Recommended Practice for Transport Stream Verification; SCTE 142) assumes the reference point in a digital TV network will be between the emission remultiplexer and the QAM modulator.|
While CableLabs is exploring the issue of getting device data out of cable modems, both the SCTE and ANSI approved a new standard that describes the parameters a video transport stream should conform to, and defines strict limits for each parameter.
As a practical matter, however, relatively large deviations in some parameters can have little effect on video quality, while small deviations in other parameters might have a clear, adverse effect on video quality that will be immediately obvious to viewers.
“I see something is wrong, but do the problems matter? Are they serious?” Those are some of the questions that the new standard, Recommended Practice for Transport Stream Verification (SCTE/ANSI 142), tries to answer, said Triveni Digital CTO Rich Chernock.
SCTE 142 recommends ranges of deviation in different parameters that operators might consider tolerable before deciding the problem requires being corrected. “It lets you set up filters. Only the stuff that matters is exposed,” Chernock added.
T&M companies lament the onus is on them to get information about test needs from their customers. There just aren’t a lot of RFIs or RFPs going out for T&M systems.
“You have to stay close to your customers,” VeEx CEO Cyrille Morelle observed.
Now that cable, telco and satellite are all offering similar bundles of services, service quality is a competitive issue. Service providers are nearing the point where if they respond immediately to a complaint, they’re already too late.
The CableLabs effort and SCTE 142 serve to point out how quality of service (QoS) and quality of experience (QoE) issues are serving to blur whatever line there might have been between testing and monitoring.
“Yes, that has definitely started,” JDSU’s Oliver noted.
“That’s something Empirix has based our whole business on – the quality experience, from end to end,” said Tim Moynihan, a vice president of marketing for Empirix, a company that specializes in test and monitoring solutions for voice services.
“Operators are beginning to understand the value of monitoring,” Oliver said. “It’s better than dispatching guys with network analyzers.”
It must be valuable. “You can spend 10- to 15 percent of the cost of the total network build-out on monitoring,” said Fred Sammartino, another Empirix vice president on the call with Moynihan.
End-to-end quality is becoming difficult. With the growing deployment of IMS cores and true IP backbones, signaling between different providers and different applications is getting far, far more complicated than the old homogenous PSTN.
That necessitates both passive and active monitoring. Passive methods typically require storage of network data in databases that grow enormous with months’ worth of data.
The data can be useful – if a service provider helps itself by making sure the data is accessible. There’s a very typical organizational configuration that works against accessibility, VeEx’s Morelle observed. Corporate information should be on a centralized platform, but too often isn’t.
The database with all of the cable modem information is typically maintained by a provider’s IT department – which might even be contracted out to an IT contractor (IBM, for example). “The task is to get that information to operations,” Morelle said. “Better yet, the system should be able to mine that data live.”
But assuming the data is accessible, it’s great to have it to sift through so that a problem can be pinpointed.
If it can be pinpointed. It turns out a lot of problems in modern networks are ephemeral, Sammartino said. “You’re looking for needles in haystacks.” Tracking an error can be a painstaking process and can sometimes take days.
So an alternative is to load the network with test signals, with the aim of recreating the conditions that triggered the original error, hoping to force the error to occur again so that you can identify it in real time.
Starting up new services is equally complicated. “Before services are turned on, you have a whole class of products you use for turning on the physical network, so you can qualify the network,” said JDSU’s Oliver.
You need to simulate the network before you even attach CMTSs and EMTAs at either end of the network. “You need to be sure there’s no jitter, no latency, you want to check for packet loss,” Oliver said.
Simulation is also critical before adding services or features. DOCSIS 3.0, for example, vastly expands return channel capacity. JDSU's DSAM line of testers actually incorporate a CableLabs-approved cable modem. “You want to generate or sweep QAM signals on the upstream, test to make sure the HFC network can carry the traffic before launching the upstream," Oliver added.
What’s the score?
The final form of SCTE 142 also underlines the increasing importance of not just providing raw data, but presenting the results of analysis in a format that’s easy to digest. It’s becoming a theme across a wide variety of test and monitoring applications.
Last month, CED spoke with companies that specialize in network monitoring and companies that provide equipment that manipulates video as it goes through the network, and CED reported on how many of these systems are rendering processed data about video quality in the form of ratings or scores – for example, on a scale of 1 to 5, or from 1 to 100 (see “Are you experienced?” June 2009).
The approach harks back to the telephony world, where phone companies use MOS scores, a standardized approach for evaluating call quality on a scale of 1 to 5.
T&M vendors have increasingly been doing the same thing, providing equipment that can take readings from various tests performed by installers or technicians, and rather than generating a table of numbers or a graph or a wave trace, it provides a score or a pass/fail indicator.
JDSU is providing field technicians with test equipment that provides results similar to MOS scores, Oliver explained, but on a scale of 1 to 10. “That’s different with today’s equipment from before,” he said. “Now if I tighten a connector, I know immediately if I’ve fixed something.”
And, of course, network test is guiding JDSU into monitoring, as well. The company has a system that sits between edge QAMs and consumers. “With eQAMs, you’ve got GigE in and RF out,” Oliver said. “An eQAM is a very complicated device. It does a lot of multiplexing, time stamping. ... If the video coming out of it is bad, you want to determine that quickly.”
That process also requires some monitoring at the headend to make sure the source is good in the first place.
Getting to both ends to attack the middle is another means of improving testing, monitoring and overall network quality. “It’s a divide and conquer strategy,” said Trilithic product manager Steve Windle. “In plant troubleshooting – say you have ingress. A cool thing we can do is show what’s happening at the hub site and at the remote position simultaneously on the screen.”
That allows the technician to isolate the position of the problem. “If you have impulse noise or ingress at the headend, but not at the test point, then the problem is nearer the headend,” Windle said.
VeEx’s VePal line similarly includes field testers with upstream signal generators, and it can be used in conjunction with VePal monitoring equipment installed in headends to pinpoint the location of various faults in the network.
For more details on SCTE 142 and what it accomplishes, see “SCTE 142 sets new standard for transport stream monitoring” by Triveni Digital CTO Rich Chernock. First published in November 2008.