While VoIP (voice-over-IP) service reaches deeper into millions of homes and businesses, and its place in the triple play bundle rises to near-must-have proportions, the testing of the service's tricky nuances and complexities is entering the mission-critical stage.

For the most part, the status of VoIP testing is at best a work in progress. Integrating tedious new voice technologies into legacy systems designed for video and more recently for data, and the extremely high quality and reliability consumers have come to expect from phone service, are nagging issues with VoIP providers. The result is a seachange of sorts in VoIP testing procedures and the tools needed to deploy them.

Figure 1
Figure 1: Minacom cable VoIP testing layout.

Meanwhile, the stunning growth of VoIP service, its complicated technology and integration issues, and its infancy as both a technology and business, are pushing VoIP service providers into uncharted testing waters, and elevating the need for quality tests, tools and procedures.

And rightfully so. Gone are the days of simply signing up a few thousand VoIP customers. Now, it's hundreds of thousands, with millions in the forecast.

Greg Quiggle
"It's all about automation for VoIP testing. The telcos support tens of millions of customers through various touch points because they know they can't handle them manually," says Greg Quiggle, executive vice president of marketing for Tollgrade Communications Inc., a VoIP test service provider. "That's where cable will get to, but they're still very early in the market. When they get to 10 to 15 million subscribers, if they don't have a way to automatically support that number of customers, they'll have problems. But the cable operators are getting it and are understanding common failure points. They're pretty progressive."

Most in the VoIP space believe there is indeed progress being made with new test tools and procedures to improve the quality, reliability and cost efficiencies of installing and maintaining a top-rated VoIP service. Yet they also believe it comes with a host of challenges.

"Probably the biggest point of focus is thorough installation practices. Once the data and measurements are collected from the home and a number of test parameters are determined, we can benchmark data from the homes. Then, the technician can look at the install quality based on test equipment," maintains Brady Volpe, vice president of advanced technical solutions for Sunrise Telecom.

Brady Volpe
Volpe recently delivered a VoIP troubleshooting seminar at Cable-Tec Expo in Denver (see "Typical" below). His advice? "In a VoIP network, it's very easy to create impairments in RF and IP domains that mimic IP and RF impairments. In the classroom it may look like a problem, but it may not be. So, we encourage the use of test tools before a truck roll to measure RF signals. Use the test equipment at the headend first, and use all the equipment at your disposal," he advises.

Yet some, Volpe admits, aren't exactly following those instructions. "Most cable operators don't have these procedures in place and aren't taking advantage of the equipment and test practices available. But they're re-evaluating test practices and pushing VoIP test vendors to provide test equipment to measure voice quality. The telcos test every port and every signal on the day of installation. Cable must do the same."

Some VoIP providers, including Vonage and others that don't necessarily manage or own last-mile networks their services are traversing, are fully aware of the VoIP testing priority, and are looking ahead to better testing days.

"There is lots of data, and it requires slogging through different elements. The big problem is analyzing the data and determining how best to use it. Our goal is actionable data," says Louis Mamakos, executive vice president and CTO at Vonage.

Vonage, he notes, approaches its VoIP testing using two methods. It actively tests the network to make sure all calls are routed correctly, and uses an end-to-end synthetic testing procedure with the goal of emulating the customer experience.

Louis Mamakos
"We are working on revisions to score each call and keep the data to look at overall trends to see if there are common problems. And, we're pushing new software into new boxes which are insulated from the customers. That's where we're doing lots of lab testing," Mamakos adds.

Once the subscriber has been tested and the data collected, maintains Michel Nadeau, president and CEO of Minacom, a provider of service level test systems, testing can take place right in the home.

"We came up with a specification on how to loop back and interact with switches to handle the test properly, and have expanded to DOCSIS standards," he says.

And standards are crucial to VoIP testing. Adds Nadeau: "There are too many homegrown standards on the telco side with DSL and fiber. VoIP's standardization with DOCSIS has enormous cost savings advantages."

Testing for packet loss, jitter, latency and other problems on the day of installation or before can be a cost-saving bonanza as well, experts maintain.

"Our first step is to create real-time, practical tests using the server at the headend that receives information from the field and provides results. The biggest test for VoIP is related to traditional RF systems. So, we need a way to isolate where the problems come from. Deeper integration with OSS, back office and workforce management would help. And, there's a massive amount of data to sort through. Tying it all together would make it easier," says Steve Windle, product manager of the instruments division for Trilithic Inc., a provider of test, analysis and quality management systems.

Yet Windle and others offer a caveat, particularly to the cable industry. "Cable could tighten up their monitoring of the reverse path because of so many impulse and ingress problems," he says. "We can thoroughly test the install for voice services, but could use more feedback on the usage of VoIP and how well we're meeting the objectives."

Rick Berthold
Those objectives have led to a number of trends in the VoIP testing space, says Rick Berthold, CTO of Proxilliant Systems Corp., which recently launched its Cable Access Management System (CAMS) to help prepare networks for VoIP services.

"VoIP testing and monitoring must provide root-cause rather than provide independent symptoms of a problem. We must also provide real-time testing and integrate all layers of technology such as RF, IP, QoS and QoE (quality of experience), and it must be performed exactly when needed by automated systems, field technicians, network operations and customer service reps," he explains.

Key VoIP testing trends, Berthold notes, include:

  • End-to-end coverage. Testing collectively has to have complete network coverage.
  • Correlated, cross-protocol testing. Has to cover, and correlate RF level issues, IP, dynamic QoS and provisioning assurance.
  • Automated testing, dynamic conditional testing. Has to be automated, with a defined criteria for deciding when, how long, what type and where the test is performed
  • Service management focus. Has to be comprehensive for "VoIP service health" calculations.

The health of any network is a primary consideration before any VoIP testing can be done, experts maintain, and factors into the VoIP testing equation.

Figure 2
Figure 2: Minacom MTA loopback testing layout.

"The different architectures are playing a role in VoIP testing and the tools to test with. However, you can test all you want, but if the network is not running properly, it will do no good. There's a much higher quality expected of networks today," says Foad Towfiq, president of Promptlink Communications, a VoIP test and analysis company that counts Time Warner Cable among its customers.

The emergence of VoIP as a viable business model has spurred many cable operators to take a closer look at the overall health of their networks, a crucial component to the deployment of VoIP and the testing processes that support it.

"Basic plant quality is a key issue for cable operators, and has made them go from neutral to solidly behind plant quality for VoIP. There are higher expectations to maintain with VoIP," says Dave Holly, vice president of the cable division for test and measurement specialist JDSU.

Meeting those expectations will take a concerted effort, he admits. "We need both instrument tools and systems to play together."

New testing tools are becoming available that can quickly find and pinpoint problems, Towfiq says. Yet for many VoIP service providers, particularly cable MSOs, the tools must be specifically tuned to voice services.

"MSOs are trying to use embedded, existing tools. But with VoIP, it's more data related and testing for eMTAs. Some are using several different tools, but not one specifically related to voice. Now, there are tools to test, calculate and measure much faster, and before deployment," he notes.

Yet with the increasing subscriber counts for VoIP, Towfiq admits it will be a stretch for service providers to maintain quality testing procedures and the tools to support them.

"It's a new technology (VoIP), and lots of experience is needed," he says. "There aren't that many tools today that can handle large-scale modem deployments. Right now, MSOs simply don't have the tools to test for two lines, so they must do it in the warehouse before entering the field."

One thread that is woven throughout the VoIP testing quilt is the intelligent use of the massive amount of data being collected. "We have a statistical analysis problem. We understand the raw data, but we must distill it down to something actionable. We've been reactive thus far to explain the data; now we have to make sense of it all," says Mamakos of Vonage.

Help may be just a data tool away. SupportSoft Inc., a provider of real-time service management software, aggregates data from the network and verifies that the installation was done properly, and can help with auto-installs of VoIP.

"VoIP rides on the current data infrastructure, so the challenge is to bring the systems together and provide a view of all customer touch points," says Marc Itzkowitz, director of product management for SupportSoft.

Click image to enlarge chart
Figure 3: Tollgrade Communications’
cable VoIP assurance network.
(click here to enlarge image - pdf file)
And those touch points better be addressed quickly, at least according to a recent independent study by Decipher Inc. The report found that 34 percent of cable VoIP users said that a technician had to visit their home within 90 days of the initial service due to a problem. Sixteen percent had a technician visit their home two or more times.

With service calls averaging nearly $200, and the cost of winning back a dissatisfied customer escalating, it's no wonder the focus is on getting it right the first time, and that means more emphasis on VoIP testing–early and often.

"Today, with IP components, call flow and other VoIP elements, it requires IT experience not required in cable in the past, and the Achilles heel of the VoIP network is still in the return path in the HFC network. More than 70 percent of the problems will be RF- or HFC-related. But there are new test strategies, day-of-install testing and new test tools for mobile, WiFi and WiMAX. It's an exciting opportunity for cable," concludes Volpe of Sunrise.

Exciting, yes. Challenging, absolutely. Admits Volpe: "There's lots of VoIP churn due to the lack of pre-qualification, and there's a learning curve similar to switching to digital TV when there was nothing but spectrum analyzers. Now the cable industry has to pro-actively compete with other industries."

To do that, most agree that the procedures and tools to test VoIP service must be upgraded to include more effective use of data, integration and standardization of processes, holistic views of the network, addressing the RF and HFC issues, and perhaps most important of all, the service providers' buying into an all-encompassing VoIP testing mindset.

"It gets down to a matter of skills, training and having the right tools. The overriding issue is quality, quality, quality. We have to get it right first," says Itzkowitz.

And for VoIP service providers such as Vonage, getting it right first, and often, is top of mind. "VoIP as a technology has been around, but the trick now is to package it where mere mortals can install and use it. Testing VoIP for quality and to ensure a good customer experience is the key. That's our goal," concludes Mamakos.


Typical VoIP testing gear

  • Spectrum analyzer–front line for RF analysis in the headend;
  • Handheld field meters–front line for field testing;
  • VoIP monitoring analyzer–call analyzer capable of monitoring and generating thousands of simultaneous calls;
  • Third-party traffic analysis systems–test calls to installed eMTA and display tests via Web browser on handheld field meter.
Recommendations for VoIP testing
  • Maintaining VoIP networks is a multi-discipline function, spanning RF, DOCSIS protocol and IP protocols;
  • VoIP networks are complex systems. Be thoughtful in diagnosis and avoid pointing fingers toward RF problems before considering other possible problems;
  • Know the three main VoIP impairments and what can create them.

Source: Sunrise Telecom