The industry is still figuring out what exactly to measure.

Service providers have a problem looming that could get very large very fast: There is currently no way to ensure quality of service (QoS), let alone quality of experience (QoE), for over-thetop (OTT) video.

Companies that specialize in test and measurement (T&M) and monitoring equipment for the communications industry say their customers are pressuring them for solutions. The T&M companies are confident they’ll be able to devise something eventually, but not soon – certainly not in 2011.

The thing is that OTT is not just for the competitors of multichannel video programming distributors (MVPDs) anymore. OTT delivery is being embraced by MVPD partners, potential customers, and in one notable instance, an MSO. Having to ensure quality for OTT was not really widely anticipated, and it might be possible to blame network neutrality agitators for that.

Tektronix SentryIn 2006, Shaw Cable extended an offer to its subscribers who were also using Vonage: For a $10 fee, Shaw would guarantee QoS for Vonage calls. Vonage and its supporters turned that into a public relations debacle for Shaw. A year later, Comcast was discovered to be actively interfering with BitTorrent traffic, which turned into a mess involving net neutrality warriors, the FCC and members of Congress.

So the service providers’ response to third-party service providers like Vonage and the OTT video companies that soon followed boiled down to: “Fine, then. We won’t touch your signals. Distribution on our networks will be best effort all the way. Oh, and by the way, don’t whine to us about quality problems.”

And for the most part, OTT video providers don’t whine about it (much, at least not in public). And for the most part, their customers put up with lower video quality, buffering interruptions and other impairments.

Service providers never took OTT lightly, but neither have they panicked about it. All along, they’ve known two things that it took a couple of years for most everyone else to grasp: 1) they have exclusive rights to the majority of quality broadcast programming, as well as early rights to major films; and 2) they own the market on quality video.

Thus TV Everywhere was an effective countermeasure to OTT services from putative competitors like Netflix, Roku, Amazon and Hulu. Service providers were able to provide premium content with high video quality.

And their lack of panic was justified by the number of people canceling subscription video services (aka, cord-cutting) in favor of OTT video, which to date remains low. So low, some MSOs report, cord-cutting is undetectable.

On the other hand, the rate of adoption of subscription services among young adults appears to be low. This group certainly is not buying TVs in the numbers one might expect.

So maybe subscription viewership isn’t being abandoned by those who have it today, but the evidence could easily be interpreted that subscribership is not being replenished by young consumers, and that’s as worrisome as cord-cutting, if less immediately consequential.

And all along, the demand for OTT video grows by astonishing leaps and bounds. Netflix, all by its little lonesome, accounts for nearly 30 percent of peak hour traffic on the network, up from 20 percent just seven months ago, according to Sandvine.

And because OTT is increasingly how video customers want to consume more and more of their video, everyone is beginning to jump on the OTT bandwagon.

Programming partners like HBO and ESPN have begun delivering streaming video, over-the-top, albeit only to authenticated subscribers.

Shaw Cable just started its own streaming service, called Shaw Movie Club, to compete with Netflix, which recently started operating in Canada.

All of which begs several questions. How soon will viewers with devices that are capable of displaying HD start demanding HD-quality video and a better streaming video experience?

Programmers like HBO understand that with OTT, the distribution channel is best effort, but how soon before they start agitating for QoS/QoE guarantees from their distribution partners?

If Shaw Movie Club is successful, other MSOs will try to emulate it. How will they secure QoS/QoE for themselves?

MVPDs cannot charge their customers for QoS/QoE, but eventually OTT traffic will increase to the point where it will be economically unviable for MVPDs to carry it all for free. One solution is so obvious that even Wall Street is asking the question:Why not charge the OTT companies to provide QoS/QoE?

Because it can’t be done.

Danny Wilson“You want an SLA? How do you quantify it?” asked Pixelmetrix CEO Danny Wilson. “We don’t even know what ‘good’ is. We don’t know how to describe it. With phone service, you can promise a BER. OK, I’ll pay for that. With overthe-top, we can’t do that. We don’t even have a ruler.”

For starters, OTT video will always be distributed through at least two networks, and frequently more, and no company will ever be able to guarantee QoS on some other company’s network.

“Even companies like Cisco, they’re providing a big part of the infrastructure for everybody, and they have no way to track it,” said Eric Conley, vice president of video strategy at Tektronix. “Nobody has anything that assures video is delivered over the network OK.

“If you’re going from HBO to an iPad with HBO Go on it, who knows where it goes?” Conley continued. “Say I’m a Comcast subscriber getting a show from HBO. It may go over HBO’s network, over its ISP’s network, over a backbone network, and then to Comcast, into its DOCSIS/HFC environment, and into my house.”

In a TV Everywhere network, it’s a little more straightforward, but it’s not problem-solved. You still have to monitor sources. But if it’s OTT? “All bets are off,” Conley said. “If a Comcast subscriber calls in saying, ‘I linked into the Xfinity app,’ if that person is out of the Comcast network, Comcast has no idea what’s going on.”

Tek provides products capable of source monitoring. Conley said cable operators are only beginning to do that. Beyond that? “It’s a tough challenge. We have a partial solution, but we haven’t solved it all yet. [MSOs are] coming to companies like us to solve the problem, and we’re collectively looking to do that,” Conley said.

Eric Conley“We’ve got nothing to give them,” agreed Wilson. “We end up asking, ‘Can we work with you to solve this?’”

In the past, with broadcast TV, cable TV, satellite TV, there was a direct relationship between QoE and signal impairment. You could detect the source of a signal impairment.

But packets on a TCP network? There is no impairment, per se, Wilson explained. Packets always get there. Maybe not on time, but they’re resent until they do reach their ultimate destination. So when you’re talking about QoE with streaming video, what are you measuring?

Perhaps one question to ask is: “How much time and bandwidth are you wasting retransmitting stuff?” Wilson observed.

Almost everybody is using adaptive bit rate (ABR) streaming. Multiple bit rates are possible, say 100 kbps, 450 kbps, 900 kbps and 2 Mbps, and the rate delivered to any single device can vary depending on any number of factors. Furthermore, those conditions can change in (literally) mid-stream, and the delivery rate may change in response.

“A lot of people are working on how to switch gears,” Wilson noted. “What happens if no one is watching the 2 Mbps stream because the network is too slow, even if you’re offering it? You have to be able to measure that.”

And what happens if your device starts flipping back and forth between two rates, an occurrence common enough to get its own neologism: “flapping”? That could very well be a QoS measurement.

Wilson said Pixelmetrix has defined seven suggestions for metrics for defining OTT QoS – which the company isn’t ready to share yet – another indication of how not imminent a solution is. “We’ll have products next year,” he vowed.

Jack WebbMeanwhile, “you’ve got BER and MER now,” said Jack Webb, product manager at Sunrise Telecom, referring to bit error rate (BER) and modulation error rate (MER) measurements, “but typically with BER, that’s a short test, maybe ten, twenty seconds, but the truth is that’s a pretty small sample. The quality of the sample will be low. We just have to do more tests. You might see a resurgence of downstream sweeping. We’re beefing up our digital testing; we’ve expanded our line to do distortion and sweep on it.”

Checking quality across networks isn’t even possible today. But even though an MVPD has some control of quality within its own network, it doesn’t have much.

Conley said that while operators are beginning to check video quality at the interface with the network, and some do some monitoring of their networks, little of it is video-specific.

“There are a lot of hiccups in adaptive bit rate streaming, and there’s no monitoring for it. There are no products on the market that address the QoS issue once it hits the HFC network,” Conley said.

The broadband channel simply wasn’t built for assuring video quality.

“QoS is an issue on the downstream on the DOCSIS portion of the network. On the downstream, there’s no protocol for it; it’s always been best effort,” Webb observed. “A data stream is a data stream, video or not.”

Channel bonding, one of the biggest boons from DOCSIS 3.0, can actually complicate the ability to detect a problem, Webb noted.

“We’ve been doing VOD for some time,” he said, “but there, the video is on a single QAM. If there’s an impairment in a channel, it will be reported. But for over-the-top? The data may be shared over multiple downstream channels. Before you could troubleshoot, but with DOCSIS 3.0, you have no idea where an impairment is. You need to test all of the channels on the downstream for data.”

More traffic and more channels will cascade into an additional complication.

“At peak times, with more channels in use, problems will be tied to utilization rates,” Webb added. “Now the network is ambivalent to load, but the network is going to get more load dependent.”

This all could turn into a vast monitoring challenge. “Say you’re a large operation with multiple CMTSs, each with eight bonded downstreams,” Webb explained. “You’re going to see more downstreams dedicated to data, and it will require monitoring all of them.

“In the future, it will all be expanded to include additional monitoring points, including data from modems and set-top boxes. That’s going to have to be linked to network maps. Testing will have to be set up intelligently. It will have to include GPS info, and you’ll have to provide access to all of it by mobile technicians. You’re going to need historical data. There’s going to be a lot more interface between field operations and network monitoring.”

That has ramifications for increasing the skill level of field techs, he noted. Later, Webb reconsidered at least part of what he was expecting to happen: “The thing I foresee is that we’ll have no insight into cable modems and set-tops.”

He acknowledged that an operator can run SNMP queries on some CPE today, but that doesn’t render as much useful information as could be hoped for. It would be desirable to get more event logging capability in CPE to track at least some of the information that would be useful in troubleshooting, he suggested. “I’d like to see that as an SCTE standard.”

“That’s the last piece that needs monitoring,” Conley agreed. “It might be impossible at the device level, but you might be able to check at the modem and the gateway. But that stuff doesn’t exist.”

Alcatel-Lucent is trialing a new managed service called AppGlide that aims to enable MVPDs to do some monitoring of online QoS that goes beyond rudimentary BER and MER measurements.

AppGlide Video Analytics combines data from video player plug-ins, QoE agents, content delivery network devices and routers to deliver a view of subscriber QoE, viewing trends, content usage, and CDN and network performance.

It shows the volume and quality of video being delivered and gives visibility into viewer behaviors (what is being watched, how long, why viewers stopped watching, etc.).

“Having access to the individual data sources is meaningful, but – according to service providers we've spoken with – the cross-correlation of this information is powerful and unique to AppGlide,” said AppGlide general manager Buck Peterson when the service was announced two months ago. “For example, being able to see that a viewer stopped watching a video after only a few minutes, and this coincided with poor quality of experience will enable the provider to take very specific remedial steps.”

Furthermore, the system will be able to compare the performance of different content delivery networks – Akamai vs. Limelight vs. Level 3, for example – or the performance of different content providers – Hulu vs. Netflix – or the difference between different types of content from a single content provider.

“I think there will eventually be things in SLAs and peering agreements to provide for some service guarantees,” Conley said.

Assuring end-to-end QoS will require undoubtedly complicated contracts among MSOs, ISPs and content originators, Webb agreed. “The lawyers are going to get rich.”