Cable operators and IPTV providers have recognized the importance of having comprehensive video service monitoring coverage. In fact, many have been buying quality of service (QoS) or quality of experience (QoE) monitoring systems to augment traditional (video wall) practices to improve video service quality as a way to compete for new subscribers and combat churn.
Many have found, however, that traditional monitoring tools are difficult to use and often generate so many alerts that it is hard to determine which to react to first. As a result, operators are getting a diminishing return on these tools as they become impractical.
Just like any other investment, a return on investment (ROI) assessment should be considered before a monitoring system is purchased and the results closely tracked following deployment. Service providers should consider whether the monitoring system can help identify critical subscriber-impacting issues, especially those which are intermittent, and annoying issues such as video tiling and audio disruptions in the digital TV system.
These types of digital errors can affect a large subscriber base and are difficult to catch with traditional monitoring tools. Therefore, an ROI assessment of any monitoring system needs to be considered in the context of its ability to quickly isolate problems, provide real-time information for root cause analysis and prioritize outstanding issues to ensure the most serious are fixed first. This will help keep operational expenditures to an absolute minimum and subscriber satisfaction high.
1. Be aware of the "garbage in, garbage out" scenario. Get the (video) source covered first!
Less advanced monitoring tools often send service providers on a wild goose chase trying to narrow down the root cause of an issue. Often these tools suggest problems further out in the IP network when the issue is actually upstream at the video headend. For example, errors at the satellite receiver, encoder and multiplexer output travel downstream and manifest as video black/freeze, tiling and audio loss and can affect a large subscriber base.
Problems will propagate, or even become worse downstream. If the source is not monitored appropriately, these issues over time will significantly impact operational expenditures. Effectively monitoring the source in the video head makes problem tracking and isolation more effective and alerts more actionable.
2. Identify "stealth" issues proactively and fix them before subscribers complain.
A stealth issue is a serious subscriber-impacting event that will not be detected by traditional QoS metrics such as continuity counters, IP network jitter, etc. The screenshot below shows serious QoE issues (blue line drops) indicating major tiling and/or other video black/freeze issues (refer to the video capture), but at the same time, there were no QoS problems (discontinuity or IP network jitter, as indicated in the screenshot).
These QoE impacting events could happen hundreds, or even thousands, of times on a daily basis. Probes that are unable to provide real-time analysis at the audio and video content level will usually miss these intermittent problems, therefore missing the opportunity to add value to the bottom line. Service providers should make sure their monitoring systems will catch these stealth issues to augment existing monitoring practices.
3. Prioritize troubleshooting tasks based on subscriber impact.
Intermittent and recurring video and audio errors such as tiling, frozen and black screens are results of gradual system degradations. They are the issues that subscribers most often complain about, and due to their irregular nature are the most difficult to catch. The monitoring system should allow for setting alerts which gauge the subscribers' experience, as well as the system performance. This enables operators to discover and quickly resolve high-impact issues, which can substantially improve the troubleshooting ROI.
In analyzing the impact, both qualitative and quantitative analyses should be taken into account. For example, a single minor subscriber-impacting event can be a nuisance, but if it continues for a few minutes, or even throughout a program, it will surely irritate subscribers and may result in costly customer service calls. The alert function should be powerful enough to detect, and flexible enough to accommodate, alert triggers based on frequency (circled in red in the screenshot), as well as severity level (circled in blue in the screenshot) for every event.
The monitoring system should offer powerful alert-triggering capabilities, which give operators automated, reliable and objective ways to gauge the subscriber experience (QoE) and system performance (QoS). This enables them to prioritize troubleshooting tasks accurately.
4. Generate relevant information and make it accessible and relevant to the entire organization.
ROI is better realized and optimized when timely and relevant information is provided to different types of users in the organization, including technicians, engineers, managers and executives. It helps people communicate better and solve problems more efficiently. The ability to provide timely and accurate reports also provides confirmation of the issue and gets everyone, including equipment vendors and program providers, on the same page with the service provider.
Instead of being bombarded with a rash of red lights (especially during an alert storm), NOC personnel need to be able to make use of a numerical count of the alerts to understand the location where the problem originated from and the type of problem in order to dispatch a trouble ticket to the right group of people. See the blue box in the figure below indicating one MPEG problem coming into the master headend (alert count of 1), and one more added (alert count of 2) while leaving the master headend, and the red box indicating one (IP) datagram loss event has happened (3 minus 2) while video was transported from the master headend to the regional headend 1.
Upon receiving the trouble ticket, the MPEG and IP engineers should be able to use the same tool to generate historical reports for time-based troubleshooting and root cause analysis. Depending on the monitoring location, intermittent video/audio degradations can be caused by digital program insertion, encoding/decoding, stat-muxing or program source /transport issues. The monitoring system should allow for all of the reports to be printed and shared with the vendor with enough data to forego time-consuming offline analysis.
Managers and executives can receive weekly reports automatically e-mailed on a regular basis, highlighting QoE and QoS areas of concern. For example, a report listing the top-10 offending programs (see sample below) and quality rankings by location over a configurable time period (24 hours, weekly and monthly) aids in quality trending and tracking improvements over time.
A monitoring system that is useful to different types of users, internally and externally, will optimize its ROI tremendously. NOC personnel can respond and dispatch a trouble ticket more accurately, engineers can use the same tool to identify the root cause, and managers/executives have an objective and accurate way to understand continuous service quality improvements made over time using a solution that provides for both QoS/QoE monitoring and reporting.
5. Future proof your monitoring solution to maximize its impact on ROI.
Cable operators' architectures and the services they provide subscribers are in a constant state of flux. Accordingly, a monitoring system must provide a future-proof platform, whereby features and adjustments are offered through a software upgrade rather than requiring new equipment to replace obsolete technology.
Over the past several years, service providers have added large volumes of HD programs and applications such as tru2way, digital program insertion and switched digital video (SDV). These changes require new advances in monitoring to provide for the successful deployment of these applications and services. If the monitoring system provides a future-proof design, the same ROI will continue to be realized without costly replacement of older, proprietary products.
Optimizing ROI should extend not just to revenue-generating products, but also to products that provide operational cost savings. This practice starts with picking up the right video monitoring system, developing a monitoring strategy (with the vendor) and closely tracking its progress.
The service provider and (monitoring) vendor can work together to fully operationalize the tools and share the same ROI objective based on cutting-edge technology, a future-proof platform and proven methodology.