2007 CTO Roundtable: How Sexy is HFC?Bandwidth is a perennial topic amongst cable engineers. Last year, during our annual CTO Roundtable, it was all about “the tools in the toolbox”–any number of “levers” operators could pull to preserve bandwidth. This year, cable’s chief technologists point to the potency of logical node splits, mixed with switching, to show off those most enviable of HFC attributes: Adaptability and efficiency.

On the line for this year’s CTO Roundtable: Tony Werner, CTO of Comcast; Mike LaJoie, CTO of Time Warner Cable; Marwan Fawaz, CTO of Charter Communications; and Chris Bowick, CTO of Cox Communica-tions. CED Contributing Editor Leslie Ellis posed the questions. An edited transcript follows.

CED: Let’s start where we always seem to start: Bandwidth. By now, the readers of this magazine are well-familiarized with the “tools in the toolbox” that all of you have to conserve and expand bandwidth. We’ll skip that question this year, but there is the matter of the upstream path: Too skinny? Too vulnerable? Or am I worrying too much about this?

Werner: You’re worrying too much. The return bandwidth is not on the worry list right now, for a bunch of reasons. For one, we’re splitting a lot of nodes based on the success of voice, high-speed Internet, and VOD. In other words, all based on downstream requirements, not upstream.

On HSD (high-speed data), I’m using two to three 3.2 MHz carriers (upstream). A lot more than that are sitting fallow in my CMTS cards. In most markets, I still have 12 MHz of bandwidth I can reclaim from circuit switched voice, once we migrate off of those platforms. So for now, the 5-42 MHz to me seems plenty adequate.

Bowick: Ditto. Your numbers are interesting, Tony. I’ll give you a statistic: I’m averaging about 2.1 (3.2 MHz) upstream channels, not two. I’ve got quite a few out there at 1, some at 3, but the average is 2.1. And, we’re still operating at 16-QAM. Which means we can still convert to 64-QAM. And, like you, most of our node splits are happening as a result of our downstream needs. So we’re in pretty good shape.

Fawaz: Same here, except that 64-QAM is definitely something we will be activating at Charter. Also, there are some areas of the upstream bandwidth that we’ve traditionally avoided. I’m talking about the lower part of the bandwidth, between 5-12 MHz. We’ve avoided it because we hadn’t put the effort in to clean it up. But we can start taking advantage of modulations like 64-QAM, and pushing S-CDMA (Synchronous-Code Division Multiple Access) and A-TDMA (Advanced-Time Division Multiple Access), both available within the DOCSIS 2.0 specification.

CED: What kind of carrier-to-noise or signal-to-noise ratio do you need in order to run 64-QAM in the upstream? Hasn’t that been the issue?

Fawaz: (speaking as former CTO of Adelphia): When we handed off the Adelphia systems, we’d activated 64-QAM in more than half of the plant. I think it was close to 70 percent. The good news we got out of that was, we can run mixed modulation types, and let the modem decide which one to use, based on conditions. Generally speaking, 70 percent to 80 percent of them jumped onto the 64-QAM.

Werner (speaking as former CTO of Liberty Global): We ran 64-QAM in parts of Europe. But realistically, with the tools already offered in equipment based on DOCSIS 2.0–A-TDMA and S-CDMA, specifically–it’s not clear that you need any higher performance than that, for 16-QAM. LaJoie: We’re operating some (upstream) plant in mixed mode, 16- and 64-QAM. Thanks to Marwan.

CED: What other options are you looking at to expand the upstream? Upper spectral area stuff, above 1 GHz? Mid-split, assuming you were to do “all digital” in a way that is equivalent to “no analog”?

Werner: Right now, it’s still a matter of why. We’re not driven to it. We’ve got circuit switched phone that comes off, and it’s eating up a lot of bandwidth. And we still have a bunch of spare bandwidth. On the mid-split, and the other ideas, they’re all options, and–famous last words here, but–I can’t imagine a service that will drive us to it.

Fawaz: We had this same (mid-split) debate during the (DOCSIS) 3.0 specification development, and we came to the same conclusions. Plus, the mid-split is not as easy as people think. It’s an option down the road, in five, 10 years.

CED: Switching seems to be gaining favor with all of you now, as a means of bandwidth conservation/expansion. A frequent question seems to be this: How do you do the math on actual bandwidth savings, attributable to switching?

LaJoie: It’s the wrong approach to think of it that way. Switching, what it does for you, is you establish an amount of spectrum you want to dedicate to switching. You can add as many program sources to that switching fabric as you want. So the bandwidth savings depends on how many program streams you’re offering. But essentially, if viewing patterns don’t change, meaning most viewers watch a few channels–as long as that curve remains the same, you can add as many programming channels as you want. It depends. Maybe you need 8 QAMs, 12 QAMs, to establish the switching fabric to accommodate the demand. But once you establish the pool for spectrum, you can run any number of channels through there. Thousands.

Werner: Mike (LaJoie) and his group are the experts on this. They’ve been the first. Now we’ve all gotten into it. When I was working for an international operator, I didn’t have to worry about it. We didn’t have the same pressures. But at Comcast, we’ve looked at it a lot. It’s clear that the last 200 to 300 channels are watched such a small fraction of the time...if you never have more than 40 streams watched out of 200, you can serve 4 QAMs for what was 20 QAMs. So we’ll go in easy, but we’ll easily pick up 16 to 20 channels.

Bowick: I think it should be added that switching video can cost you bandwidth up front, because you do have to allocate some spectrum to the switch. But look at it this way: You have 135 channels in front of 500 homes passed. How many of them watch simultaneously? 50 or so? Let’s say that in that same node, we place 1,000 switched channels. Will viewership change? Will it go to 75? Then for 10,000 channels, would it increase to maybe 85?

LaJoie: You reach a certain saturation point. After that, the number of programming sources doesn’t matter. They don’t drive the demand placed on the switching fabric. So that means that how much bandwidth you save depends on how many channels you run through the switch. I don’t think you’ll ever get to a point–and it’s been this way since we started offering a broad array of choice–where you have more than 60 or 70 channels watched at the same time among a 500-home group. Not more than 60 channels will be watched, regardless of how many channels are available. For bandwidth savings, that says since I’m putting in 10 channels, it’s 40 percent. If I put 300 channels in the same spectrum, it’s more like 70 percent.

Werner: Basically once you get your switching fabric set up, the number of program streams we can carry on our plant is infinite, without adjusting the architecture and running fiber deeper. It’s a powerful technology.

Fawaz: The business model for Charter is to expand the number of channels, or content streams, that we’re delivering. The first benefit from switched digital is really expanding our HD content. As there is more digital content going forwards, SDV will give you some bandwidth efficiencies.

LaJoie: Look at it this way. When we first started offering VOD with 100 movies, we used four QAMs. Now we have thousands of hours of content, but we still only use four QAMs. Because it’s switched. Once you reach a point of choice, to add more variety in program sources, it doesn’t drive demand. It doesn’t increase the load on the switch.

CED: What about the costs of switching? Field people tend to be concerned that while switching may not cost a lot in pure capex, it may cost more in opex, because of all the hub re-wiring.

LaJoie: When you install this stuff, installation can be capitalized. So yes, you do have to segment your networks, and you do have to wire it up properly, but it’s not a cost issue.

Bowick: We do that stuff all the time for VOD. It’s the same type of re-wiring.

Fawaz: In terms of operational support for SDV–for VOD, we had to allocate resources at NOCs (network operations centers) to manage the platforms. We will do the same for SDV.

LaJoie: But it’s not significant.

Fawaz: No, it’s not. It’s the same model as for VOD. You’re adding switching service groups. The biggest part of the investment is really the equipment in the field. The edge QAMs, the additional statmuxes. That’s the biggest part of the dollars spent.

Werner: Edge QAMs are the predominant economic component of switched digital video today. They’ll come down over time as well. As Mike says, those are capitalized to go in the front end. On operational expense, like anything else, the plant has to be working, and you can’t have any non-responder boxes. That’s a part of the business that’s been getting continually better. There might be a bit of opex in making sure you have stuff in shape, but I think VOD and digital voice have driven most of us in that direction already.

LaJoie: The operational expense that would go up would do so only because we’re introducing more electronics. The sheer number of units goes up. So you apply the same failure rates. You may have some marginal increase, because you’re managing more physical equipment, but it’s de minimus.

Bowick: Digital simulcast also required that we clean up plant. As the plant looks better and better, from an ingress and noise perspective, the product performs very well. I don’t think we’ll see much of an increase.

CED: Node splitting–just curious, are any of you splitting nodes yet? From what to what? And while we’re at it, brush us up on how many remaining dark fibers you have coming into a 500-home node, on average? Of the four to six you started with?

Werner: I haven’t been out counting fibers lately. These guys keep me pretty busy. But I have started looking at node splitting here. The nice thing about node splitting is, it works so well, from an efficiency perspective. Say you have a market with 30 percent penetration of HSD. Some neighborhoods might have 15 percent penetration, while other real hot spots might have 60, 70, 80 percent. There, we may split to 125 homes.

But it’s all usage driven. As we hit 70 percent utilization, we issue a work order to split the node. But it depends on utilization. Usually we set it to split to 250 homes. And for us, 65 percent of our node splits are really decoupling of nodes at the headend. Maybe you had three or four nodes sharing a laser. We call that a virtual node split.

LaJoie: We call it laser fan out reduction.

Bowick: We call it a service area group reduction.

CED: I’ve also heard it called “one-to-one combining” and “logical node splits.”

Fawaz: There are two types of node splits. There’s the logical node split, which happens at hubs and headends, and is a lot easier. Then there’s the physical split. Both are success driven. As the number of DOCSIS devices in that particular service area grow to a certain level, it triggers capacity requests to split a node. Either way, it’s not significant, from a total capital spend viewpoint, today.

LaJoie: This is an important thing to fully understand. What Marwan just laid out is the flexibility we have in logical service groupings and physical node splitting. Those two things are very key components in how we manage HFC. It was envisioned in the original design of HFC. You’ve probably heard Dave Fellows say, “we build fiber to where it makes money.” The natural evolution of HFC, over time, is the gradual extension of the fiber plant closer and closer to the home. On top of that, and well before it, we can introduce multiple logical networks, with different segmentations, on top of the network.

Werner: We’re all saying the same thing. It’s the way in which we can take nodes, in the beginning, and group them in such a way as to get the maximum network utilization. That’s what keeps the economics so good. We can take two nodes in an area with low take rates for HSD, and combine them with a node in another area, with a high take rate. One laser feeds all three nodes. Put another way–there’s one downstream channel of 38 Mbps coming off of a CMTS (cable modem termination system). If, all of a sudden, we get to full utilization on a node, we can take a look at it: Which in that group of three is pushing you over the limit? Then, you either add a laser or you add a downstream port from another CMTS, which is now directed at those other two nodes. And you’ve doubled your bandwidth. The real work is on the database side, where you have to re-assign modems that moved to the new CMTS blades. But we have scripts to do that automatically.

Fawaz: As an industry, we generally talk about having ratios of 2:1 on the upstream, 4:1 on the downstreams. That’s typical for most MSOs. When you run out of that, when you get to 1:1 everywhere, then you’ve run out of logical splits. The average node size now is a little less than 500 homes passed. So when you get to a 1:1 ratio, upstream and downstream, only then do you go to physical node splits.

Bowick: And it’s all done in the headend or hub–you don’t have to travel node to node. The other thing I’d add, as Mike mentioned–a lot of this was architected from the beginning. We looked at Cox Business Services, business demand sets, to see how much fiber we’d need to serve businesses. And we anticipated traffic growth. That planning drove us to deploy “X” pieces of dark fiber to each node. But the way we look at it, the very first thing we’d do is service group reductions. Logical node splits. Once we get to 1:1 combining, then we add a second or third data channel. We do that before we go and split a node. Because that’s probably the most costly. We like to stay very pragmatic–bandwidth on-demand. Based on traffic modeling and measurement, not based on a node of a specific homes-passed size.

LaJoie: This all ties back into, what’s the bandwidth savings from SDV? What we get out of SDV is as we move more and more broadcast video into a switched digital reality, it frees up the spectrum that we’re moving it away from. To do things like what Chris said. If you want more throughput for your customers, the easiest and least expensive way is to add another DOCSIS carrier. It’s basically free. So if I can free up the QAM channels by moving things from broadcast to switched, now I have the ability to take advantage of things like dedicating a QAM channel to commercial, or adding a QAM for super fast residential data service. That’s the idea behind switched digital. It gives us another key tool for managing our spectrum.

Fawaz: If you look at the typical HFC rebuild/upgrade over the last 10 years, we’ve typically dedicated six fibers to every node. We’re only using 30 percent of that. So we’ve deployed more fiber than any competitor. Obviously, fiber-to-the-home buildouts are catching up, but today, in general, fiber is, on average, less than 1,500 feet away from any home passed. There’s more fiber in cable networks than in any other competitor.

CED: Let’s shift gears to digital video. From a purely technical perspective, what would you like to see happen with OCAP this year? LaJoie: Deployment.

Werner: Us too. Successful market deployments is what we want to see.

Fawaz: A top priority for us at Charter is to have a successful pilot deployment, to understand the technical requirements as well as the application needs. So we’re laying the foundation for that. We hope to complete it this year.

Bowick: We want to improve operational stability. We have trials going on at Cox, and that’s both with the CHILA (Cable Host Interface Licensing Agreement) TV sets as well as OCAP set-tops. We hope to expand that, later this year, to be ready to rock and roll in ’08.

Fawaz: The second part of it is laying the foundation for OCAP support in the future. So for us, that is DSG (DOCSIS Set-top Gateway) deployments. We’re doing that in a lot of our top markets.

LaJoie: From our perspective, and at a technical level, we will have all of our headends OCAP-enabled by year-end. We’re right on the cusp of having DSG installed everywhere. We right now have thousands of set-tops, either in headends or in employee homes, running OCAP software. So you will see OCAP running in production in our networks.

CED: How much of your going-forward digital set-top footprint will be OCAP activated, or at least OCAP-capable?

Werner: Most of what we buy this year will be OCAP-capable. Several will go out with native guides and applications, but we will have the ability to download OCAP into them. It’s a matter of getting DSG in place, as well as the servers. Fawaz: Just to clarify, high-end boxes, as in HD-only or HD-DVR, that we buy today, will support OCAP. And any host, two-way boxes or TVs we receive in May and beyond will support it. The low-end boxes–we’re still deploying them, but not a lot, because most of the high-end boxes are displacing the low end. Anyway, those are not OCAP-ready. Bowick: Well, keep in mind that the low-end boxes may not support the full OCAP footprint, but they will support OnRamp to OCAP, which is a subset of the OCAP APIs.

CED: Say I’m a program network, and I have an OCAP app written and ready to go. What’s the framework for getting it “to cable”–to the contiguous, all-of-you-guys cable footprint? More specifically, what still needs to happen in order for there to be a national delivery framework, from source (program network) to destination (your boxes), for OCAP or ETV apps?

Werner: We, Comcast we, have a national content delivery network today. As we look at OCAP and roll it out, a lot of our servers will sit high up in that architecture, because it’s easier and faster to do it that way. So once we get OCAP out there with scale, and we get people out there developing cool apps, I don’t think it will be an issue to get it to us.

LaJoie: Keep in mind there are bound and unbound apps. Bound apps come down with the video, embedded in the stream. For that, the distribution model is what it is today. Same way as with broadcast video. For unbound apps, the distribution would work similar to the way we do third-party apps today. First, you have to have a deal with an MSO. If you’re a third-party apps provider, and we want your application, then that application gets delivered the way any other application does. From a technical perspective, there’s very little challenge there. Now, taking the business issues, the deals we need to do, and how those applications show up–that’s a subject for another hour-long call...

Fawaz: What we’ll do is to add servers in our regional and national networks, to support OCAP applications. Those are connected nationally. So any content providers of applications will download to OCAP data centers, if you will. Then we’ll have the proper connectivity to those providers and to our regional or national networks, using VPNs or whatever. It gives us a quicker way of getting those applications to all of our headends. By contrast, in the past, if you didn’t have a regional or national network, you had to put a server in every headend.

CED: DOCSIS 3.0: Last year, all of you said you’d hold out for a straight, 3.0-based solution. Is that still the case, or are any of you softening on the notion of “pre-3”?

Fawaz: I haven’t changed my position. We’re pushing for a full 3.0 solution. In the meantime, we haven’t completely taken advantage of a lot of the capabilities already present in DOCSIS 2.0. From a competitive response perspective, 2.0 will be sufficient in some areas. Longer term, the foundation for competing with FTTH is DOCSIS 3.0.

Bowick: I agree. We took a step back and said, gee, do we want to do something pre-3? We came back to the same place as Marwan. It hasn’t made sense yet. We’re still holding out for 3.0, but we still have a lot of 2.0 out there. There’s lots of room to grow before we need 3.0.

LaJoie: We’re waiting for DOCSIS 4.0 (laughs). We’re in the same boat. But I will say that we have some pre-3 stuff we’re messing with in the lab. We want to familiarize ourselves, but, I don’t see any driving need for pre-3. DOCSIS 2.0 serves us just fine, through ’09 and ’10. And in the event that there’s a driving need, our vendors will show up with product in a way that’s ready.

Werner: We’re very bullish on 3.0, and certainly will be early adopters of it in a number of places. But we’re definitely waiting for 3.0-compliant hardware. There’s a lot of activity going on between CableLabs and the member companies, and we’re seeing promising timelines. You’ll see us doing a lot of things. Like Chris said, when Verizon rolled out their 50 Mbps, we had a few systems say, should we have a response? We’re prepared and have a plan, but then they came back and said, there isn’t any market pressure. It’s more commercial. So it’s not a marketplace issue right now, and pre-3 brings so many back office issues, you end up doing things a couple of times. That said, we’ll do what we can do to accelerate the 3.0 timeline.

CED: Speaking of the 3.0 timeline: What is it?

Werner: That’s better answered by CableLabs. But products will be in certification wave number 56 this year. It’ll be early products. You’ll see people dabble. A lot will enter the certification wave.

Fawaz: I’m very encouraged by what I hear from our suppliers about (DOCSIS 3.0) timelines, but, that question is best answered by them, too. What we’re hearing, and Tony can confirm it, but, significant commitments from existing and new vendors, for 3.0 product.

CED: Commercial services is a higher priority than in years past. To provision small/medium businesses for broadband services, do you need to do anything significant to your CMTS or elsewhere?
LaJoie: No. It’s the same stuff. It’s data service.

Fawaz: It’s a product definition vs. technology options. It’s what features you want to provide. Could be static IP addresses, could be additional storage, more e-mail accounts–but really, it’s a variation on the product definitions we’re doing today on the residential side.

Bowick: And it’s not just data. We’ve been in the commercial business for a while now. And yes, we provide a very strong product over DOCSIS and HFC, and also over direct fiber with Ethernet. We’re providing hosted IP phone services, and video, too.

LaJoie: True. We just turned on a 12-line product, based on DOCSIS, for phone, in three markets. We also have a very robust business in direct connects. Larger customers, who want a TDM product over SONET, or Metro Ethernet–the message is, DOCSIS is a very suitable technology for commercial services, of a lot of different descriptions. But more to the point is that our plant and networks are capable of offering data and telecom to a variety of business sizes.

Fawaz: On the data side, we’ve taken advantage of DOCSIS. We’ve provided products to the SMB (small/medium business) segment. We’re doing the exact same thing with our soft switch architecture. We’ve launched several Charter markets with that. And this year, every residential market will have a commercial telephony offering.

Werner: (Commercial) is a different business, and we all recognize that. The treatment is different, but the technology isn’t a big stretch. Right now, just high-speed data and voice are killer apps in that space. We think there are several years of real business there, selling good high-speed data and voice.

CED: How do you decide which type of business gets DOCSIS, versus a direct fiber?

Fawaz: Most of us have six fibers to nodes. We can extend those in, or we can go with DOCSIS. And there’s a ton of markets where we can [go] with DOCSIS. Internally, the debate is not driven by technology. It’s pure ROI. So, 70 percent of our data offering on the commercial side comes from DOCSIS. The other 30 percent has a very healthy ROI associated with taking fiber deeper. It’s usually a large business, with high-end demand.

Bowick: We’re also seeing significant growth in wireless backhaul. We intend to focus on that significantly this year. Again, the technology component isn’t a stretch. Still, there are some things we could really use. We need SIP to be fully functional and available in EMTAs. We need help with diagnostics–loop-back testing on EMTAs, and some more hardened solutions for EMTAs, especially in multi-tenant situations. We’ve grown up on the residential side with provisioning, and we’ve gotten very, very good at it–but we could use some help on the provisioning side, especially relative to the provisioning of VLANs. So, there’s still some things we need and could get some help from the vendors on.

2007 CTO Roundtable: Chris Bowick, Marwan Fawaz, Mike LaJoie, Tony Werner

CED: Speed round. In a sentence or less, what’s the CTO’s view on these “tech current events”? Let’s start with Slingbox.

  • Werner: Interesting, but niche.
  • Fawaz: Still looking for the business model.
  • LaJoie: Cool, but limited applicability.
  • Bowick: It’s a cool app.

CED: Joost.

  • LaJoie: Another cool app. Pretty compelling. Also it’s more of a responsible attitude toward network consumption.
  • Bowick: That was more than one sentence, but, ditto.
  • Werner, Fawaz: Ditto.

CED: Kevin Martin?

  • All: Pass.

CED: AppleTV?

  • Werner: Again, I’d say interesting, but niche.
  • LaJoie: There’s not much of a “there” there, but it’s interesting.
  • Fawaz: Same. Interesting.

CED: July 2, 2007?

  • Bowick: That’s my 30th wedding anniversary. Is there something else going on? (laughs)
  • Fawaz: We’ll be there. It will be a challenge, but we will be there.
  • Werner: We have small armies working on it. We’ll be there.
  • LaJoie: Same.