Level 3 tried to wrangle an invitation, but no dice.

The recent battle between Level 3 Communications and Comcast has produced an entertaining volley of press releases from each company, complete with multi-point jabs and rebuttals. Level 3 has asked Comcast for more free transmission capacity. Comcast has provided some but has requested that Level 3 pay for the remainder.

Level 3 is claiming the issue with its peering agreement with Comcast is not peering, but network neutrality. Comcast does not agree. Level 3's network neutrality argument was novel, but it failed to convince the final arbiter, the FCC, which in February sided with Comcast.

It’s a basic truism that the Internet is a network of networks. Each network does one or more of three basic things: 1) provide content, 2) transport content or 3) consume content. Local distribution networks can be thought of as content consumers; they have the networks that primarily serve end users. Each type of network has its own set of costs.

Level 3 and Netflix traffic...

Content producers must first create or procure the content, store it, and then get it on the Internet. Let’s use Netflix as an example of a content producer. There is a cost to licensing television programs and movies, there’s significant hardware and storage required to hold the content, and then there are large amounts of “onramp” capacity to the Internet purchased to ensure the content can get to the consumers. In this example, Netflix could build its own network or outsource it and buy the Internet bandwidth it needs from a content delivery network (CDN), typically operated by a transit provider.

The value of a transit provider is measured in how much of the Internet it can reach. Building a transit network that spans the globe is not cheap, either. In the example of Level 3 Communications, it dug up the ground and laid fiber in both the U.S. and Europe. A transit provider also needs to buy the most advanced routing and switching hardware available. Finally, to stay ahead of the demand curve, the transit provider has to constantly upgrade its backbone capacity.

Local distribution networks, meanwhile, must build robust networks to provide bandwidth to individuals. They must have connectivity to the transit providers to allow subscribers to access content. This usually means a content consumer will also buy Internet bandwidth from a transit provider.

Note that the transit provider charges for the same traffic twice: once coming and once going. Content producers must pay the transit provider to get their content on the Internet, and, likewise, content consumers must pay the transit provider to have access to the entire Internet. A successful transit provider connects with as many content providers and consumers as it can.

Not all consumers and producers are connected to the same transit provider, of course, because there are a handful of large transit providers that provide the same basic service. Network service providers (NSPs) – the largest being the

Tier 1 providers – understood that no NSP could connect to every network, but if they each connected, then at least they would have full connectivity to the routes that comprise the Internet. Tier 1 NSPs also expected that when interconnecting with one another, they would be sending and receiving roughly the same amount of traffic with each other’s network. If you couldn’t send and receive approximately the same amount of traffic with another Tier 1 network, then you probably didn’t belong in the Tier 1 club.

Level 3, Comcast & Netflix deal

The accounting scheme they created to simply share reciprocal traffic is peering. Since each network is sending and receiving the same amount of traffic, there’s no need to bill each other for something that’s basically a wash. Simple peering arrangements subsequently evolved into “settlement-free peering.” With settlement-free peering, each provider agrees to split the cost of the circuit between them, they each agree to fund the cost of the router and port on their network, and they agree not to charge each other for the traffic exchanged.

When a network is first built, it has no traffic and no leverage to argue for settlement-free peering.

This was the case when Level 3 built its backbone in 1998. Even before it had finished laying its own fiber, it leased network capacity from others. In October 1998, Level 3 acquired a small ISP, Geonet. Geonet didn’t have a huge customer base, but what it did have was a number of settlement-free peering agreements. Geonet had established these peering agreements in prior years, when the Internet was a much smaller place.

Assuming Geonet’s peering agreements instantly allowed Level 3 to reach large portions of the Internet without having to pay transit fees. It took several years before Level 3 grew its customer base large enough to truly justify settlement-free peering with all of the major providers at the time. 

In the late 1990s, UUnet (now owned by Verizon, but previously owned by WorldCom) was the most dominant backbone (it continues to be one of the highest-trafficked parts of the Internet). At the time, Level 3 did not have a peering agreement with UUnet, the last major part of the Internet to which Level 3 lacked free access.

What made it difficult for Level 3 to get peering with UUnet was the traffic imbalance. Without Level 3 having a significant customer base, either consumer or producer, UUnet simply refused to peer with Level 3.

With the dotcom boom in full swing, it was easy to find content-producing start-ups with funding and Web servers eager to provide content and (hopefully) make millions of dollars doing so. These customers were attracted by Level 3’s colocation centers, large data centers where customers can place their equipment directly on the backbone. This threatened to cause imbalance at the peering points, because Level 3’s content producer customers originated far more traffic than they terminated.

One way to bring the balance of traffic in line was to target customers who would consume traffic. Level 3’s acquisition of Xcom technologies in April 1998 served this purpose. While the main goal was to acquire a key voice over IP technology, Level 3 also acquired a dial-up modem business that eventually became the largest revenue generator for the company for many quarters.

Peering protocol...

Level 3’s dial-up modem business did two things: It brought eyeballs to the network, and it generated revenue by means of an accounting scheme called “reciprocal compensation.” But that revenue was, in fact, not reciprocal at all. Reciprocal compensation was a construct that the telephone companies had long before created to compensate themselves for carrying and terminating telephone traffic originated by others. The expectation was that companies would originate and terminate the same amount of traffic, so it would all be a wash. But in case there was an imbalance of traffic, someone would pay. This scheme was in a sense a predecessor of peering.

There was a funny thing about Level 3’s modems used for receiving dial-up Internet access calls. They terminated a bunch of calls but never seemed to make any calls themselves. Recall that if you owned a bunch of modems that did nothing but terminate calls, then you had the right to be compensated for terminating those calls. Level 3 became a major modem provider for one of the largest dial-up ISPs for its time, America Online, and for several years, the dial-up modem business was very good for Level 3. It not only generated the majority of the revenue for the company, it also served to keep Internet traffic balanced on the backbone. Maintaining balanced traffic helped Level 3 continue to grow its settlement-free peering base.

AOL was paying Level 3 for that service but sought to reduce its payments to Level 3. So it built its own network (the AOL Transit Data Network, or ATDN) and then turned around and asked Level 3 for settlement-free peering.

This, combined with the shift away from dial-up modems toward broadband, effectively killed the revenue of the dial-up modem business for Level 3. And remember, it was the dial-up modem business that helped keep Level 3’s traffic balanced. (By the way, the director of engineering at AOL at the time who orchestrated the shift from paid transit to settlement-free peering was John Shanz, now the executive vice president of national engineering and technology operations for Comcast.)

So as we turn back to the dispute between Level 3 and Comcast, it’s clear that all companies involved (including Netflix) have a number of factors that compose their cost of service. As any good business would, each strives to reduce its cost. As a network provider, one always strives to use settlement-free peering where it can, where it makes sense and where agreements allow.

Level 3 is trying to shift toward the CDN model. In so doing, it faces the same challenge it did as a colocation provider. CDNs work by offering to outsource the storage and distribution of content from the content provider to the network. The CDN takes on the burden of storing and delivering the traffic. A CDN sends far more traffic than it receives, which is not consistent with settlement-free peering. In fact, the numbers being quoted by both Level 3 and Comcast for the Netflix deal would cause Level 3 to send nearly five times as much traffic to Comcast as it would receive.

Peering arrangements...

A major factor affecting Level 3’s profitability is that the cost to create and deliver bandwidth might not be covered by the revenue generated by the service. In the late ’90s, the price of bandwidth was on the order of hundreds of dollars per Mbps. Today, the price of bandwidth can be lower than $10 per Mbps (Cogent Communications is currently marketing itself as “Home of the $4 Megabit”). Part of this is the result of a competitive market; part of it is the result of irrational pricing. But to continue to win business with prices like that, transit providers must keep their network costs as low as possible. Using settlement-free peering instead of paying transit is one way to do that.

Settlement-free peering agreements have been shaped over the years, and ISPs that have balanced traffic exchange with one another view peering as a mutually beneficial relationship. But for everyone else, the Internet community has coined the phrase “sender pays” to point the finger at who should bear the burden of cost.

But this dispute between Level 3 and Comcast could lead to a tighter definition of what constitutes “roughly equivalent” traffic exchange between backbone networks. Peering agreements could become more complicated and more formal. Today’s agreements are essentially “memoranda of understandings,” but this dispute could create the need for detailed contracts. New networks seeking to build their business and achieve peering could have a much harder time, and possibly the field of those that are entitled to settlementfree peering could narrow.

Had Level 3 won this dispute, the precedent would have led to an unsustainable situation.

In a climate of government regulation of network interconnection agreements, network service providers might no longer offer settlement-free peering at all, and networks would have to accept the substantial cost of keeping records of originating and terminating traffic loads, just like the old telephone companies did. It could cause the Internet to take one giant leap backwards.

While peer-to-peer traffic dominated the Internet a few years ago, now video dominates. More video being sent over the Internet requires more network infrastructure and more bandwidth. How will that additional capacity be paid for?

In the end, it’s the subscriber who will wind up having to pay more money to keep the model working. Whether that money is in the form of increased broadband fees paid from one network to another or in the form of increased subscription fees paid directly to the content providers (which indirectly flows to the transit provider and the consumer network), it may be the only way to continue to enjoy the connectivity that keeps up with the pace of our consumption. This may lead to the end of flat-rate pricing for broadband end users, so that those who consume video will pay for their fair share of network capacity.

Operating the Internet is a business like all others, but it has a lot of moving parts, each with differing cost and revenue models. Level 3 continues to try to find a business model that, for the first time, would allow it to earn a profit. Part of its model seems to be settlement-free peering with Comcast, without the traditional burden of originating and terminating roughly equal traffic loads.

In the end, it’s the Internet community’s own rule of thumb that should apply: sender pays.