Maybe it was the adrenaline rush of being essential during a very real crisis. Maybe it was the thrill of proving to Excite@Home and the world– but especially to Excite@Home– that cable operators aren't as dumb as fireplugs after all. Maybe it was both.
Whichever. Regardless, hundreds of AT&T employees decided to be the hands that caught the broadband data subscribers dumped by Excite@Home on Saturday morning, Dec. 1, 2001. And catch they did: More than 850,000 high-speed Internet customers, originally serviced by Excite@Home and AT&T Broadband, were shifted onto a new AT&T Broadband network–in six days.
Did customers applaud the switchover? Of course not. No matter how swiftly or carefully AT&T restored service, people are people. Change can make them uneasy. Lost e-mail definitely makes them angry. Consequently, and predictably, AT&T Broadband was blistered by the mainstream and trade press.
But none of that clouded the focus of "Project Redwood," AT&T's code name for a detailed contingency plan it put in place in early October. It was then that a darkened E@H, and/or a hostile split from it, became alarmingly plausible. (Code name descrambler: Excite@Home is headquartered in Redwood City, Calif., at least until it closes at the end of this month.)
There wasn't time to worry about bad press. From early October until Dec. 6, the Redwood team–consisting of about 400 employees from at least four major AT&T units (Broadband, Business Services, Labs and Network Services)– worked 18-hour days, seven days a week, to assemble backup plans. If E@H really did shut down, emergency migration procedures would be sorely needed.
The potency of the emergency was real. Huge parts of AT&T's broadband data service relied on Excite@Home. Consider that AT&T owned the HFC plant that connected its cable modems to its customers, but E@H ran the databases that held critical identifying information about customer accounts (name, phone, address) and cable modems (Media Access Control, or "MAC" address). And, AT&T owned the glass that lit the nationwide backbone, but E@H ran the regional networks that moved data between the backbone and headends.
Without a swift and solid backup plan–executable with or without E@H's cooperation–AT&T was at risk to lose some of the 850,000-plus customers it spent five years hooking up. Annualized, the $40 those customers contributed every month equated to $408 million in at-risk revenue.
Unanimously, Redwood participants recall the six-day, 'round-the-clock cutover as an exhausting but strangely uplifting experience. They'll forever remember Redwood, they say, as one of their most profound professional accomplishments.
Still maneuvering as an act-as-one group, it follows that none wanted to be singled out for specific mention in this account of what transpired between Oct. 8 and Dec. 6, 2001.Ripened resentment
The kindling that became the winter bonfire between Excite@Home and AT&T Broadband had been ready to ignite long before October 2001. It was death by 1,000 cuts: Successive resentments on both sides bred a working environment sorely lacking in mutual respect, or trust.
In essence, cable saw E@H as an over-rated Silicon Valley jock, too focused on content and equity greed and not enough on customers or day-to-day business operations. Excite@Home viewed cable as stodgy, slow and downright clueless about the Internet.
One of the longer-running spats between AT&T and Excite@Home involved an E@H database called "BOS," for "Back Office System." BOS was a big part of how E@H stored customer and modem data collected during provisioning. It was also a control point. Both sides knew it.
In 1999, when AT&T (then TCI.net) became wary of a long-promised, never-delivered replacement of BOS, it requested and received reluctant permission from E@H to get BOS duplicates. The database replicas turned out to be a key reason AT&T was able to switch so many customers over so few days.Working against an invisible clock
Because more was unknown than known when the Redwood team convened on Oct. 8 to start planning, it began with three scenarios: 1) Lit-to-lit without E@H cooperation, where customers were moved from a working Excite@Home network to a working AT&T network; 2) Lit-to-lit with cooperation, where E@H went down but helped customers migrate to AT&T; and 3) Dark-to-lit without cooperation–which is where everything wound up.
Certain items immediately topped the requirements list. One was provisioning software. Another was swift receipt of DS-3 circuits to headends–a nearly oxymoronic notion–and a regional network to link to the backbone. Third was a way to synchronize with field employees: Part of the switch required physically removing cables from E@H's routers/switches, and hooking them into new, AT&T-owned routers/switches.
Customer care was huge, as was a marketing plan for handling customer communications and outage-related refunds.
The Redwood team surveyed what it had on hand to feed into the plans. An alternate provisioning system, developed for AT&T's open access tests in Boulder, Colo., moved quickly into the foreground. The provisioning software, developed by AT&T Labs and called "SAS," for "Service Activation System," was immediately honed for service. That meant including the ability to accept data from the duplicate BOS database. Other BOS data was poured into the billing and electronic mail systems.
AT&T Network Services expedited the DS-3 circuit deliveries; marketing developed notification plans and refund scenarios, and customer care devised a way to staff up quickly, for an inevitable onslaught of calls.Living the plan
There's a sizeable conference room on the eighth floor of AT&T Broadband's Englewood, Colo. headquarters, which will be forever remembered as the "situation room." A paper sign on the door reads "@Homeless Shelter." Black and beige beanbag chairs buttress a long conference table that forms a right angle with a couch and coffee table. There's a small fridge, and a corner filled with PCs.
On Nov. 30, its walls, papered with chain-of-command and reference charts, flanked 18 fidgety Project Redwood people. It was Friday night. The bankruptcy judge had ruled in E@H's favor earlier that afternoon. AT&T and E@H executives were still negotiating, trying to find a mutually agreeable arrangement. The talks continued into the night.
Redwood was on red alert.
Two nights earlier, the team noticed Excite@Home removing CMTS flash memory, which made the devices impervious to rapid reconfiguration. Also suddenly missing was CMTS telephone access, so that AT&T couldn't dial into their CMTS units. The telephone link would resume in a few days, E@H told them.
Coincidentally or not, "a few days" surpassed the scheduled decision by U.S. Bankruptcy Court Judge Thomas Carlson about whether Excite@Home could terminate its cable contracts.
Redwood got the "go" call at 9:43 p.m. Instructions: Move Oregonian customers off of the E@H network at midnight. Project Redwood moved into action. At 12:01 a.m. on Dec. 1., it began the cutover. A dozen telephone bridges were opened, for ongoing conversational coordination among locations. A remotely-located "commander" calmly ticked off the 45 steps to be completed, per CMTS. Field staffers enacted the detailed checklist, announcing the completion of each item.
The situation room suddenly felt like an air traffic control center, participants said, or a NASA Space Shuttle launch.
A few hours into the Oregon switch, Excite@Home's NOCs (Network Operations Centers) stopped accepting AT&T calls. Just after that, E@H shut off all Tier 2 customer support. AT&T high-speed Internet customers seeking help wouldn't get it from E@H.
The energy was palpable: Some Redwood participants worked 36 hours straight, and still didn't want to leave the vortex of the situation room. All brought changes of clothes. Some got stomach aches and rashes.
Staffing was divided by technology type: DOCSIS or proprietary. After 50 percent of the connected modems came up on a particular CMTS, the field expert bolted for the next headend, while someone else stayed behind until 90 percent of the modems were up.
By dawn on Saturday morning, Oregon was done–5.5 hours ahead of schedule. Stunned, Project Redwood communicated its first success over the elaborate internal outlets it had devised: E-mail updates every two hours, telephone round-robins, a toll-free line, a Web site.
Soon, though, AT&T started seeing cable modems in its other markets de-registering–as if the lease time of dynamically-assigned IP addresses had been shortened, and refresher IP addresses not assigned. All of AT&T's data subscribers were without service by lunchtime on Saturday, Dec. 1.
Only one market, representing less than 10 percent of AT&T's total broadband data subscribership, had been migrated off of Excite@Home without service interruption.
On the other hand, Redwood was running like clockwork–much better than anyone had anticipated. The smooth, swift work in Oregon prompted AT&T to accelerate the next switch–Washington State–by 24 hours.
Moving roughly West to East, the Redwood plans were in motion. There was no turning back.
Each time a market completed a switch, it and other already-cutover markets briefed the next scheduled market, to talk about what did and didn't work. That part alone was invaluable, participants said.
Customer care left phone messages for affected customers, but those customers still had to find their way back into the fold. Making that painless was important: Unhappy customers churn, and alternative broadband providers were already beckoning.
To mitigate the pain of switching for consumers, and to eliminate the need to send a CD-ROM to customers to reconfigure their PCs, AT&T Labs developed a piece of software known as "the Configurator." It reconfigured customers' PCs to remove E@H software, and to reorganize domain names, personal Web addresses, and e-mail suffixes to AT&T's new domain, "ATTBI.com." The point was for customers to resume service effortlessly, and not have to change things on their PC.Dixie cup champagne
Market by market–Oregon, Washington State, Texas, the San Francisco Bay Area, Illinois, Colorado, Utah, Pennsylvania, Michigan, the Rocky Mountains, Sacramento, Calif., and Connecticut–switch after smooth switch occurred, each faster than scheduled.
On Dec. 6, six days after it all started, they were 98 percent done. Battle weary and bleary-eyed, Project Redwood team members paused to sip champagne from Dixie cups and let it sink in: The plan worked.
Looking out from inside the situation room, 850,000-plus customers had been switched in six days–well over a week sooner than imagined. Major success factors included the BOS database duplicate, and the early work on SAS provisioning, for starters. But mostly, it was the precision teamwork between everyone involved, without the exasperation of professional turf protection, Redwood participants said.
Problems still lingered here and there, as newspapers diligently illuminated. AT&T will be lucky, one paper taunted, to retain customers after the switch.
One definition of luck, though, is that it happens when preparation meets opportunity. And in that sense, Project Redwood was very lucky.