Vendors Struggle to Build Open Access Platform

Integration of MSO and ISP Billing, Back-Office Systems a Headache

Somebody sure answered Time Warner Cable's prayers, because it's having a heck of a time with its trial-architecting a platform to put multiple ISPs onto its cable broadband system in Columbus, Ohio. Using teams of technicians from Time Warner Cable (TWC), hardware vendors, and interconnect specialists from middleware, mediation and billing companies, the company is pushing to link ISPs to its cable broadband network under time restraints imposed by the approval of its merger with AOL.

TWC was to have EarthLink on board the network first, followed in 60 days by AOL, and two ISPs 90 days after that. Says one technician involved with the project: "They said, 'Let's not do it half-ass, it won't be viable that way,' ... and what we've done is work very hard."

Unprecedented Scope

What the engineers and project managers are trying to do is unprecedented, which is to take a cable system-a one-way network originally designed to send traditional cable programming into the home-and turn it into a platform that not only allows as many ISPs as possible to link with it, but lets ISPs automate ordering, provisioning, network management, customer care and billing, and bandwidth management with the MSO's back-office systems.

Issues are yet to be worked out, such as IP address management, identifying and routing traffic from various ISPs quickly and efficiently, and identifying network glitches between the cable modem in a household den and the interconnect point for the rival ISPs on Time Warner's cable infrastructure.

And there are the unknowns that won't show themselves until customer behavior begins to show itself once such a platform is up and running. For instance, does the end user call the cable company when Internet service is down? Or does he call the ISP to complain? If he calls the cable company, how does the cable company notify the ISP that the problem exists? How do you handle tiered pricing, when one member of a household orders 128 kbps service and another orders more bandwidth?

Reconfiguring MSO Network

The teams in Ohio are in effect reconfiguring MSO equipment so their operation support system can pass on the Internet end user's information, including billing, usage and quality of service data, to the ISPs' back-office systems-an experiment that has everyone learning on the fly. "This is all brand new stuff; it does have a long learning curve," says Mike Luftman, spokesman for the Time Warner project. "We're looking at a tiered level of service. You need to be able to monitor that, share the billing information with ISPs, and it all has to interconnect. We may do the billing, or they may do the billing. It's a complex task."

Not only is the project complex, but Time Warner was keeping a tight lid on information surrounding the project. For instance, Luftman would not disclose the other vendor participants. But AOL in January signed an undisclosed multimillion dollar, multi-year contract with Portal Software to lease its Infranet software platform. When asked if the signing of the Portal contract was connected with the Time Warner project, Luftman said: "That's a separate AOL deal."

Defining Billing Challenges

Before ISPs on broadband cable networks can bill end users, they must be prepared for large numbers of subscribers passing by its interconnection point, and determine whether those customers are theirs or not, and whether the content that belongs to its customers is billable or not. According to some estimates, more than 27 million customers will use high-speed Internet by 2003, generating $8 billion in broadband revenue. That means these MSO-ISP systems will have to handle an increasing load of customer services, track recurring charges and usage-based products, assign them to the right customer, and assign and reassign IP addresses as customers churn or access multiple ISPs in a single day.

Figuring Out Pricing, Billing Schemes

Billing specialists at CSG Systems, who have participated in technical trials surrounding multiple ISPs on cable networks, say cable network owners like Time Warner and AT&T Broadband are using network monitoring to determine how much bandwidth consumers and ISPs are using to determine what to charge ISPs that ride their networks.

"Because these are technical trials, the MSOs are offering a flat-rate charge, with some usage-based billing," says Sally Else, vice president of product management at CSG. For the most part, MSOs are selling bandwidth wholesale to the ISPs and letting them parcel it out and charge their customers accordingly. But the MSO needs to determine the right balance between making a profit from the ISPs and its own customers and finding a price that consumers will accept. To accomplish that, the MSO needs to know who's using how much bandwidth.

According to Randall Cardinal, CSG's chief technology officer, CSG uses a mediator to mark and categorize the usage portion of the traffic, including e-mail, Web hosting space, video-on-demand streams, and voice over IP-originated streams. CSG's mediation partners include AP Engines, Narus Xacct and others. The mediator interprets the packets and looks at the usage designator on the record that categorizes it by usage type. "All that is coming into our usage rater via the mediation process," he says. "Packaging can come in premium formats and usage formats to determine the amount of each usage type included in the fee." For instance, if the basic package includes 5 megabytes for Web hosting, and the ISP is actually using 10 megabytes, the MSO can determine what to charge for the extra bandwidth. "They're seeing which thresholds hit the mark," Cardinal says.

Determining ISP Profitability

"To help our MSOs to determine which ISPs are more profitable, we gather the detailed usage and the summarized usage and store it in data warehouses. The MSOs can dip into that and perform a financial analysis," Else says. The analysis lets the MSO determine how to price its wholesale bandwidth, as well as individual services for its own customers. "At some point, there's a threshold that's appealing to the consumer that also makes sense to the end user," Cardinal says.

MSOs are also trying to determine the best way to bill ISPs and end users. For instance, Cardinal says, the MSO might simply put the access charge on its statement and let the ISPs bill the other charges, such as recurring and usage charges, on the consumer's credit card.

MSOs also are considering taking the ISP's customer usage and access charges and billing the end user directly as a service for the ISP. For instance, $50 in access charges as well as usage for other services would appear on the cable company's bill to the end user. A third model has the ISP billing the end user for everything based on what it's been charged by the MSO for bandwidth.

Ordering, Provisioning Pose Challenges

Building ordering functions to handle the various avenues of customer contact will be complex. Technicians and developers have to be able to combine the cable network back office to the ISP OSS to handle those orders quickly and accurately. Problems will multiply as more and more ISPs link to the cable network; as more customers come online, both back-office systems will show the strain.

Handling ordering and provisioning from several sources, says Rangan Wilkinson, vice president of broadband business development for AP Engines, is difficult. The company is providing the mediation software and provisioning, as well as the OSS-to-OSS integration for the Columbus trial.

Customers can order Internet service by phoning the cable company or the ISP, or by ordering the service over the Internet-and there are more possible avenues, such as mailing of CD-ROMs to households for a 90-day free trial. "If someone calls in to a Time Warner or AT&T service center and says, 'I want your cable modem, but I want EarthLink or Juno as my ISP,' that's one sales channel," Wilkinson says. At that point, the cable network owner has to pass those orders on to the chosen ISP.

Or a customer may order Internet service directly from a particular ISP via the Web, but that customer may also want to order a cable modem from the MSO. In that case the ISP has to notify the MSO about the modem order, which in turn installs the modem for the customer.

Or the order may come in through the MSO call center. The MSO will have to provision and activate the cable modem, then send the completion order over to the ISP so that the ISP can configure addresses, level of Internet service such as e-mail, and level of bandwidth ordered. All this needs to be done automatically.

According to Luftman, Time Warner has completed the business rules surrounding provisioning so the customer can order through either the ISP or Time Warner. Development of the tracking software was almost finished in January, and installation was to begin in February, he says.

Trouble Reporting

Trouble reports, including interruption in service, inefficient bandwidth delivery, broken cable modems and a host of other problems, will have to be sent through the combined platforms. This task, too, will grow in complexity as more ISPs are linked to the network.

"If a consumer calls in and says, 'I can't get Internet access,' and that call comes through the ISP, and the ISP is looking and says, 'Well, the Internet pipe is OK, the e-mail services are OK,' now they need information about the broadband network," Wilkinson says. "Maybe there's a cable outage in that particular area which is affecting this consumer's Internet access. If so, that information is managed by the MSO and needs to be sent to the ISP so it can tell its customers." If a call, however, about a service outage comes in to the MSO, the MSO inspects its portion of the network. Finding nothing, they then have to query the ISP to see whether an Internet connection is down, or determine whether the e-mail or Web servers are down for maintenance. The ISP has to let the MSO know so it can pass that information to the customer.

But the flow-through between the MSO and ISPs is still being worked out, a vital function if the entities are to handle the burgeoning number of ISP customers and the service combinations they order.

"It's a huge job," Wilkinson says. "We've been working with IPDR on defining at least a de facto industry standard interface that allows us to plug ISPs into a cable infrastructure so we can send messages back and forth. Right now it's an XML interface that describes all the business procedures that need to be supported, and it shows all the messages that have to go back and forth."

IP Address Management

Then there's the problem of doling out IP addresses to eventually hundreds of thousands, if not millions, of customers, a role the MSOs will take on in an effort to control customer information and billing and settlement issues. Single ISPs running over traditional networks can usually handle IP addressing with a single Dynamic Host Configuration Protocol (DHCP) server; this will not be true with multiple ISPs trying to use the same DHCP server. As it now stands, ISP technicians manually configure the IP addresses on those servers when fulfilling orders. But once an MSO begins receiving thousands or even tens of thousands of orders a day from ISPs connected to its system, it will be nearly impossible to keep up with orders if techs have to manually configure those addresses.

The answer, says Erik Gilbert, Cisco Systems solutions manager for managed broadband access, is to develop tools that will let one tech automatically configure thousands of IP addresses. "Think of a technical person spending a day doing this; maybe he or she manually handles several hundred customers. All of a sudden you're seeing a thousand customers and you need five or six of these people; then you're at 100,000 customers and arguably you need 20 or 30 of these technicians. Now you've got a big problem." He says automated DHCP configuration software tools could be ready sometime later this year.

"What anyone will have to do," says Tom Goldman, president and COO of Apogee Networks, "is be able to adjust to a network where IP addresses are dynamically assigned. When you determine a piece of content, who consumed it, and you get ready to put it into your billing system, you have to query the DHCP server, to identify who that IP address belonged to at that point of time." Doing that for millions of events per hour could affect the ability of the system to align the right addresses to the customer.

Configuring Routing Techniques

Technicians in Columbus are also experimenting with traditional routing protocols and techniques to find the best way to route packets through the integrated networks. "Normal routing protocols on the Internet take into account the interface on a box, the interface that a packet is coming in on, and the destination address on that packet," Gilbert says. "If you want your traffic ... to go through a particular ISP, you have to do something a little more."

In traditional destination-based routing, he explains, the traffic just goes where the routing protocol tells it to go. "It could be the case that your traffic is being routed through UUNet as an ISP, and then all of a sudden a router breaks or a link gets congested, and now your traffic is going through another ISP. But if you're a [different] ISP you say, 'Well, it's my customer, I want the traffic to come through me.' "

To prevent the traffic from going through a rival ISP, technicians can set up the routing so the original ISP's traffic, heading toward the exit on the MSO network, can be sent to a particular router at the edge of the MSO.

Turning on Policy-Based Routing

"That's where you turn on the policy-based routing feature," Gilbert says. That feature overrides the normal destination routing and starts to look instead at the source address field. And because the source has been assigned to EarthLink in the DHCP server, and the end user's PC has the source address that belongs to that ISP, it routes the traffic to that ISP. "And so, no matter what happens in the regional network, that traffic will go to that ISP," he says.

But Cisco, in a white paper on the project, believes there are weaknesses with policy-based routing. For instance, the PC is assigned the ISP address by the MSO provisioning system; it requires configuration and routing expertise to ensure that both the MSO and the ISP route packets properly in both directions. The technology is also difficult to scale because of complexities of management.

Using VPN Routing Approach

A virtual private network setup can be used to more accurately route traffic within an MSO. This is achieved by setting up VPNs on the regional MSO network for each ISP to handle that ISP's traffic. "The nice thing about this is that the network routes it anywhere it wants, routes it around failures and redundancy, but the egress point from that regional network can be located in a number of places," Gilbert says. "You can have two, five, or six interconnect points with a particular ISP, and if one of those peering points goes down, the traffic will still get there through the other interconnect points."

But Cisco also cited weaknesses in this approach in its white paper. First, it requires MPLS on the entire backbone; because it's limited to a single network, it cannot cross network boundaries. Cisco also sees weaknesses in the amount of memory needed for such large-scale systems.

But Will It Be Fair?

The background to all of this, of course, is whether the owners of the cable networks, such as AOL Time Warner, will treat rival ISPs equally, or use their networks in some way to favor one ISP over another. The industry is not convinced that "open access" is going to be what it's supposed to be. Considering how RBOCs treated CLECs and DSL providers through stalling tactics and favoring their own customers over their rivals, can anything different be expected from the cable network owners?

Time Warner Cable has assured the FTC and the FCC that rival ISPs will be treated equally. But questions arise about whether some interconnection points are better than others.

As for the points of interconnection, differences in quality of service will depend primarily on the ISP.

"From a purely technical perspective," Cisco's Gilbert says, "the size of the circuit the ISP is interconnecting and possibly where they're interconnecting into a regional network could affect the service-or if they're connecting into a more remote location that takes more hops. But I don't see that as a likely scenario because most of these regional networks are fairly symmetrical, so the interconnecting points are going to be equivalent."

"Peering issues are resolved with money," says Ori Cohen, CEO of Narus. "If you're the 10th ISP and want the speed of the first ISP that interconnected ... it will have to be by having a faster interface, a faster router. There will be some network differentiation from the head end to the Internet backbone. Big ISPs will be able to buy bigger pipes, while smaller ISPs will buy T-1s and cram as many customers on as possible." The quality of service could also be affected by the business layer the ISPs put on their network level, such as the kind of applications they add to differentiate themselves from the others. If they're improperly layered, that could degrade bandwidth performance.

Another concern is who manages the domain name servers, Cohen says. "My guess is there is some concern the cable operators will build their own fast DNS for their users and route other users to slower ones, thus degrading the user experience" with rival ISPs.

Time Warner, AT&T Models for Future?

The eyes of the telecom world seem to be on the various multiple ISP technical trials in Columbus, and AT&T Broadband's trial in Boulder, Colo. Pressure for open access comes from several sources: the FTC, which made it part of the AOL Time Warner merger; the FCC, which wants the company to eventually include instant messaging and interactive TV in the open access formula; and members of Congress, some of whom prefer that the marketplace dictate the level of open access. Then there are the ISPs you've never heard of that need to know what's at stake so they can plan accordingly.

If it works, many believe the architecture that Time Warner is building in Ohio or AT&T is configuring in Boulder may become open-access models for letting the "little" guys in. What remains to be seen is whether the owners of the cable broadband networks will ensure open access through their service level agreements and other dealings with their ISP rivals. Perhaps that's something that can't be integrated so easily.