Road littered with OSS, billing failures

Billing managers must walk carefully through minefield of vendor choices

The billing project manager is staring into space. He’s thinking about what he’s just been told by the CEO at the department meeting. Within 18 months, his company will launch a wireless Web service to let customers purchase goods, access stock quotes and see sport scores with portable handsets. The company also wants to give residential customers access to multiple ISPs over leased cable lines. And, of course, it wants to be able to sign up customers, offer discounts, and track, rate and bill for it all.

The billing manager now faces a choice that increasingly pops up in the reinforced bunkers that house telecommunications billing systems: How do I change my billing and OSS to handle the business rules surrounding these new services? Do I “decompose” my present system—dismantle the pieces I don’t want—and replace the traditional billing and OSS modules with so-called best-of-breed components? Or do I purchase a software suite from a single vendor with all the functionality and flexibility to meet my new business model?

Even if the software suite is advertised as “preintegrated,” there are plenty of integration headaches and new APIs ahead. The service provider then has to test and retest the flow-through to ensure that the OSS receives and processes the switch activity and customer information and sends it on to the billing system without burping out a stream of mistakes. All the while, the old billing system must be up and running to collect revenue and stay on top of costs. And you can bet you don’t have the staff to do it.

World scarcity of integrators

“Even if telcos were to try it, they more likely than not have a shortage of knowledgeable integrators,” says John Schmidt, practice leader for New Media and Communications at AMS. “At the typical telecommunications provider, you rarely have an internal IT staff that has the knowledge to custom-build all the code you need for the whole range of OSS functionality and deal with the integration issues,” he says. But telcos are finding ways to get new functions in, even if it means adding them upstream from the billing systems or building new systems parallel to older ones.

But can providers buy that all-encompassing software system, install, integrate and test it and introduce new services in a reasonable time? Are there vendors—whether independent or part of a partnership of companies—that can easily modernize the end-to-end function of the carrier’s billing and OSS? It depends on who’s telling the truth. Billing managers should be skeptical when vendors claim they can install complete, plug-and-play, preintegrated software systems, Schmidt says.

“There is this false notion that there is an end-to-end, off-the-shelf solution,” he says. “There is no single preintegrated software suite that supports [everything] the enterprise needs. Usually those systems supply maybe 40 percent to 50 percent of the necessary functionality at best. If a company goes out and buys the best-of-suite [package], you get the best with the bad. The ordering function, for instance, may not be up to speed, so you’re compromising in some areas. There’s a trade-off.” But service providers, when considering a suite of “preintegrated” products or a single module, should weigh two factors: richness of functionality and implementation speed, Schmidt says.

ILECs: Finding some flexibility

Some ILECs are in fact implementing best-of-breed applications in parts of their business—specifically in new areas such as cable, broadband and Internet services, Schmidt says. “The lines are blurred because ILECs are becoming CLECs, and as CLECs grow, they experience all the problems established ILECs face,” Schmidtsays. “Anytime you put applications together that have been developed independently, whether they be best-of-breed, off-the-shelf applications or home-grown legacy systems, there will always be some overlap and inconsistency issues that have to be resolved.” Or ILECs may want to implement a product suite to replace in-house legacy systems to simply control the runaway costs of endless application development, he says.

When to cut bait

But most ILEC systems are like a ball of tangled up fishing line—disparate systems with separate data talking to each other in a snarled Web of point-to-point connections. “Decoupling the existing linkages and implementing a new best-of-breed application with a nice simple communications infrastructure is not easy,” Schmidt says. “If you’ve ever tried to unravel a knotted up fishing line you’ll know that it’s usually a lot quicker to cut the line, even if it means losing the fish and starting over. This is why it’s so important to use middleware technology and implement it with a repeatable methodology. The middleware tools help to connect existing legacy systems with minimal changes, and the repeatable methodology avoids the high failure rate of large-scale enterprise projects.

“There is still a lot of point-to-point integration going on, but application vendors and product suite vendors are starting to put more emphasis on supporting middleware technology,” Schmidt says. “Some of them see it as strategic—they can differentiate themselves by having a better integration strategy—while others are doing it because some RFPs from end-user organizations are starting to add it to their list of requirements.”

EAI vendors have a way to go

Enterprise application integrators face three chief challenges if they are to succeed in the integration space, Schmidt says: · Implementation: Getting large-scale projects up and running is very difficult. A good EAI tool is only the first ingredient. An organization also needs a strong corporate commitment, engaged managers and users, skilled staff, and a repeatable methodology. · Breadth: Building high-quality, low-cost connectors to a wide range of vertical industry plug-and-play adapters. · Partnering: Integrating different work flow and process automation tools with their message broker capabilities—either through partnerships with other vendors or through acquisitions.

· “Another problem with these application server platforms is that they can be pretty complicated,” says Floyd Berus, executive vice president at Alltel Information Services. “It takes a while for IT organizations to learn how tune them correctly. Adoption of these standards by an organization can also mean that the development organizations need to learn new tools and languages. The learning curve can be pretty ominous.

“Finally, we are still learning how to use these standards and platforms. For example, there are no good design patterns or reference systems to help software designers make good choices. For all intents and purposes, most organizations are attending the school of hard knocks.”

Time to market unknown

The incumbent carriers can choose to buy the system-wide software package from such vendors as Cisco, Lucent/Kenan, SAP and several other vendors, for instance, and pay that vendor to install, integrate and test the new system before bringing it online. But that can be extremely expensive, especially considering the number of players involved—program managers and teams from at least three entities: the LEC, the billing and OSS vendors, and the integrators.

The time it takes to launch a new system is anyone’s guess until someone gets inside the legacy system and investigates the status of the code and the health of the database. And it’s difficult to get the staff from the telco, the system vendor and the outside integration team to work in synch. “What we find to be the most difficult are getting the subject matter experts in the other companies aligned,” says Michelle Nowak, vice president of product management at Info Directions Inc. “The new people working at [the incumbent] weren’t around when the first system was put in, and they don’t know where the data is.”

“The OSS road is littered with many failures of integration of systems,” says Jim Allsman, OSS program leader at Frost & Sullivan. “Sometimes the integration can literally take years. They—usually whoever decided to go that way—they lose their jobs. They’re caught in this internal struggle; the more they get into something, the more behind [the integration] gets. A lot of companies claim to have interfaces that don’t exist. So when a service provider goes out and buys it, … the vendor, during the honeymoon phase, says, ‘Oh we can do that!’ But they either don’t have [the interface] or it doesn’t work.”

Sleight of hand?

Instead of tearing out the old system and starting from scratch, RBOCs and older LECs find a simpler way to tie new services into their existing billing and customer care systems, says Bob Marino, president of Convergys’ information management group. In a sense, they surround the front end, so the customer doesn’t see the difference in the back office. In the short to mid-term, merged carriers might integrate the front-end systems while keeping their separate legacy billing and OSS systems in place. That postpones for the IT departments the arduous, risky and expensive task of moving millions of customer names and other billing data onto a single platform.

It’s an imperfect way to get an imperfect view of the combined subscriber bases, but carriers increasingly use this method. The carriers might link them through the front end, so a CSR at a call center somewhere can draw on the separate customer databases. They often tie the CSR systems together, so that if a customer calls in to order new wireless service, for instance, the CSR might have a separate screen or even a second VDT to type in the customer information. (See “Technical Tutorial: Enterprise Integration,” Billing World, June 2000.)

Transparency at customer contact point

This approach is especially helpful when the LEC merges with a company that already has a billing system. “The carrier says, ‘I can’t take out these systems, so I have to look for transparency for the customer contact point,’ ” Marino says. “Therefore, they may look for a common front end and a convergent, consolidated billing back end. Suppose the telco wants to provide local, wireless, ISP, long distance, paging and video service—they put a ‘wrapper’ in the front and a ‘wrapper’ in the back, by the bill. The wrapper in the back is a bill aggregator, which takes all those systems that talk differently and consolidates them into one bill. So it looks like you got your single bill from a single entity.”

But that’s one short-term answer for getting a service to market quickly. It doesn’t prevent typical problems that can pop up when carriers launch new services, which the former Bell Atlantic did when it merged with three wireless carriers and launched buckets of coast-to-coast wireless long distance, sans roaming charges. It subsequently launched wireless IP over the handset with such features as online trading sites, local, national and international news, weather and personal organizing sites. Verizon Wireless declared itself a single wireless network entity even before the ink was dry on its merger with AirTouch Cellular and PrimeCo, yet its customers’ bills continue to bear the Bell Atlantic Mobile logo, and at other times, the Verizon logo. But its customers suffered through such network problems as dropped calls, four-day delays for receiving voice mail messages, and other quality of service problems—even before the labor strike began in August. If a large carrier gets its new services up and is, in the least, able to provision and bill for them, it can begin looking for a more flexible system that can adapt quickly to the consumer market.

“The cost for a Tier 3 long-distance company to change its business plan makes it difficult for them to take a fully integrated approach,” Alltel’s Berus says. “That means having to conform your business to the end-to-end.

“Either that, or you have to customize, which adds to integration cost. If you have to customize over 20 percent of your software, you’re starting to look at the possibility of an integrated best-of-breed approach.”

The problems with pedigrees

“Best-of-breed” is one of those industry terms marketing writers and analysts are loathe to let go—after all, this is not a dog show, it's about revenue. But what does it mean? The company that one integration vendor considers a best-of-breed partner may be held at arm's length by other integrators.

“It’s who has the biggest market share,” Schmidt says. “It entails the normal things when you look at product selection. How well will this product meet business needs? How flexible is it? Ten years ago people were looking for off-the-shelf products and asking, how well will this packaged [software] solve our business needs? Now the additional question is, how well will this thing integrate with my other systems? How well-tested is it?’

“One of the common problems of the best-of-breed approach is managing the product catalog,” he says. “There’s a lot of manual work to synchronize the product catalogue when it’s spread across multiple systems. After months and years, you have multiple vendors and multiple application systems that are supporting different aspects of your portfolio. Broadband services are handled by broadband systems, IP services handled by IP systems—each has a piece of the product catalog. And each has some overlap, since your product catalog is implemented over several platforms. Such things as bundling and cross-product discounting become very complex. That’s where you need an integration strategy and architecture, to come up with a consolidated view so that you can create a single product catalogue.”

ILECs and the consolidated view

ILECs might find such consolidation difficult, says Marino, who has had experience with the old dinosaurs—which he characterizes as billing and customer care systems built before 1995. (He began his career 30 years ago in wireline, and has been in wireless billing since 1982.) Marino says RBOCs and large LECs rarely have the ability to reinvent their legacy systems with internal resources. “Let’s say you have an incumbent system written in monolithic code on a 20- to 25-year-old mainframe,” he says. “It’s very difficult to isolate subsystems and separate it into components, because the code is inter-wound into different subsystems. You can’t tell where one subsystem begins and another ends. There may not be a clear delineation between service order entry, provisioning, billing, rating, and accounts receivable, for instance, and therefore it is very difficult to accept single or multiple components. It’s very difficult to lay a component in.”

Mongrel interoperability standards

Put a pack of “best-of-breed” software components together, and because interoperability standards vary—or have been ignored by the vendor—the telco can end up with a string of software components that have a difficult time communicating. “It isn’t really that cut-and-dried when you try to put a bunch of solutions together from several companies, even though we are supposed to have interoperability standards,” says Allsman at Frost & Sullivan. “Even though a standard is set forth, every company takes its own angle on how the standard should be implemented—the data fields may be different, that type of thing. The best thing for the service provider to do is find a vendor with as many pieces of the puzzle as possible, and then pick solutions from reputable companies that fill in the other parts.”

Berus agrees: “Everyone takes liberty with standards for introducing new services. Standards bodies move slowly, software companies move quickly. So if a customer is in a hurry to get a service to market, and no standard exists, the software company writes one of its own. Billing and OSS developers have to look carefully at the software to make sure that what they’re buying is not too far off from the standard, to make sure it’s compliant. But everyone is going to claim it’s ‘compliant.’ ”

The magic buses

What the industry developed is a simpler, standard, easy-to-install “component bus” to greatly reduce the need to write custom interoperability code, Berus says. The component bus architectures are defined by such industry standard groups as the Object Management Group (www.omg.org) for the CORBA Component Model that is defined in Version 3.0 of the CORBA specification.

In the case of the Java 2 Enterprise Edition (J2EE), the standard is managed by a consortium led by Sun Microsystems Inc. The middleware that implements these component bus standards are generally developed and marketed by the same vendors that have been actively marketing CORBA implementations for the past 10 years. The vendors of these application server products include BEA Systems’ WebLogic, IBM’s WebSphere and iPlanet Application Server, ONA’s iPortal Server product suite and Borland/Inprise—Inprise Application Server 4, among others.

“The beauty of it all is that they all claim compliance with one or usually both of these industry component bus standards,” Berus says. “Theoretically, if we write an Enterprise Java Bean to implement an enterprise rating engine, then it should run on any of these vendor implementations.”

This technique accomplishes several objectives, Berus says. · It keeps IT developers from assuming the role of middleware developer, leaving that work to the vendors focused on making a living at it. · It prevents a company from getting locked into a single vendor; thus maintaining flexibility. · The products are inherently Web-enabled.

· The Toaster Effect

Say a homeowner goes to buy a toaster; he looks at its color, whether it holds four slices of bread, and so on, and buys the toaster he likes best. “What we don’t do,” Berus says, “is take the toaster out of the box, examine the plug to see if it matches the outlets on the wall, because we take it for granted that electrical engineers have worked out those standards for plugs.

“In the software industry, we don’t have that luxury. The [software developers] didn’t talk to each other; they were constantly cutting the plugs off our appliances and making new ones that fit other receptacles.”

According to Berus, the lack of a universal interface code pumps up Alltel’s integration costs. “We are always creating a point-to-point communications channel between each of our systems, one at a time. We end up with literally hundreds of communications bridges, each separately built, designed and maintained, and that drives our operational costs way up.”

This manual integration occurs even when mediation, billing and OSS vendors pool their offerings and sell telcos on the one-stop-shopping concept. The TMN model, designed to provide detail on each component in the system and its functionality, also outlines how the components are meant to talk to each other. But here, too, not all software vendors move within the TMN guidelines. “We don’t have enough time and money to develop the functional components suggested by the TMN model,” Berus says. “That would take us 30 years.”

So, what’s the answer? “The component bus now is defining for everyone how the software talks to each other, so we don’t have to worry about it anymore,” he says. “We implement it at the infrastructure layer. We don’t have to worry about all that plumbing work anymore and can now concentrate on business rules. This starts to ease the integration problems. It makes the best-of-breed practice attractive again for large telcos.”

Case study: Essential.com

Essential.com, which sells local, long-distance, wireless and Internet services through an “online supplier marketplace,” is not burdened by legacy systems; and its billing and OSS plans are based on XML interoperability. But it must track, bill and manage customers who order services over its Web site from an aggregation of suppliers. The company, based in Burlington, Mass., acts as a reseller and tries to find the best price on telecom services for customers who visit its site.

The company is on what is known as a “migration pattern,” meaning it began with a low-end system in-house (CostGuard, from Info Directions) for rating, billing and customer care. It plans to install more sophisticated software as it obtains customers, revenue and attract investors.

“They start off with the low-end, SQL server system, and as they grow, they begin to want scalability by moving to an enterprise system with N-tier capability,” Nowak says. The system, which utilizes the Internet Explorer browser, includes a thin, Web-enabled XML client to keep customer information on the site. The low-end system and the customer database remain in-house, and as Essential.com grows, more pieces are fitted into that framework.

“We just install the new enterprise software and the minimum software requirements to run the Web [OnlineRep]; the user at the client downloads the data from the customer database using XML to the client’s GUI. About half the trouble ticketing and collections management are integrated into the system. Whenever Essential.com wants another piece put into its system, the vendor installs the components.

Installation isn’t always easy, Nowak says. “If I’m going with an OSS package, you have to build connectors between the two systems. If the system has a connector already built in, it can obtain records—line information, 800, equal access lines, recurring charges, non-recurring charges, discount information about each customer. Because the OSS sends that into the billing system, the system must have an API that accepts those records in real time. Info Directions doesn’t buy off-the-shelf middleware, but instead relies on the integrator to write the code to link the pieces, Nowak says.

“The integration between the OSS and the original system can take 3 months, but 6–7 months is not uncommon,” she says. “If the project manager at the telco is not 100 percent committed, the project will take longer than 6–7 months. They have to have someone who can order the appropriate hardware, write test plans and execute testing.”

The integration and installation continues as the telco’s original billing system stays live. The billing and OSS vendor, with the help of the other project managers, has to replicate the production database, including mirroring the customer accounts. They create test scenarios that validate the ability of the billing and OSS to handle the planned services. In simulations, Essential.com and Info Directions can use Nortel and Lucent test facilities that include live switches.

“The OSS would have to feed into the billing system the new service in the form of a line simulation,” Nowak says. “The OSS would set up all the attributes of the new services and send the transactions real-time into the billing system. You have to get the APIs right at this point. There are one-to-one connectors from the OSS database to the billing database. The first thing you do when you try to link a system that needs to talk to the billing database is to map the data to the database, find the touch points. For instance, you have to find the customer name stored in one system and determine the length of characters used, etc. There has to be a field-for-field match between the OSS and the billing system and the data elements that are required for billing.”

Before mapping, planning

Before mapping data between systems begins, all parties involved in the installation and integration must meet to plan the mapping, Nowak says. Using data dictionaries, the subject matter experts from each team sit in a room and “literally map each field, one by one,” she says. “Data dictionaries can be hundreds of pages long, but you don’t need all the information listed in them.”

Players in the Essential.com integration included Danet Inc., BusinessEdge Solutions and Essential.com’s project manager. The three parties can work out data mapping over three or four days. Essential.com has 50,000 to 70,000 accounts, which puts it in the lower Tier 2 category. But the greater the variety of services that will be launched, the more intricate the mapping becomes. Surprisingly, it has little to do with the size of the customer base, Nowak says.

After the mapping, CommTech plans to build the connector that lets the CostGuard API talk to the telco’s OSS API. The simulated order goes into the OSS and it feeds the data through the connector, through the API and to the billing system, Nowak says. “If it doesn’t go through, the billing system will indicate ‘bad record,’ then the OSS will know how to respond accordingly,” Nowak says. “There has to be 100 percent accountability for every transaction. There has to be traceability, so you know where every record is at every point.”

Hot-swapped and ready to go?

The system is then hot-swapped and put into service, and—hopefully—works like a symphony with the original billing system. Program managers cross their fingers at this point. “They’re going to monitor every transaction that goes across to make sure the OSS is sending the transaction to the billing system, and that mediation is polling the switches and sending the call records to the billing system,” Nowak says. “The results have to be 100 percent correct.”