Data Reconciliation: Service Provider Salvation, or Just a Back-Office Bandage?

For most service providers, the back office of OSS and billing—despite the best efforts of CIOs—is a Gordian knot of mismatched data and disconnected processes fostered by standalone applications. With Wall Street pressure to improve profits, the scramble is on to find ways to clean up the mess and reduce operational expenses. Recognizing a market opportunity, several OSS players have recently launched products and expanded offerings in an under-appreciated area called data reconciliation. They promise “found money” by recovering lost revenues and stranded network assets. Is this the next great wave in service provider productivity, or just OSS snake oil?

Just What Is Data Reconciliation?

As the name suggests, data reconciliation involves comparing related data elements across two or more sources, whether via manual or mechanized means. For example, a service provider’s billing system may store customer account and offer subscription data. This should correspond to “network facing” data in the provisioning system, such as circuit IDs, number assignments and start dates. This in turn would link to network inventory data such as switch IDs and port numbers. Information that would link customer data across one or more systems would likely be a customer account number, circuit ID or telephone number. Simple, right? Wrong.

Unless one has a completely mechanized back office (and who does?), the customer account, order, provisioning and network data elements are usually established in multiple systems by human beings in a process dubbed “swivel chair automation.” Service orders entered into the order management application may require manual re-entry in a provisioning system with the proper assignment of elements in inventory. And all too frequently, in the quest to meet aggressive delivery schedules, process corners are cut in favor of service turn-up, to say nothing of simple input errors. These flaws may leave data needs unmet for other processes, including billing. Once again, manual entry to clean up the discrepancies is needed, raising the cost to deliver service, with no guarantees of accuracy. Now network assets are committed to a customer contract, the customer is receiving service, but the carrier isn’t receiving any revenue from these efforts.

In a data reconciliation effort, the linking or overlapping data is joined to compare records across two or more databases to ensure that the services provided are indeed the services billed. “The need for reconciliation is huge,” says Karl Whitelock, OSS program director for Stratecast Partners. Unfortunately, most carrier reconciliation is spotty, if done at all, he says, adding that service providers typically view such efforts as expensive and time-consuming, in spite of a promised payback.

Once Upon a Time…

Historically, service providers have sought to keep network costs under control by auditing invoices from wholesale transport providers, comparing the billed amounts and circuit identifiers from the wholesaler to internal inventory and provisioning records, a process known as intercarrier billing reconciliation. This was especially important for resale service providers, for whom transport represents the majority of operating expense. Firms such as Vibrant Solutions and TEOCO have made their names in the communications industry by providing applications and service bureau capabilities for this purpose.

At the same time, intracarrier reconciliation across multiple systems, if done at all, was little more than a loose collection of homegrown kluges that compared large portions of production data stores, according to Mark Unak, chief architect at Fujitsu DMR. Output reports would direct the staff to go on the hunt for the root cause of anomalies. Not only were these applications unwieldy, but they also typically functioned outside the main production job stream, and as such received little executive attention or funding for maintenance and improvement.

Seeing an opportunity to leverage its intercarrier expertise against the internal conundrum, Vibrant acquired startup Longitude Systems last November. With the Longitude product it will be able to approach existing customers and offer to attack the internal systems reconciliation problem, on the premise that what works for intercarrier needs is also true within a service provider’s own shop.

Other niche players such as ConnexnTechnologies have also emerged to help service providers scrub internal system relationships. Among other things, it offers applications that audit service and feature provisioning records against retail billing records, pinpointing underbilled or unbilled services.

How Big Is Big?

Are these pervasive problems? Experts disagree on the precise amounts, but a few key metrics drive the point home. Looking internally, CEO Charles “Chuck” Crenshaw says that Connexn’s experience comparing provisioning and billing files at more than 30 providers has revealed error rates from 10 percent to 80 percent, with an average of 30 percent. Be it inter- or intracarrier reconciliation, clearly there is room for improvement.

In its 2002 Access and Network Cost Management Report, Pricewaterhouse- Coopers evaluated the full scope of the domain, including external relationships such as intercarrier costs and dispute management, as well as internal decisions such as network optimization, routing analysis and telecom cost planning. A full 40 percent of service providers surveyed either had no staff directly responsible for cost management (15 percent) or a team of three or fewer people (25 percent). Ironically, the results for those with a formal cost management function are highly satisfactory: 20 percent saved over $5 million last year, and another 41 percent saved up to $5 million in the same period. One carrier cashed in over $50 million in savings—directly to their bottom line.

Is all of this attributable to data discrepancies? “Probably not,” says Gary Ross, vice president of product management for Connexn, “but it impacts revenues, and there’s also a huge operations cost when you look at the time that’s wasted by technicians who are trying to perform their job based on inaccurate or invalid data.” Stratecast’s Whitelock comments, “One [service provider] said that its circuit provisioning systems were, at best, only 50 percent accurate, pretty normal for the industry right now.” Given those odds, it’s a safe bet that field technicians regularly perform tasks based on faulty work orders.

There are so many areas of a service provider’s back office requiring reconciliation that savings from one project’s success will likely bankroll additional initiatives. “But the first thing [service providers] have to do is own up to the fact that they have a problem,” says Verne Anton, principal analyst with Gartner Group.

How Did We Get Here?

If it is true, as many in the communications business say, that data drives revenue, then why are service provider databases so woefully out of synch? Systems integrator Joe Kutay of Plano, Texas-based TNNT points to the 100-plus year history of telephony as part of the problem. “You’ve had all these files being worked on by so many people for so many years,” he says, “and then you have some poor service order administrator with maybe two years of experience being asked to untangle them. It’s not realistic.”

Whitelock goes further, saying that in his experiences as a system planner with an ILEC prior to the Telecom Act of 1996, carriers “would allocate an amount equal to 10 percent of network capital expense for the funding of operational capabilities.” Over the last five years, however, the push to grow the market left operations in last place for funding priority, resulting in weak, catch-as-catch-can processes operating on mismatched support systems grown year over year in independent “silos.” Layer on top of that the current press for operational improvements, and so begins the downward spiral into bankruptcy.

This is more than just the telecom version of an urban legend—it’s fact. Washington, D.C.-based CLEC NetTel, bankrupted into oblivion in the fourth quarter of 2000, was reported to have significant “billing problems,” a term that experienced OSS practitioners recognize as a symptom of disconnected systems and processes all the way back to the original order.

Revenue Recovery Enters the Third Generation

As service providers look for more opportunities to recover revenues from internal data reconciliation efforts, software vendors are responding with more powerful tools.

Vibrant Solutions proposes to move beyond data file comparison routines as well as “table-driven” interpreters. According to Joel Halpern, consulting architect to Vibrant Solutions, the aim is to create an object model that ultimately represents a normalized view of the data shared between the involved systems.

For example, he says, “You pick a small number of systems which have relevant data with a significant overlap and you start from there. You build a part of the system as a reconciliation engine that has components for bringing information in from the OSS. You build the model, then build the data map scripts that bring in the information [from the source systems].

“Then you populate the objects and it does the reconciliation; it compares the information on an ongoing basis and says, ‘Wait a minute. We found an inconsistency.’ So you get the discrepancies highlighted.” This process can take anywhere from two days to two weeks, depending on the complexity of the problem.

But what happens when the data relationships change? Halpern says this is where this new generation of version control capabilities come into play. These protect the user from having to manually reestablish the entire model every time the service provider changes a business rule. “We built the whole system around the notion that the data models themselves are subject to change continuously,” Halpern says. “Our modeling tool is designed to understand these changes, represent the relationship between old and new models—not just old and new state of object—and then apply those changes to the running system.” This is especially powerful when the service provider wishes to add applications to the shared model to perform ever more sophisticated analyses.

Another newcomer is Waltham, Mass.-based Lavastorm. Once a large-scale systems engineering and development shop, Lavastorm now dedicates itself to revenue assurance products for the telecommunications marketplace. Much like Vibrant’s offering, the Lavastorm product mechanizes the interfaces from a multitude of data sources using extendable parsers to interpret the source, updating an Oracle database along the way.

Rather than a platform approach such as Vibrant’s, Lavastorm is looking more at specific applications, aiming squarely at classic problems such as billing-inventory-provisioning mismatches, and using traffic analysis modules that identify rating inconsistencies and “transit” traffic for which the carrier receives no compensation.

According to Tom Nolting, vice president of product management, and Mark Garvey, vice president of product development, the Lavastorm product is now in beta testing at three different service providers: an ILEC, a facilities-based CLEC and a UNE-P CLEC. The product is scheduled for production shipment in late spring or early summer this year.

One thing is certain with these two companies: the VC community is impressed. If financing in today’s constrained capital markets is a vote of confidence, Vibrant and Lavastorm are winning the election. Since its renaming less than one year ago from the original Information View and Telecon LLC companies, Vibrant has received a $26 million boost from investors Bessemer Venture Partners and Columbia Capital. Founded last summer, Lavastorm, originally known as JLM Technologies, has seen its efforts underwritten by the equally well known Oak Investment Partners and Hummer Winblad Venture Partners.

Connexn Technologies is also evolving its cache of revenue recovery tools. It recently developed a Java-based component that works with the existing applications. John Kaplan, Connexn’s vice president of software engineering, claims that the beta version of its new component will reconcile more than 2 million records in less than 10 minutes, significant power that will enable Connexn to satisfy the productivity demands of its larger, more recently acquired clients.

Wanted: The Right Stuff

Both Vibrant and Lavastorm assert that their products can yield an in-year return on investment (ROI), with some efforts paying off in as little as six months. For an expenditure that generally runs in the low seven figures, according to Vibrant’s Halpern, these claims indicate a dramatic opportunity for revenue recovery.

Will service provider CIOs see the new data reconciliation tools as panacea or peril? Initial trials appear to be bearing financial and operational fruit. MCI WorldCom’s residential service billing team saved $1 million in just six months with one business rules-driven audit system, according to Mark Madigan, president of systems integrator IT Cadre, formerly director of residential billing and product reference at MCI.

Smaller projects are also picking up steam. Halpern says that at one carrier, a two-system reconciliation of only 1,000 records yielded $60,000 in stranded revenues from records that were designated as “clean” by the carrier’s operations team. Connexn’s Ross comments that in his previous position as director of revenue assurance for Qwest’s U S West group, a 100,000-circuit line audit took 24 hours. With today’s tools that process takes only 2 minutes.

As exciting as these developments may be, one caveat is in order: the choices available do not always clean up the source data, leaving the unsavory task of continuous maintenance of the reconciliation process and files to the service provider’s operations team. In fact, in the cases of Vibrant and Lavastorm, there is now an additional database to manage, however accurate or useful it may be.

Industry pundits disagree on the value of the “third data store” approach, arguing that it adds an unwelcome burden to the service provider’s already stressed IT environment. Stratecast’s Whitelock says, “Having another data store somewhere is not the answer.” He points to Connexn’s optional correction module that assists the user in auto-correcting as a better approach, albeit an intrusive and potentially dangerous one.

Vibrant and Lavastorm executives are quick to counter with the benefits of a nonintrusive application that allows the users to recover the stranded revenues first and focus on process and system cleanup later. One has to wonder if this posture is born of the fact that auto-correction capabilities are still on their drawing boards.

In defending its approach, Halpern points to the importance of a long-term view. He says a scrubbed, normalized view of the business is a crucial steppingstone toward healthier operations. He offers that this third data store can ultimately be used as the provider’s database of record, running significant portions of its daily operations.

If that is so, won’t the performance of this new database eventually suffer, even for medium-sized service providers? Whitelock thinks so.. “No matter how big you scale it initially,” he says, “there’s going to come a time where you’re going to exceed your engineering definition. So then what do you do?” And while he applauds the theory of a single object model, Fujitsu DMR’s Unak reluctantly agrees that the Vibrant product has some maturing to do saying, “We’ve struggled with some of the performance characteristics of the product. Small volumes of data still take what we would call a lengthy period of time for the reconciliation to occur. The guys from Vibrant recognize the situation. They’re working on really trying to get some large volumes through the product now.”

Whatever the merit of the technical approach, Vibrant CEO Rich LaPerch keeps the debate focused on the service provider’s biggest concerns these days: cost control and revenue recovery. “People always look at ‘as built’ networks. Quite frankly, I think the most important thing is the ‘as billed’ network, because that affects your bottom line.”