French Operator Builds OSS/BSS for the Multiplay

Service-oriented architecture (SOA) has become something of a de facto standard for service providers to integrate complex systems and processes in a loosely coupled, and therefore highly flexible, fashion. But SOA isn’t the only integration technology in town. Many carriers have invested heavily in enterprise application integration (EAI) technologies and need to continue to leverage them to achieve their operations goals. Completel, a wholesale broadband provider and fiber network operator in France, provides a prime example of a carrier that made a big bet on EAI technology, but applied it using concepts that are fundamental to SOA. The resulting, loosely coupled, repeatable architecture has allowed Completel to get into the triple-play game after signing a unique wholesale arrangement with a major French retailer. Brian Nolan, vice president of information technology for Completel, spoke with Billing & OSS World Editor Ed Finegold this fall to discuss how his organization delivered a fully automated fulfillment and self-care system for DSL, and then for triple play, in an uncommonly brief period of time.

Billing World: Brian, please give our readers a bit of background on the French telecom market, the role you play in it as a “carrier’s carrier,” and the forces that spurred you to create a new OSS/BSS infrastructure.

Nolan: It’s a very competitive broadband market in France and the existing telecoms have a large installed base. There are also strong alternative providers like Cegetel and Free. We were entering from an unusual perspective. We only work within France and are about 700 people. We were previously focused mainly on business-to-business and wholesale or carrier solutions with a strong background in fiber. We started as a local loop fiber company with a full range of voice and data services (VoIP, e-mail, and various Layer 2 and Layer 3 VPNs, etc.) over our networks. In 2005, we decided to branch out into DSL to complete that fiber network. We already had a strong position with MGE (middle-to-great, or large) customers, those with anywhere from 150 employees to 5,000 or 10,000 employees. In addition, we have a very strong wholesale offering.

The OSS project for us was actually two projects. The first was a fully automated DSL provisioning and deployment. We did this in about six month’s time. Once we went live with that provisioning, we signed a large triple-play deal with a large retail outlet. [These two] products (DSL and triple play) were very new to us.

When we started into these two projects we weren’t focused on selling to residential DSL customers. It’s one of the lowest cost markets for broadband in the world. It’s quite a tough market. We knew that we had to have an automated solution in order to manage our costs. We started with very little experience with DSL and triple play and tried to engage with some consultants that had strong telco knowledge of those solutions. We hired developers who had experience with what we were deploying. But starting from scratch was a big advantage because you have a clean solution and you don’t have to deal with problems like migration or a mixed technical environment. We could apply an up-to-date technical solution.

Billing World: How did Completel’s rapid growth exacerbate challenges it had with its previous OSS/BSS architecture? When and why did it become apparent that a new approach to integrated architecture was necessary?

Nolan: Before we started we did an analysis of the potential solutions. One was to modify existing systems. The second was to go find an integrator that can bring a fully integrated solution forward, like a Capgemini, Accenture or one of the big systems integrators. The third approach is to own the solution ourselves and work with a mix of experts from outside and experts from inside, trust our own judgment, and roll out the solution ourselves. We very quickly realized that modifying our existing systems was going to cost too much. It would not get us to market quickly enough because our systems were more customized for high-value fiber networks, but not necessarily designed for totally automated fulfillment.

We realized we needed to put out a greenfield system that was fully integrated end to end. We needed a Web portal to communicate in the wholesale marketplace and have a strong front end to deal with the wholesale partners. We needed a strong network inventory to track and generate all of the technical parameters of activation and such. We went with Cramer for network inventory. For activation, we chose the Kabira solution because we needed a strong generic tool since the activation system had to talk to about 15 different types of equipment (switches, routers and DSLAMs, etc.). We needed a strong trouble-ticketing system to manage the customer service issues and we went with the full Siebel (now part of Oracle Corp.) CRM solution. And we decided to maintain our billing system, which is Arbor from Comverse. We also needed a strong EAI bus to connect all of it together, so we went with BEA Systems Inc.’s WebLogic suite (which included BEA WebLogic Portal for the portal, BEA WebLogic Integration for the integration and BEA WebLogic Server for the application server).

We asked the vendors to produce a proof of concept based on a fairly simplified specification and we tested each one individually and got the vendors to develop the proofs of concepts for us themselves, rather than having an integrator do it. That was important because, following on from that, we’re starting with little DSL or triple-play knowledge. We didn’t have much knowledge on this new IT technology we’d be using as well, so it was important to have the vendors tied into the project from the start so that we could use their best practices. When you have a close relationship between the systems integrators and the vendors, you can never be sure if the integrator really knows the value of the product and [wonder whether they are] doing custom work instead of using the full potential of the product? We made sure the vendors were tied into the solution, and not just dropping off a license. We also made sure the technology we were rolling out was an SOA type of architecture and included XML interfaces so that the solution would be modular and flexible, especially regarding integration.

Billing World: How was the concept of using an exposure layer conceived? What results is this approach producing that you did or did not expect?

Nolan: We actually did not go about this as an SOA project at all. It was a project to bring DSL and triple-play provisioning to market as quickly as we can. So the object was not at all to build an elegant “SOA IT project.” It was a project where we knew, with the DSL project, we needed a flexible architecture to encompass the follow-on projects, like triple play. We knew we were a small organization and could not afford to have it maintained by outside consultants. We knew we had a strong team and that if we added some new people with the right knowledge and experience, we could trust ourselves more than a large consulting company. We were confident in our own ability. We found that the vendors we were dealing with were of a size that they wanted our solution to work because they valued our reference. Cramer, Siebel, etc., were independent when we engaged them and they wanted to put in the resources to make this work. I was confident we’d get the best of best practices from them. We were on a very leading-edge technology because fully automated DSL provisioning was still relatively new. It ended up being a very good decision, going with the vendors. We did a good job of driving our specifications.

Billing World: Ultimately though, you built an SOA-like architecture using EAI tools. Do you plan to transition that to a true SOA? I’m sure there are many carriers out there asking themselves a similar question right now.

Nolan: We started this as a DSL project. There were multiple steps involved in the fulfillment process, including communicating with France Telecom, and we implemented a zero-touch solution with no human intervention, other than for handling certain complex exceptions. In parallel to that, we rolled out a new business unit for small business customers. We were challenged with having multiple streams for different types of customers. So, we turned it into a DSL factory so we had one code stream that could handle any kind of DSL provisioning, be it wholesale, retail or part of the triple play. Triple play was ADSL provisioning, but for large customers, it was multiple-pair provisioning and all sorts of different parameters (ADSL, SDSL, etc.) and things involved in the different business units.

We went the extra mile and we went down that SOA architecture path without realizing what we were doing as we took the logic from our DSL system and applied it to our triple-play solution using SOA types of concepts and technology. We needed to maintain this SOA-type approach of having loosely coupled connections between the bricks in the provisioning system so we could modify the components for future changes and growth. When we rolled out the triple-play solution, we used the same basic components and we built a service factory with the same portal type interface and then we could decompose the orders and have the service factory deliver the provisioning for voice, data and TV provisioning. We built interfaces that enabled our wholesale partners to have an Internet portal and flow their orders straight through to us.

Billing World: What factors helped you to choose BEA as the integration vendor for this?

Nolan: We chose BEA as part of the proof of concept. They were on the short list with other vendors like SeeBeyond (now part of Sun Microsystems Inc.) and TIBCO Software Inc., as well as some smaller vendors. We also looked at what other operators were using and tried to do some analysis on what the other guys had done, which is not always easy to accomplish. We found, at that time, that many of the workflow tools were man-to-machine, but we realized we needed a machine-to-machine type of bus to automate the process. We found that BEA was very stable and easy to work with in the proof of concept. We also found the company was prepared to work with us. To some extent, and it might sound corny, but it was on a handshake with the French managing director of BEA and they were committed to making the solution work. We went through a full analysis, looked at the market share of the various companies. I wanted a company that was focused on the tool itself and that if they failed on that project, it would really hurt them.

Billing World: Self-service is a hot topic in the consumer space today, but it is even more important for business-to-business relationships like Completel has with its customers. What types of demands do your customers place on you for self-service tools and how has your offering in this area evolved while working with BEA?

Nolan: We have very strong information flow back and forward to our wholesale partners and have lots of status and acknowledgements in terms of what stage of the process we’re in for any order. We also give them the ability to upsell or downsell — go from double play to triple play, partially to fully unbundled, that sort of thing. We developed a portal for the end-user customers themselves so they could use the TV screen to check their order and their bills, and our wholesale customers could use that for some upselling and to help teach customers how to use features just by viewing their TV screens. The customer portal software is bundled into the TV middleware software, but it then goes back and integrates, using BEA WebLogic, back into the rest of the EAI-connected systems.

Billing World: This project was accomplished in a very short period of time. What enabled Completel to make so much progress so fast, and why don’t we see these kinds of rapid time frames more often when it comes to major integration and architecture projects?

Nolan: The timeline for the first project was six months start to finish, starting in about October 2005. We did two months of specs, two months developing and two months to roll out. This was from the first of March to mid-October for the full triple-play project with a very large French retailer. No one has ever done this “fully integrated [systems integrator]-to-[systems integrator]” type of model in triple play. We tied their systems into ours to provide completely automated flow-through and did so on a very aggressive time scale. We did it on budget and on time. We did it with a dedicated and motivated team and built basically from experience in the DSL project and were able to reuse a lot of the architecture with confidence. Once you build on an existing project, things become exponentially more complicated. When you start from scratch, there’s nothing to break, but when you’re building on an existing architecture, you have to be much more careful and precise.

We didn’t have huge volume to start with for DSL, so we had some time to learn and to breathe. But on triple play, we saw very high volume in just a few days. Though we benchmarked and tested the system for these high volumes, we thought we’d be okay. But we found that it worked slower than tested, because each piece worked faster than all of them together. There were peaks and valleys and the peaks put a lot of pressure on the complex process. It was tough to anticipate the stress points and we faced a number of re-tuning issues. Each of the vendors was involved in various independent issues and they helped to resolve them. The guys who advised us on the architecture were pulled back in and they helped us solve the problems, in some cases developing new patches, and in others, tuning the system to run more fluidly. It is difficult looking back to know how we could have anticipated that — perhaps with more time — but given the time frame we had, [I] was hot under the collar to get some of this resolved.

The second issue probably was that we did a good job of generating the specs of the system, but a continuing effort needs to include monitoring and specing the solution in the production environment. There’s an enormous amount of different types of errors that come through for various reasons. You have to monitor and track those errors, and some you can replay automatically and fix them, and others are due to IT bugs or external to the system (such as errors from France Telecom or from the various equipment switches, etc.). You need to monitor all of the steps in the process. Developing the monitoring tools was a bigger challenge than we had anticipated. You basically learn as you go.

One thing we did that was fortuitous is that we separated the whole workflow process and we did an uncoupling of OSS and BSS so that we had the flexibility to uncouple the customer interface from the network interface. We had to maintain a fault tolerant and uniform interface to the customer and had to make stringent SLAs; Also, uncoupling the BSS and OSS layers meant we could take the OSS layer down without affecting the BSS layer and customer interface. You have to have the system tuned to catch up rapidly.