Large Systems Integration in a Multivendor Environment

The prepackaged solutions available in today’s billing and OSS market can achieve what would have been thought miraculous only several years ago. As recently as the early ’90s, many carriers with complex pricing, billing or product requirements still had to develop many of the applications that today can be plucked right off the shelf.

With this progress, however, comes a new set of management challenges. Vendor software usually simplifies the management environment to some degree. But the ever-increasing scope vendor systems provide creates a new world of risk. The involvement of so many vendors and internal organizations, each with its own culture, contract and areas of specialization, creates uncertainty when executing large projects.

Managers who assume that running a large project is similar to running smaller ones, but on a larger scale, are mistaken. Such thinking leads to failure, and many organizations learn that lesson the hard way.

Managers can't think small

Small projects can often be managed through linear thinking: “If the following three things are done, the goal will be reached.” Large projects, however, create traps—optical illusions that lead one to believe things are easier than they appear. Large, complex projects require a more sophisticated train of thought: “If the following 300 things are done, a number of unpredicted things will result, and we will reach the goal only if we have accurately identified and managed big risk areas.”

"The challenge of installing a new system is large in itself, but the real complexity comes from continuing to support a business that is growing rapidly while also bringing up a new system," says Jim West, CIO of U.S. Cellular. "The resources who are critical to the new system are also those critical to staying competitive in markets supported by the previous system."

U.S. Cellular recently completed a migration to a new platform that supports 1.6 million subscribers.

Anatomy of a large project

Defining a project as “large” depends less on the number of people involved than on the complexity and the expertise it requires. Large, complex projects consisting of interacting moving parts gives relevance to the term “chaos theory.” Such projects usually have more than 20 people involved and require creativity, not just repetitive tasks.

Large projects may involve multiple sites, multiple organizations, and a large number of external stakeholder organizations. The more complex a project, the greater the number of risks. Opportunities for mistakes and miscommunication can be best avoided by thorough planning.

Lessons in the hypothetical

Let’s take the following hypothetical project to look for lessons and special areas of caution. This mirrors similar large projects major carriers performed in the past few years.

The wireless subsidiary of a large multinational carrier is installing a new billing system. The project's goals include:

? Order entry for call centers, retail stores and Web access
? Retail store inventory control
? Network mediation for switch provisioning
? Network mediation for usage data collection
? The system will be deployed in existing call centers, but a new outbound dialer will be installed to support increased telemarketing volumes.
? The system will be operated internally, except for handset and bill print fulfillment, which will be outsourced.
? Collections will be performed by an existing collections organization within the parent company and is currently doing collections for long-distance products. Collections for the current wireless system are already outsourced.
? An existing internally developed application, interfaced to the new billing system, will conduct credit checks.
? A contractor will perform conversion from the existing system, but the contractor has not yet been selected.
? Key interfaces such GL feeds, central inventory and reordering, bill print, and other functions will be custom-coded by a combination of internal resources, billing vendor staff and third-party contractors.
? A systems integrator has been retained to manage the software installation and development of revised processes. The integrator will also be responsible for the procurement of all hardware, system software and networking infrastructure, which involves 12 vendors and 38 products.
? The hardware vendor will install the hardware, and the carrier’s IT organization will configure the space in which the hardware will be installed.
? The operating system will be new to the carrier’s IT organization, but the database will not.
? The billing systems vendor will train the users.
? The IT organization will manage the project, with the exception of a data warehouse under development.
? The marketing organization will implement the data warehouse, which will be built by a vendor.
? The project is sponsored by the COO and will be funded from the operating budgets of the markets. The IT organization developed the business case with the support of a market research consulting firm.
? The system is intended to serve all customers from low-end consumers to complex business customers.

In this example there are at least five major internal organizations, one internal organization under outside management (collections), three major consulting suppliers (one of which is not yet selected), numerous systems software suppliers, several hardware suppliers, and several programming contractors.

Project Strategy and Goals

The physics of systems management dictates that of the three general goals of good, fast and cheap, only one can be optimized at any given time, and only two can be substantially achieved. Getting the best combination of all three is the essence of good management. A project team that believes it will simultaneously optimize good, fast and cheap will get a nasty surprise. The project will almost certainly turn out to be worse than what could have been achieved with a realistic understanding of the trade-offs.

A multi-vendor environment, complicates the project in several ways. Differences between what project plans and what each vendor is responsible for can be significant. These differences must be built into vendor contract negotiations. Flexibility to adjust later in the project is often limited. For example, the overall project may be optimized for time-to-market reasons. For the conversion sub-project, however, the highest priority might well be quality, with the overall goal achieved by starting conversion very early in the project. Because, in this case, the conversion contractor has not been chosen, this is an early indication of a risk. But that risk would not be evident if you assumed that conversion could start sooner than needed.

Cross-vendor coordination further complicates things. If the data models for the billing system and the internal credit check system don't match, then the changes required to align these models must be defined, including deciding who will implement and pay for the changes.

Schedule Planning a balancing act

Multiple organizations can upset the balancing act required to establish aggressive but achievable time lines. Many large projects falter here, when unachievable goals create a “best we can do” philosophy. The best that can be done is often spectacularly good, but this mindset creates the habit of not accurately budgeting for deadlines and can lead to bad habits if shortcuts persist.

To guard against these pitfalls, use these management techniques:

? Create realistic deadlines. Use an objective third party if you are hearing only what you want to hear. Determine whether adequate contingencies are included. What if someone is sick or leaves the project? What will the ripple effect be elsewhere? Do backups exist for important resources?

? Plan sufficient time into the schedule for a careful review of deliverables such as requirement definitions and designs, where the definition of success is somewhat subjective.

? Keep in mind that human nature is to please by doing more than what is reasonable, and that workers will assume productivity levels based on the levels defined in the schedule. Talented and experienced people create timelines based on how long it would take them to do the work.

? Even if there is a contractual commitment from a vendor or team to deliver something, remember that a contract does not make the impossible suddenly become possible. Be sure to have adequate checkpoints at areas of significant risk, because contracts rarely protect one against the costs of late delivery or inadequate functionality.

? Create an environment where bad news and legitimate, well-reasoned concern are permitted. If your team is discouraged from bringing up problems, you’ll still get the bad news in the form of a bad outcome. And you won't have time to correct it. Scenario planning is essential for dealing with the unexpected paths a project might take.

The Crystal Ball: Budgeting

Budgeting is a way of seeing into the future. Yet while it is an imperfect process, budgeting is especially important to maintain discipline necessary to determine risk points, understanding what contingency is necessary, and providing feedback for the work plan and schedule.

Budget in perspective is to calculate the cost of a project day. This is money spent on a daily basis, but does not include costs that do not vary by day, such as system software, which has a fixed cost. To get the schedule back on budget, make up the daily cost later in the project.

Tips for writing the right budget:

? Identify choke points where contention for resources may slow progress. For example, if contracts with four principal vendors are set to end on the same day, will there be a delay because of the time required for legal review? Do you have a commitment from the vendors to begin work while the review is underway? If not, the resulting delay must be factored into the schedule.

? Factor in the experience of the various teams. What is being done for the first time, and what is being done by teams that have done the same thing before in a similar environment? Tasks being done for the first time should always get an additional time allocation of at least 30 percent.

? Do you still need to hire staff? Have you assumed a realistic time line? What if these skill sets cannot be found? What will the backup plan cost?

? Factor in the time and expense of cross-training for pivotal skill sets, which will seem an unaffordable luxury during an aggressive project. Having a good knowledge base in only a few resources usually leads to even greater costs.

? Even if the overall budget is dictated in a top-down manner --"We have $XX million to do this project”--one must still write a detailed, bottom-up budget. There is often resistance to budgeting at a low level, because project teams realize that their ability to see into the future is limited, and they fear being held to projections made with imperfect data. While some of this reluctance is justified, the attempt to do a detailed bottom-up budget is still vital, as it shows major risk points where scope may need to be adjusted.

? Factor in time for main review points, such as approval of business requirements, physical design and the system's test plan. Related expenses such as travel between project sites can easily run into millions of dollars on a large project.

? Factor in a percentage for inefficiencies, such as disconnects in work plan dependencies, contention for important resources, and an inability to schedule meetings to match team travel schedules between different sites.

? Adjust the budget throughout the project. Will functionality be added or dropped? Will the business goals be changed? Can the schedule be changed? If these questions aren't resolved during the initial budgeting process, risk points won't be identified until it’s too late to add to the contingency budget.

Avoiding a Project Riot

Overturned cars are unlikely. But the looting, destruction and generally uncivilized behavior characteristics of a riot are never far away if an uncoordinated free-for-all occurs in acomplex environment.

One of the absolute rules on any project is that everyone must feel accountable for the overall success of the project, not just an individual part. The silo philosophy, which says “I will deliver my area on time even if I am unable devote time to ensuring that my area is properly coordinated with related areas” is a sure sign that your project is in jeopardy. The overall manager must identify points where this could occur and ensure proper coordination.

One place to look for uncoordinated cooperation pertains to the usage collection network mediation vendor and the billing vendor. This area would require coordination under any scenario, because similar edits and control totals can generally be produced from both the network mediation function and the billing message processing system. Our sample scenario includes four different organizations: the systems integrator, the mediation vendor, the billing vendor and the carrier’s network engineering group. In such situations, it is the integrator’s responsibility to ensure that the edit and control requirements are properly defined and placement across the systems is correct.

The project governance structure should include representation from organizations participating as stakeholders or member of the implementation team. This should balance the lines of communication and minimize the degree of counterproductive influence. This includes representation from each major vendor, the parent organization’s collection department, each internal organization and the project sponsor.

Flexibility cannot be assumed

Don't assume the same flexibility that exists in an inter-organizational project will exist when several vendors are involved. For example, on a project using 50 full-time people, if one area falls behind, one can move people to that area to balance progress. This is rarely the case if 40 of the 50 people are from four separate vendors. Even if skill sets could be moved around, the organizational boundaries make this highly unlikely.

Advances in the technical architecture have dramatically increased functional and technical flexibility in billing systems. Witness the highly configurable nature of the leading systems and the increasing use of advanced technical standards (e.g., CORBA) that lead to greater flexibility. The options this creates must still be managed closely, or the team might define or redefine requirements late into the project development cycle.

Standards and Communication

Organizational design must also foster rapid communication. "An entrepreneurial organization which fosters rapid decision making and close communication is essential for the success of a large project," says Rian Wren, president of Comcast Telecommunications.

Improving communication is often overlooked in the belief that communication is intuitive and need not be taught. There are quick and easy ways to maximize efficient communication.

? Identify the elements of methodology that must be shared across teams, and create standard definitions.
? Look for areas where technical standardization must exist. One must have a common definition of the data model across teams and software. The use of standards like CORBA or a middleware communication layer can further standardize technical communication between systems. Edit logic must be synchronized; the message processing layer cannot edit for one thing while the billing layer expects another.
? Create a communication matrix. That defines what information will be communicated by a given organization in a given forum.
? A plan describing standard meetings is vital. Veterans of large projects are familiar with the phenomenon of uncontrolled meeting proliferation, such as holding three meetings when one will do. Decisions are delayed because necessary participants do not have time to meet until, that is, the deadline for decision has passed and productivity is lost.
? Determine how decisions will be communicated. If they are not communicated quickly to the right parties, decisions become diluted.Then there's the dreaded change control process. Change control is often steeped in bureaucracy, however, but a measured dose of bureaucracy prevents chaos. Undocumented “hallway” decisions wreak more havoc and lead to even more wasted effort.
? Create a work plan task to confirm specific hand-off points between major processes and organizations. For example, does your billing vendor know what your call data collection/network mediation vendor thinks? This is another reason to select product sets with pre-integrated hand-offs where possible, such as between network mediation and billing/message processing.
? Avoid vague contracts or deliverable descriptions that neither clearly specify scope nor provide flexibility to support changes in the schedule or budget.
The steps outlined above can be a rapidly implemented , especially when the answers to these issues are well known.

Crucial questions in our project

The points above, applied to our sample billing system project, raise questions worth investigating:

? Can the billing vendor provide user training on the entire scope of the system? Are they really in that business, or would a specialist vendor be more appropriate?

? Are the roles related to hardware procurement and installation clearly distinguished between the systems integrator, the hardware vendor and the internal IT organization?

? Who will be responsible for the transition of collections authority from an external vendor to an internal--but still separately managed--vendor?

? What will govern the interaction between the data warehouse vendor and the billing vendor? Have provisions been made for the data warehouse vendor to access the billing system data model, even though it may be proprietary?

? If the business case was generated by the IT organization, does it adequately reflect benefits to the revenue side?

? How will differing priorities regarding low-end customers and high-end business customers be resolved?

There will not always be clear answers. However, the key to creating a manageable degree of risk is to answer as many of these as possible before the momentum of the project creates loss of time and money.

Large projects becoming more common

Billing, OSS and related systems have recently expanded in size and scope. Just a few years ago what was often implemented as separate and narrow projects are now massive efforts. It is tempting to believe that projects can be easily completed. But all the soft issues--such as budget and communication-- cannot be easily solved. Ask yourself, “How were the pyramids built?” It was a simple matter of organization and management, as much as it was a matter of engineering.

Vince Shaw is a director in Deloitte Consulting's Communications and Media practice. He specializes in systems integration and process improvement for telecommunications providers.

Sidebar:

Tips for managers: Don’t do all the work

First, managers must truly manage. Managers define the process by which others accomplish work, rather than doing the work themselves. Large projects require managing through empowerment by letting individuals do it their way, though it may not be the absolute best way. To do otherwise creates an over-controlling environment that stifles decentralized decision-making. Many managers find it difficult to give up some control especially when a vendor is involved.

Second, managers must avoid the mindset that if things start to go wrong, they can immediately jump in and fix the problems. The project will careen out of control if intervention is needed in several areas at one time. The loss of time and money can quickly overwhelm managers trying to put the project back on the right path.

Risk of chaos increases when numerous organizations are involved in the project. Therefore, smart, detailed planning early in the project is vital. Such planning takes discipline, because detailed planning takes time. Managers may get heat when the planning appears to delay progress, just as pressure for rapid progress escalates. "Clear scope and priorities must be established early in the project as part of the requirements definition process," says AT&T's Surendra Saboo, director of the Western Local Platform. "If scope is not managed or priorities are established late in the project, then schedules and focus will be unclear."

Third, it is crucial to maintain discipline to negotiate contracts, establish an integrated work plan to tie the individual work plans together and build in contingency and intermediate milestones for risk areas.

Contracts are too often treated as legal formalities or adversarial weapons, when in fact they are an important part of the process of planning and establishing clear communications, expectations and key assumptions. Time spent clarifying these contract points is time well-spent. It is a profitable investment that will be repaid in time and money cost savings. Coordination with internal vendors requires diligence equal to that required with external vendors, to ensure good communication and adequate budgeting.