Beyond Middleware: Systems Integration for the New Economy
Edward J. Finegold
01/01/2002
The industry perspective on integration has changed. For a while the focus was primarily on enterprise application integration (EAI) as the be-all and end-all of integration technologies. The proverbial cat, however, is out of the bag: EAI's limitations have become evident, but its role also has diminished as corporations have held back from so-called "best-of-breed" implementations. The communications software sector has shifted its focus to large service providers whose needs are practical and focused on automating and integrating specific processes. As a result, alternative ideas and technologies for integration and process automation are receiving greater attention than the now tarnished EAI.
The Value of EAI-Within Limits
EAI became widespread when start-up service providers began their OSS package buying frenzy. For years popular packages were called best of breed, but that phrase has now become a bit of a misnomer. It's not that package vendors don't develop strong systems, but when every competing product is called best of breed, something just doesn't add up. (In reality, true best-of-breed technologies are few. Ethernet is a good example. Everyone uses it, it works extremely well, its success has essentially eliminated any competing technologies in the local area network, and it is being embraced in the metro network, too.) But because no one can decide which of the numerous breeds are really the best, and no one implements any set of applications the same way, there's no truly repeatable way to bring them together.
That's where EAI comes in. Many service providers, especially start-ups, ended up buying applications that couldn't communicate without the help of something like EAI to provide everything from message transport to data translation to mapping, plus some amount of repeatability in the form of connectors or adapters. Had there been agreement on data models-the data structures and nomenclature native to each application-there wouldn't be a need for the complex translation and mapping aspects of integration. But since there is no such agreement, integration is the most difficult and costly aspect of any OSS implementation, regardless of the underlying technology.
"No matter what you pick for your integration technology, you still have to fight the battle over defining your data model," says Mike Bender, chief technology officer of OSS provider Sodalia North America. "A common model has been lacking in the industry, despite all of the various projects that have attempted to set standards or guidelines."
EAI is supposed to ease integration, but without standard data models, system integrators consistently run into problems on such projects. EAI systems generally do a good job of moving messages back and forth across an integration bus, but that leaves out three critical functions: mapping, translation and workflow. "You're not managing anything" with an EAI system, says Bender, "you're just moving data."
EAI provides some mapping and translation functions at a basic level, but in complex environments, integrators often must add a layer of intelligence on top of the EAI bus to really make it work. This is where the heavy lifting in any EAI implementation takes place, because integrators can spend a large percentage of their billable hours performing tasks such as mapping data from the fields in one specific application to those in another, or building mini-workflows or provisioning plans into the middleware to make up for a lack of such functionality in the applications themselves. In other words, application-specific details are coded into what's supposed to be an application-agnostic information bus.
So, when EAI vendors tout benefits such as the repeatability of their application adapters, or how adapters can insulate each application from changes to others, they are telling the truth, but not necessarily the whole truth. When layers of intelligence are added to bolster the underlying EAI product, some of these benefits are diminished. Suddenly, if applications or processes change, the changes must be accounted for in the intelligence layer, which means a lot of custom development and maintenance costs in the long term.
EAI packages don't come cheap, but the package cost is nothing in comparison to the multimillion-dollar fees involved and the inherent risk of failure that has more to do with people than technology. In fact, maintenance costs can escalate over the long term, as more applications and processes are introduced to the EAI backbone and the intelligence layer becomes heavier and more complicated. This isn't so much a scalability problem-most IT shops will just throw hardware at the processing requirements-as an issue of creating a fix that needs constant medical attention: someone has to make sure the custom splints and bandages are doing their varied, stop-gap jobs. While process automation and integration can deliver enormous benefits by reducing the cost to do business, a technology whose cost increases over time is antithetical to improving profitability.
That being said, EAI is not worthless. When applied to its purpose-creating a high-level enterprise view of operations-it can be very effective. "The best application for EAI is whenever you have to integrate multiple systems across an enterprise, not necessarily when you have to automate and integrate something like a provisioning process. When you have 20 different systems integrated at a macro level, that's where EAI comes into play. You use it for subscription and notification, but you're not using it to automate that detailed process," says Bender.
EAI can run into trouble when it's expected to handle the kind of complex and voluminous transactions native to any service provider's systems environment. So it's no wonder that large service providers would approach EAI with a healthy sense of skepticism as they look to automate and integrate specific transaction- and data-intensive processes. To address this skepticism, American Management Systems (AMS) developed a system for tackling one of the biggest challenges associated with EAI-type projects: product catalog integration.
The Product Catalog Challenge
The challenge surrounding product catalog integration isn't exactly news; however, integrating product catalogs is a high priority for most large service providers. "It takes four months to roll out a new product. That's a major problem. Part of the reason is that the product has to go into 20 systems," says Bart Steimel, senior principal with AMS' new media and communications group. He explains that because so many service providers are amalgams of merged and acquired companies, it's not uncommon for large carriers to have 30 to 50 different applications with separate product catalogs. Those applications run on a range of platforms such as the AS400, mainframes, Unix boxes, NT servers and so on.
In many cases, the product catalog is the heart of each application, so it can't be stripped out or deactivated. Once again, the lack of a common data model pretty much rules out synchronization-especially when the catalogs are being synchronized manually. The question becomes how to tie existing product catalogs together in a way that leaves them intact, but creates a complete view of all of the provider's offerings. The last thing a provider wants to do is add a new, full-blown application to take over all product catalog responsibilities, but that's essentially what's necessary.
AMS, for example, after spending considerable time building custom intelligence into middleware to integrate multiple product catalogs for various clients, decided to develop a platform-independent, logical enterprise product catalog that acts as a single point for entering or retrieving product catalog data. This logical catalog then integrates to and feeds the variety of disparate product catalogs in the background, effectively abstracting the product catalog mapping into a central product catalog interface. A logical catalog isn't a totally new catalog with its own robust data repository. It is a series of pointers directing information to and from the background catalogs, enforcing processes, and then integrating the relevant data to provide a CSR or business analyst a complete view of product offerings across the organization.
This logical catalog employs N-tier-level data structures and metadata, which should provide enough flexibility to support a data model that can accommodate the idiosyncrasies of the background systems, but remain lightweight enough not to be another full-blown application. Beyond the technology, however, AMS stresses that there's a philosophical shift at hand. "CRM has become critically important," says Steimel, "but product catalogs don't get the respect they deserve." The problem, he says, stems from the fact that product catalogs are viewed as adjuncts to other systems. AMS argues that the product catalog should be seen as a critical entity unto itself. "If I have products all over different systems, how do I tell my customers what products I have?" asks Marty Rosensweig, vice president of solutions architecture for AMS' new media and communications group.
AMS' approach can work with an EAI bus because it abstracts the complex intelligence that's otherwise custom-coded on top of the EAI layer. Further, it drives all product catalog-related processes through a central point, which is basically overlooked when product catalogs are mapped together under the EAI covers. In this case, AMS' approach is practical and attempts to solve complex business problems without throwing loads of new technology at the service provider's problem. In contrast, when the focus shifts to standardizing technology, replacing old systems with new ones, or other grandiose ideas, business goals are lost and costs can get out of hand.
A Look at the Long Term
Economic conditions drive integration approaches. The current economy dictates a shift toward more practical lines of attack that can solve problems and pay for themselves rapidly. But there's still a big picture to keep in mind. "Large carriers still have dreams of integrating the enterprise on some bus architecture. They're still saying that's their vision and that's where they want to go," says Rosensweig. This statement suggests that service providers are still thinking about solving business problems in terms of specific technologies. From a business perspective, service providers want a complete, centralized view of their entire operation and as many flow-through processes as possible. But moving to a huge information bus architecture isn't necessarily the best strategy. The idea of putting the systems infrastructure up on a lift and sliding an information bus with a load of custom intelligence underneath it doesn't always make sense. There are other ways to address complex enterprise integration, and one of them comes from the world of distributed computing.
Particle Physics and Integration?
Particle physics research has contributed greatly to distributed computing. The reason is that the kinds of problems particle physicists try to solve require an enormous amount of computational power. A traditional approach to solving the computing problem was to build supercomputers, like the monsters from legendary computer maker Cray. But an alternative is to apply distributed computing concepts. By taking advantage of many computers on a network, utilizing available processing resources transparently, one can theoretically employ more computational power than even a Cray can provide. According to distributed computing fundamentals, processing power and functionality of the overall system can be expanded by adding different machines to the network incrementally.
For those involved in applying distributed computing principles to solve physics problems, finding a technology that could put theory into practice proved difficult. This is the exact problem that plagued Michael Ogg, chief technology officer of Princeton, N.J.-based Valaran, in his days working at the National Science Foundation. This work actually led him away from particle physics and into distributed computing, which eventually landed him at Bell Labs. After years of challenging development, Ogg's world changed when he met Sun Microsystems' Jim Waldo, the now infamous Jini evangelist. Ogg soon co-founded Valaran, a company that applies Jini technologies to integration and change management problems inherent in any service provider organization, network or OSS infrastructure.
Jini Versus Other Integration Approaches
"One of the issues that comes out with conventional solutions is there's not a clear enough distinction, if any distinction, between a physical view of components that constitute a system, and the logical point of view," explains Ogg. "Part of our mantra is that the high-level logical abstraction is the business process you're trying to implement." That is, integration usually is focused on tying islands of automation together. Each element performs a specific function based on its individual capabilities, and the process is dictated by what the systems can do-adding patches wherever holes might appear. Jini takes a reverse approach, defining the high-level business process and its goals, and then looking to general services available from the underlying systems to meet those goals.
Jini requires that systems be looked at in a new way. "Often folks can't entirely grasp the concept that Jini services are generic, not application-specific," Ogg says. When developers stick to old thinking, they end up mapping an application's API into a Java interface and calling it a Jini service. But all this accomplishes is putting a Java interface on an already rigid and application-specific API.
As Yoda told Luke back on Dagobah in the famous Star Wars line, "You must unlearn what you have learned." Likewise, Ogg explains that the proper approach is to consider what general properties an application, such as a billing system, provides. "If I concentrate on the physical aspects of the biller," says Ogg, "I'll find that if I want to swap it out I'll have to redesign the process again. If my abstraction was better to start with, then I could more easily bring in another system providing billing functions."
Those general properties can be designed into the process model and Java interfaces so that any billing system can be swapped in or out of the process as long as it provides the correct general functions, such as IPDR rating or jurisdictional tax calculation. Further, multiple systems are usually involved in a process. Jini's service discovery mechanisms make it possible to design high-level processes and then call the appropriate services from the available applications. People often ask where services physically reside, Ogg says, but "the answer is Zen-like. The service is in many places at once-in the network elements, in the element managers and in the back office."
Addressing Practicality
Given limited budgets, service providers are keeping integration projects within departments isolated to just a few key systems in a process. Ogg says service providers have presented Valaran with tactical challenges, though they often see Jini as a "disruptive" technology providing a strategic solution. Jini can fulfill many roles, and while it's innovative, Ogg says, it's not disruptive. "It's actually the least disruptive, because it lets you use completely unmodified systems, integrate them, and gradually introduce more and more Jini services," he explains. "We can integrate with legacy applications, or even legacy integrations, without touching anything." In this sense, Jini can be applied tactically without sacrificing strategic goals or disrupting integrated pieces of the business that already work.
Complexity Within Simplicity
Perhaps the most compelling aspect of Jini, from technological and economic perspectives, is its simplicity. "Whereas most systems I come across focus on adding more and more features, Jini asks, 'What can I take away and still maintain the core function?' " says Ogg. Jini is such a clean technology that it has only required one revision. "Some have criticized Jini because it's only had one revision, saying that it can't be up to date," says Ogg. "But the revision didn't change the core at all, and that shows how a system should be designed."
This is a fascinating irony of sorts, in the sense that people have become so accustomed to flawed technology that now they're junkies for the inevitable new releases, updates and patches. But constant updates and revisions wouldn't be necessary if the technology were thought out and designed correctly to begin with. Jini is elegant, and where some technologies cannot scale economically because their inherent problems become insufferable or too expensive to maintain, Jini becomes more effective as more systems and services are added. As Ogg puts it, "Given the choice between a beautiful solution and an ugly solution, more often than not the beautiful one is the one that's right."
Just Take It off My Plate!
Sometimes service providers face problems that are so ugly they just don't want to deal with them. Rather than reengineer their systems and take on massive integration projects, they just outsource it. For example, Synchronoss, based in Bethlehem, Pa., has taken some of the ugliest problems out of the most complex back-office environments around-namely WorldCom and AT&T. Synchronoss was born out of a systems integrator that specialized in automating and integrating processes and systems in large carrier environments that are so complex that even the monster integration houses want nothing to do with them.
"We don't try to push ourselves as having all of this 'best-of-breed' OSS. ... We will attack a specific business process and pull in the technology behind it. Many of our buyers are indifferent to what's in the background as long as it works, ... and we collect a range of performance metrics to show that it does," says Tom Miller, vice president of business development for Synchronoss. In fact, the company says that if it doesn't meet stated efficiency improvement goals, it pays the customer back. This is a vastly different message than one would hear from systems integrators or consultants, who generally charge fees whether the project succeeds or fails.
Synchronoss focuses on process and business, as opposed to technology, which is a message that service providers seem willing to hear. "I think there's an emerging appreciation in the industry of the importance of process. It's always been on the side because it's not as sexy," says Corky Kelleher, vice president of strategic marketing for Synchronoss.
The key is to identify processes that are mission-critical but aren't where a service provider's focus is best spent. Most often Synchronoss deals with ordering and fulfillment processes related to wholesale and enterprise operations, with fundamental goals such as taking a fulfillment process that requires 250 people to support it down to just 100. Its approach involves first analyzing all of the data points and processes and identifying inefficiencies before moving the overall process to its own OSS platform, which is built on an integration bus from BEA Systems. This platform provides what is essentially an adjunct customer support system insulated from but integrated with the service provider's back office, accessing and delivering data to and from the legacy environment, but leaving the operator's other internal systems and processes intact.
Wholesale ordering and provisioning and enterprise self-help are relatively new areas for service providers. Rolling out such functions could cause great disruption because of their links to the legacy environment. By outsourcing the functions and integration to Synchronoss, the service provider not only offloads the burden of building out new functionality, but effectively insulates itself from constantly changing enterprise and wholesale requirements. This is particularly important in the wholesale business, where margins are small and the slightest process efficiencies make the difference between profit and loss. Further, Synchronoss claims its customers see 5 to 7 times the leverage on an investment with them that they would investing the same amount in vendor products and system integrators or internal development.
It's All About Process
While this article cannot address all the many integration options available to service providers, there is a consistent message: Technology alone cannot solve business problems; it provides the tools for doing business better. When service providers, vendors, consultants or business process outsourcers get lost in their technologies or their fee structures, they lose sight of the whole point of integration, which is to improve business operations. There is no single technology that can solve every problem. There's also no place for technologies in the back office that try to forklift established processes and systems, because such disruptions lead to stranded investments, poor operations and increased costs. In large carrier environments, especially in a restrained economy, there's no room for integration technologies that can't embrace change, can't scale and cost more over time.
Edward J. Finegold leads Stylus Telecommunications, an independent consulting and research firm. He can be reached at ejfinegold@styluscom.com
Email this article
Add a comment
Printer version
Order reprints
RSS Feed
Bookmark article