Technical Tutorial: Enterprise Integration

Companies constantly struggle between a quick-fix versus a long-term strategy. Adding to that mix, technology and methodology wars within an organization can further muddy the waters.

Every company, large or small, needs an enterprise integration strategy now, even if the need for an integration technology is a ways off. If you can’t electronically connect with customers, you lose them. If you can’t automatically trade with partners, you incur manual labor costs and miss out on market economies of scale. Internally, unconnected systems give rise to manual processes that are less efficient and more expensive. Bottom line: delaying an integration strategy costs time and lost opportunity.

A good integration strategy plans for the future while accommodating the reality that most IT initiatives are measured using short-term costs and benefits. Although business objectives must dictate strategy, IT systems and their incompatibilities can get in the way.

There are many tools in the arsenal for developing an integration strategy, including a) using consistent technology; b) using industry standards; c) replacing one system with another; d) building single-system-to-single-system bridges; and e) adopting integration technology. Large and small companies alike can apply these techniques, although the situations they will encounter might be quite different (see sidebar “Integration Strategy Basics”).

Integration Strategy Basics

1) Consistent technology: As organizations grow, they can mitigate IT incompatibilities wherever possible by employing consistent technology. That is, use consistent operating systems and computer platforms, use database software from one provider if possible, and use development tools from one vendor. Homogeneity, wherever it can be achieved, mitigates or eliminates most integration challenges.

2) Industry standards: Relying on industry standards increases the likelihood that two products can interoperate, while decreasing the risk of switching products. For instance, if a Web presentation application is based on XML, generated by a relational database system, then that database system can be replaced by another so long as it supports XML generation. Users benefit from standards for the same reasons some providers avoid them—standards can turn otherwise novel products into commodities. XML adoption should help enterprises become compatible and integrated, but it is just one piece, albeit a significant one, in the integration puzzle.

3) Is one system always better than two? If there are occasions when one system can be replaced by another, the benefits of consolidation might outweigh the costs and risks involved. Besides reducing maintenance costs, one less system makes the enterprise more consistent and simple, and therefore more able to accommodate future needs. Data integration software can help perform the system migration.

4) Point to point: The quick fix? Building custom bridges from one application to another offers the benefits of low costs and short implementation times, when compared with the introduction of larger-scale integration software. If the needs for systems to interact are relatively few, custom bridges should suffice. If the needs are more numerous, the added complexity of the individually developed bridges, along with the lack of a more broadly applicable integration technology infrastructure, makes customized system bridges cost-ineffective.

5) Enterprise integration: There are products that provide out-of-the-box system-to-system bridges and a toolkit for building others. They can provide the heavy lifting needed either to overcome large-scale system incompatibilities, to provide a foundation for building new, cross-system applications, or both. When your IT environment is not able to satisfy business needs and you need an enterprise-wide fix, integration technology is an option to consider.

Although a good integration strategy plans for the future, it is difficult to predict. Looking to the past and forecasting the future can help.

Looking back, how has technology changed in the past several years? The use of database systems has evolved toward the widespread adoption of relational database systems, but overall it is not the technology so much as the accessibility (e.g. PC-based) of databases that has changed. Programming languages (COBOL, C, C++, Smalltalk, PowerBuilder, Basic, Java) have come and gone and stayed, showing much more turnover than the database marketplace. Proprietary networks have given way to TCP/IP. Client/server, and now object-oriented middleware, is still evolving and gaining support, after a seemingly interminable number of years.

Thinking forward, consider how your IT environment might accommodate specific changes in the future. Object-oriented databases were at first slow to emerge, but their momentum now is significant. LAN and wireless bandwidth in particular will most likely become a cheap commodity, with GPRS driving the economic model. Will new communication protocols—such as the wireless e-commerce protocol efforts underway in the WAP forum—need to be accommodated? What if HTTP, the existing dominant Web protocol, is replaced with something else? Taking these issues into account will help determine the success of your IT environment.

When Does Your Strategy Require an Integration Product?

Do your employees have to enter the same information into multiple, redundant systems?

Are service reps looking at several terminals to answer customer inquiries?

Are maintenance costs swallowing your IT budget?

Could customers access the Web for some services using inexpensive applications instead of costly people?

What’s The Cost?

Like any technology, enterprise integration technology has costs. To make economic sense, an integration technology must minimally solve enough problems that its costs of introduction are more than offset. In addition, the technology should support a long-term IT strategy. Determining whether an integration technology will support a long-term IT strategy is less quantifiable, but worth considering.

Since integration technologies work across business and application lines, they will tend to show up as consistent foundations for addressing many areas. An integration technology should streamline the enterprise IT environment so as to reduce the cost of implementing new applications in the future. It should hedge against the risk of technology changes and simplify the IT environment by bridging gaps of incompatibility. Ultimately, it should offer a simpler view of the enterprise and provide a conceptual framework that makes it easier to envision an IT strategy that delivers substantial benefits for the business.

How to Select an Integration Technology

If you have determined that integration technology can help your organization, how can you evaluate available products to select the best one? If you can list areas of need, identify the best mechanisms for satisfying those areas, and can see a mechanism figuring prominently across multiple areas, then that mechanism merits consideration.

The table (B&G note location of table) can be the basis for a spreadsheet used to assess a product’s fitness for a particular situation. You will need to weigh each factor appropriately for your organization—some criteria might even receive a weight of zero. You will no doubt add new criteria and modify, break out, or remove others. Such a scoring tool can help to quantify seemingly subjective elements and can play a key role in your product assessment methodology, not just for integration products but all IT products.

What Types of Integration Exist?

Although the goals of integration technology—increased system interoperability to yield new capabilities and reduce costs—are always the same, the means can differ. Three major means exist: information integration, business process/rule integration, and user interface integration.

Information Integration

Information integration simplifies both access to information—wherever it might reside—and the exchange of information between applications. In other words, each system’s information model and input/output mechanism become compatible with those of other systems. Centralized enterprise data maps (these could include catalogues, metadata, and data dictionaries) and the use of XML as a data interchange format are some of the techniques found commonly in information integration products.

Business Process Integration

Using business process integration technology, for which the enterprise appears like a vast suite of business services or tasks, two or more systems can be merged as cooperating applications by brokering function calls between systems via a centralized function repository and server. Business process integration tools usually perform protocol conversion, allowing different call interfaces such as COM, CORBA, Java RMI, C-style, and others to speak seamlessly with one another. Some products automate business processes as workflows that can be designed like flowcharts and orchestrated using a central workflow server.

For instance, the receipt of an output stream generated by one system might trigger the invocation of a function or process provided by another. Back-end integration products might offer information integration, business process integration, or both in tandem.

User Interface Integration

The remaining focus of this tutorial is user interface integration. Ironically, it is one of the least covered strategies, yet it’s very common after large-scale M&A activity.

The technique of integrating applications at the front end, by combining their respective user interfaces, has received less attention than the two back-end approaches. User interface (UI) integration comes in five notable flavors: 1) console consolidation; 2) window integration; 3) application integration; 4) treating a user interface as a programmable interface; and 5) custom-built techniques for user interface integration. The first three methods help to make a user more productive, with console consolidation being the least powerful and application integration being the most powerful. Option 4 exposes a programmable interface and thus works like business process integration. Lastly, customized integration techniques can be provided by in-house developers or consultants but are not yet packaged as commercial products.

Console consolidation, window integration and application integration mechanisms boost user productivity by simplifying an application’s presentation and interaction points. By so doing, the user can spend more time performing useful tasks and less time manipulating the application interface itself. Moreover, users focus better on their tasks when they are less encumbered by the mechanics or nuances of the interface, and make fewer errors.

Console Consolidation

Console consolidation presents all needed applications on a single terminal, thereby reducing the user’s need to switch terminals when performing a single task. With many terminal emulation programs on the market, one can usually attain this by running a single Windows desktop and one or more emulation products. With such a configuration, a single user could run host applications using 3270 emulation, UNIX applications using TELNET and/or X-Windows emulation, AS/400 applications using 5250 emulation, and Windows programs natively. Although such a configuration supports most types of UI applications, it won’t help you run an OS/2 application, since OS/2 emulators are not available. Console integration might not provide a comprehensive, long-term solution, but it is quick and relatively cheap.

Window Integration

Window integration goes one step further than console integration by presenting a set of windows that appear to operate the same way a single, integrated application behaves. This mechanism essentially glues applications together, creating the semblance of one application, albeit with some noticeable glue droplets. With one common technique, a controlling application launches a “child” application, passing a data file as a command line argument. When the user closes the child application, the controlling application examines the data file and processes any changes accordingly. Users of Lotus Notes might recognize this behavior, since it is how Notes launches Microsoft Word to edit certain documents. In order to use this technique, one must be able to either create a new controlling application or modify an existing one to act as the controller. The market for products using this and related techniques is sparse, but you might be able to find one or you can modify an in-house application.

Application Integration

UI integration nirvana exists when multiple applications actually work so seamlessly together than there are few, if any signs that the integrated application is more than one. This is what we call application integration. Technologies for achieving UI nirvana include Microsoft’s COM and Java’s JavaBeans component model. For instance, with COM an Internet browser can display a PDF file as if PDF support were built in, even though the PDF viewer is actually Adobe Acrobat running separately and unseen. With both COM and JavaBeans, developers can purchase UI application modules that they can snap into a larger, homegrown application.

COM and JavaBeans require a reliance on Microsoft’s and Java’s UI development standards, respectively, and it is not yet possible to reliably commingle COM UI modules within Java applications and vice-versa. However, if this level of integration can be achieved, the resulting application-powered boost in productivity can be well worth it. Unfortunately, many applications cannot be packaged as either COM or Java UI modules, or can only be packaged at prohibitive costs. COM-enabled 3270 or 5250 emulators cannot yet be found (although at least one company is planning to deliver one), and JavaBeans-powered 3270 and 5250 emulators are available from the tiniest number of small providers. It is not that such products cannot exist, it is that no significant market for them has yet emerged.

Callable Interface

The fourth type of UI integration, in which a user interface is treated as a callable programming interface, is both promising and rare. With a callable interface, a UI can act like any software library, allowing its capabilities to be embedded within any other application. Unfortunately, most user interfaces do not offer programmable interfaces, perhaps because it requires substantial effort and cost. Within the world of legacy applications, the most prominent use of this technique treats a 3270 screen as a programmable interface. Since a 3270-based application is simply a set of screens, and a 3270 grid is simply a block of characters, enough predictability exists to interpret any given screen’s output and simulate the screen’s input to effectively make the application programmable. When you hear the words “screen scraping” or HLLAPI (pronounced he-la-pee), you know this technique is being used. Of course, with this technique one must recreate the 3270-based user interface, incurring costs as a result.

Customization

Some UI integration techniques may not be available as commercial products but can nonetheless be implemented by a qualified in-house developer or consultant. As examples, consider three techniques. Whereas HLLAPI allows an application to interrogate an application screen and feed it simulated user entries, it is also possible to intercept 3270/5250/VT100 transmissions to gain even more control over the interaction between a 3270/5250/VT100 terminal and the host computer. The interceptor, also called a proxy, can pass transmissions back and forth between the terminal and host unchanged in some cases, perform special actions when specific transmissions are detected (e.g., the saving of a customer profile), convert a transmission to another protocol (e.g., from 3270 protocol to HTTP), or any combination of these options.

The structured, screen-oriented nature of 3270/5250/VT100 transmissions enables a number of powerful proxy capabilities: a) monitoring tasks for management and accounting purposes; b) monitoring task completion to trigger workflow actions carried out by other software systems; and c) conversion from one protocol to another, perhaps to deliver 3270 screens via HTML to a Web browser. In the Web world, proxies have been used to strip advertisements from Web pages before presenting them to the user.

Another customer UI integration technique uses software to simulate a user to interact with a Windows application programmatically. For instance, by simulating the menu clicks used to import raw data into and export generated data out of an accounting application, one can use an application as a programmable input/output process. The same technique can be used to select dialogue boxes, enter data, print reports and practically anything else a user can do. Tools to do all of these things exist on the market, but they are generally advertised as software testing packages. With advanced techniques, third-party applications can be made to run seamlessly within a hosting application, essentially achieving UI integration nirvana without using COM or JavaBeans.

Conclusion

Using console consolidation and window integration techniques, one can usually implement quickly to get a moderate level of functionality. Application integration usually requires more effort but results in higher functionality. The callable interface technique usually requires significant effort, but presumably much less than replacing or updating the underlying legacy application. Custom-built techniques can deliver substantial functionality, albeit without the support of a packaged solution. For shops with low-powered, low-memory PCs, some UI integration techniques might be infeasible without costly hardware upgrades.

So when might one use the different UI integration techniques? We see using console consolidation and window integration as short-term measures, because of their relatively short implementation times and reasonable costs on the plus side, and limited functionality on the down side. If a UI component integration option is available, consider using it if the resulting efficiency of the aggregate user interface outweighs its complexity and costs of maintenance. Consider using a customized solution if it can be leveraged against a critical mass of existing applications, and if it costs significantly less than back-end integration alternatives.

As a rule we lean toward back-end integration for two reasons: 1) the techniques work more formally, using an enterprise’s data model and process/function model; and 2) the marketplace for back-end integration is more mature now and growing rapidly. In practice, UI integration technologies merit consideration and in certain cases offer the most cost-effective solutions.

Setting an Integration Strategy

You need an integration strategy now, whether you are an early or late adopter of the growing number of integration options. With the preceding information you are ready to formulate one. Methodologies and guidelines have to make sense to you, and must be cast in terms you can articulate and substantiate to others. Plan for long-term consistency and simplicity that will help you achieve near-term gains on a recurring basis.

About the Author

Stuart Hoffman is the founder of iSoft Corporation (www.isoftco.com), a leading-edge provider of billing, customer care and data integration products. Hoffman has worked for 12 years in telecommunications and related industries since earning his master’s degree in computer science from the University of Virginia. He can be reached at shoffman@isoftco.com. Bob Sewell and Rick Gutieber, two senior iSoft technical specialists, contributed ideas to this article.