Because of the largely manual processes of discovery and reconciliation, the telco industry typically has a 70-percent accuracy rate (outside plant) for data. The seeds for such a low standard were sowed during the late 1990s when the Wall Street dream of a unified back office galvanized providers to spend money on vendor promises of tying together order management, network inventory and auto provisioning. “You got money before you knew what to do with it,” says Malcolm Rodrigues, vice president of network operations for StarHub, a service provider in Singapore that offers cellular, fixed network and cable TV services. “The reality that hit was that no one could really work across the board with all of those components.”
As service providers begin to emerge from bankruptcy they lack CapEx, so they are concentrating on improving their OpEx lines. “To do so, they must refresh and verify dirty data, or start from scratch if they are to be more aggressive with SLAs for next-generation services,” says Jean Laurent, vice president of marketing for Sheer Networks, which does real-time discovery and reconciliation for inventory and service fulfillment.
“For service providers in the past, the problem was network management, which meant finding the cause of a problem. Today, however, the business is so service- and customer-centric that you have to discover not only the root cause, but the impact on service to the customer.”
The CapEx benefits will come from getting a true picture of what equipment providers have, and the operational benefits from downloading the paths of use for that equipment, as well as the services and customers tied to that equipment.
Unifying the View
Frustrations around having to manage fragmented systems that hinder operational efficiency and network convergence can be resolved only with a truly accurate view of the current network. For that reason, standalone inventory systems in provisioning, network operations, fault management and billing are expected to increasingly give way to integrated network resource management (NRM)—the OSS components of which are network discovery and reconciliation.
Service providers are easing into NRM with step-by-step implementations, rather than comprehensive replacements. “There will be no ‘big bang’ implementations, as carriers will take auto-discovery in smaller bits, essentially attacking layer 1 or layer 2 first,” says Verne Anton, principal analyst at Gartner. Rather, he thinks carriers will go market by market or by technology vertical to slice the problem into smaller pieces. “For example, DWDM cards are very expensive, so you may want to accurately account for those cards first.” For more on auto-discovery at each network layer, see “Uncovering Layers”.
In 2005, RHK predicts service providers will spend $1.3 billion on NRM software and related services, with up to three-quarters of that amount spent on commercial software.
Auto-Discovery Defined
While discovery is the process of evaluating data to describe the current state of the network equipment and its configuration, “auto-discovery” refers to the process of discovering network equipment that was previously not known to exist. The effect of auto-discovery on inventory can be far-reaching, as it resonates to other performance processes that affect customer and network provisioning.
“Network inventory is the nearest and dearest thing in my heart right now; from a process perspective, we want to get the network inventory system locked in so we can improve trouble resolution and provisioning,” says StarHub’s Rodrigues. As StarHub evolves from Excel spreadsheets to databases and into fairly large network inventory systems, the hope is that soon order management and client care systems can be tied into inventory. Currently, Rodrigues uses Arkipelago for recognizing the transmission systems and some of the switching network in the inventory. “They have an impressive development team and the type of responsiveness in modifying the roadmap when necessary,” he says. He hopes the company will be able to help him get a handle on his ATM, DSL and IP platforms as well. He also uses interfaces between facilities management systems and order management from Vantive for billing and CRM. Ideally, Rodrigues would like to provision the SDH network, “but not even the equipment manufacturers can do that on their own networks.”
One Element at a Time
Because auto-discovery doesn’t automate processes circuit by circuit, rather element by element, most auto-discovery systems cannot see the end-to-end circuits in networks, which means manual audits of non-intelligent devices will still be part of an auto-discovery strategy. Until non-intelligent devices can be auto-discovered, service providers must have business processes in place to automatically feed data into auto-discovery systems.
While the vendors have the code, they can’t yet get solutions to work on all circuits on all nodes. “The ideal would be to have the ability to provision the SDH network; however, not even Lucent, Nortel or Sycamore can do auto-discovery well in their own networks, as most do element-by-element management,” says Rodrigues.
The step beyond managing elements will be connecting rings and meshes, which is what really matters to the service providers. “Ultimately, service providers would love to load all the circuits in there or even load the VPNs that are set up and the bandwidth and how the VPN looks,” says Granite Systems’ Mark Mortensen, chief marketing officer and senior vice president of product management. He notes that element management systems (EMSs) don’t know how to present such data, so auto-discovery systems cannot ask the right questions to enable that. According to Mortensen, the “state-of-the-art” for auto-discovery will emerge once vendors figure out how to pull information about what ports and switches are involved in the DAX, as well as the cross connects to determine the paths through the network.
While most IP, ATM, and frame relay equipment can be auto-discovered, discovering SONET/SDH and DWDM equipment is not possible today with inventory systems, which need an initial address to search for nodes or gateway nodes. “For these nodes, the service provider must provide an address of the node in order for other discovery functions to proceed,” says Goldman.
Because networks are full of different products that have not been developed in a consistent, forward-looking manner by manufacturers, the core network devices and access devices are not hooked together, making it difficult to correlate circuits across independent systems.
There is hope, however, in the fact that Lucent is developing a script of software for auto-discovery of circuit data for one of its European customers. “If it works, then it will be turned into a software tool we can use, too,” says Rodrigues, who wants the ability to manage customer circuits hour to hour. “Then, when things fail, we will know who to call and what to work on,” says Rodrigues. In the meantime, he uses an ad hoc tool to import information to his WaveStar system. For now, complete service management is difficult. “We can’t say ‘here is an E1 circuit or DS3 circuit at this node across this trunk to this node,’” says Rodrigues, who instead uses Arkipelago data to manually populate the WaveStar system. “It is important that we clearly define the process so that it is understood that every time a new circuit is installed, modified or removed, it is populated into the system.”
Reconciliation
Reconciliation is a major part of auto-discovery, as it is the process of comparing one view of the network inventory with another in order to identify and resolve discrepancies. Reconciliation happens most often in inventory systems. While companies like Cramer, Granite and MetaSolv focus on inter-domain enterprise inventory management systems, there is a trend toward consolidating inventory systems.
Inventory management, some believe, will evolve to include physical inventory (equipment line cards) and logical inventory (cross connections, IP addresses, etc.) as well as management data, such as contact phone numbers. “We see more and more that customers are looking at inventory management COTS,” says Jeff VanZwol, marketing manager at Elematics, whose intelligent network control plane technology helps carriers discover networks or end-to-end provisioning of networks at different layers. He says customers are replacing internally developed databases with automated systems that will maintain not only an inventory of physical switches and modules, but also fiber optic links and locations for circuit switches delivered through the network.
“Because of the expense of keeping interfaces current, we see that even those who build their own are evaluating commercial solutions,” says Granite’s Mortensen, whose XML-based gateway, DiscoveryXng, handles reconciliation between the database and inventory systems by working with its auto-discovery OEM partners like CoManage and Acterna.
“The first six interfaces service providers do are easy, but when they grow to have 600 elements that they have to change twice a year, the complexity of maintaining discovery systems becomes very difficult,” says Mortensen.
As a result, companies like CoManage, Acterna and RiverSoft (acquired by Micromuse) are emerging with COTS functionality for discovery and reconciliation. These companies work with inventory players to specialize in inventory and reconciliation across SONET, TDM and legacy switches, as well as ATM, frame relay, VLANs and IP VPNs. Other OSSs from companies like Micromuse, have acquired assets to auto-discover such things as fault management.
“The problem usually is reconciliation because service providers need a broad and complete data source in order to drive the overall reconciliation against the broadly deployed inventory systems,” says Andy Fraley, CTO and co-founder of CoManage, whose TrueSource has been developed to supply both discovery and reconciliation. “Right now, service providers need a very broad collection capability to cover broad technologies if they are to build accurate physical, logical and service views of services and network infrastructure supporting services across networking domains.”
Traditional discovery tools, even those with a strong enterprise focus (i.e., HP OpenView or RiverSoft), don’t go across networking domains. “SNMP-based solutions don’t work once carriers drop into the core portion of the network, where there are DAXes and classified voice switches that go across many racks,” says Fraley. He says that operators using traditional discovery methods for reconciliation often encounter matching problems in aligning inventory systems with network views. “Carriers still think of inventory in a purely physical manner—what CO, what floor, what rack, what row, which mount point. They then tag elements with location codes, which leads to confusion in matching when the same box has its own location code.”
In developing broad discovery solutions, inventory management players will model relationships between different technologies for a more seamless inventory so that carriers, for instance, could see which ATM circuits ride over a SONET/TDM core to the edge, where DSL, voice over IP, Ethernet, and VLANs reside. To recognize the paths in the core of the network, and to have discovery systems build the same image as the inventory systems, will enable true reconciliation.
Not a Silver Bullet
Many discovery solutions are designed to concentrate on one task, such as provisioning or fault management. For that reason, Fraley believes auto-discovery in and of itself is not the complete answer but part of the solution. Even if used in only a percentage of the network, it’s a start toward end-to-end visibility. “For now, it can build an accurate picture of the discoverable portions of a network and become a detailed data source for the overall reconciliation process. That is different than inventory, which models everything you touch—whether the physical structure of cards or the logical structure of cross connects,” says Fraley.
Fraley says service providers should look for auto-discovery solutions that go “broad and deep.” “You need to look for a depth with analytics and live discrepancy views, rather than using static tools for generating paper reports,” he says.
He believes discovery should be deep enough to drive all OSS functions. “By getting information from each element that builds a rich image, discovery systems should identify all networking technologies from SONET to frame relay,” he says, noting that the ability to talk to elements in the core is different than talking at the edge. “They have to speak TL1 and CORBA, as well as handle proprietary interfaces to EMSs and command line interfaces,” adds Fraley.
For breadth, he notes, discovery needs to pull that information through management protocols and mediation paths to handle various OSSs throughout the network. “The level of information collected to support reconciliation against all OSS entities should be deep enough to identify committed resources in the network, which can drive fault management as it can model logical and physical entities in the case of failure as well as a correlation path structure.” For billing, Fraley says carriers need to see end service touchpoints to see what customers were billed for what services.
For that to be possible, service providers seeking a reconciled view of key enterprise data must have NRM software that includes “intelligent and active” inventory management, capable of detecting changes that impact other systems, says RHK’s Goldman: “You have to have a system that recognizes a mapped DS3 means 28 DS1s.”
However, as carriers try to consolidate inventory systems that go across SONET, TDM, and legacy voice switches into ATM, frame relay, VLANs and IP VPNs, reconciliation becomes a top concern.
Reconcile Layer by Layer
To start, carriers should have a strong reconciliation framework in place, which allows them to synchronize external sources of data in a business process-centric mechanism with the inventory system. Then, make sure the framework interacts with external auto-discovery sources and NMSs, which feed its framework in forming an initial baseline for auditing and synchronizing inventory systems. “That has to be done in a business-centric framework,” says Sandeep Sharma, director of product management, overseeing MetaSolv’s NRM solutions, noting that auto-discovery can be augmented with correlation of auto-discovered data with data from other sources, such as asset management and CRM systems. “For example, asset management systems put asset management tags on data, and they cannot auto-discover those tags.”
On-the-fly population is a logical next step to eliminate spreadsheets and manual intervention. “When field op people remove a card with an asset tag, they can scan it with a PDA and enable the updated information about the shelf or cabinet into which the card was inserted,” notes Sharma. He recommends getting a strong initial baseline in place. “Make a composite of manual audits and auto-discovery technologies, and correlate with asset management and CRM to tie discovered data to asset tags or customers.”
Carriers can feed into a reconciliation framework to identify discrepancies and have the business process-centric mechanisms to deal with those discrepancies.
Dirty Data
Auto-discovery is just one source of data, which is only as good as what the elements allow you to discover. “Carriers don’t want to put dirty water in their shiny new buckets,” says VanZwol. “They shouldn’t feed inaccurate data into new systems, which is where discovery comes in to automate the process of extracting a real view of the network, and to bring into the database management system.”
He says it is moderately difficult to scrub old data that will populate databases once it is refreshed. Depending on the level of confidence in existing data, some companies choose to start from scratch. “Usually carriers who have gone through multiple accessions have a lower level of confidence in their data,” he says.
“However, reconciliation is not as simple as discovering the in-service inventory and replacing all the other data with the ’real’ network data,” warns Goldman, who notes other systems often contain important, manually maintained data that should be kept, even if the associated piece of equipment is no longer in the network.
What they need is a broad and complete data source in order to drive the overall reconciliation against broadly deployed inventory systems.
The Data Is Only as Good as the Process
It is necessary that the industry begin to attack the broken processes that cause inaccuracies to occur in the first place.
“Auto-discovery is supposed to fill the data quality gap,” says Robert Curran, director of marketing communications for Cramer Systems, “but it cannot magically fix data quality without resolution of issues around processes.”
If operators can address the process issue and understand how data is changed, as well as the processes that update and create data, then the data quality becomes reinforced by the process. Some operators try to resolve data quality issues by discovering the data first and the processes later. “They are gravely mistaken if they think they can crack the data quality problem with data first,” says Curran. “Then the implication is that the network knows all there is to know, as well as what you need to know.” The reality, he says, is that the network should be treated as though it has no idea. While he acknowledges there is an important role for auto-discovery in the OSS cycle, he looks at it purely as a policing function. “Because the process is actively driven from the inventory management layer, auditing and confirmation processes should be supported by auto-discovery,” says Curran.
Choosing the Solution
Carriers should be cautious in evaluating auto-discovery offerings, as there are many components to consider. “Look carefully at the portfolio of network elements supported by certain vendors,” says VanZwol. Even vendors who claim to support 75 percent of your NEs leave out the development times for cranking up support for NEs not in their portfolio. “If they take a year to support the ones missing in the portfolio, you might be better off going with a company that has a smaller portfolio but quick development turnaround,” says VanZwol.
Also, carriers need to be wary of network inventory system vendors that promise the world with relatively “virgin” software. “Every time you want to define new fields for service identification or codes, you could be charged a fee for each database customization,” says Rodrigues. He believes vendors should be more proactive in building templates against which smaller players can define their services. “If I have a Nortel installation with a Fujitsu core, they should have a solid template of data for configuration, since they work with so many companies,” says Rodrigues. “I don’t want to pay $5 million for development, after paying $10 million for the software.” In fact, he says, he’d rather pay $15 million up front to modify systems and data to that template, so he can keep up with changes. “It’s a shame that no one out there has something that is 90 percent there when you buy it to eliminate customization.”
He believes they could, “but they are afraid that their current revenue structure around maintenance and professional fees to configure would collapse,” says Rodrigues.
He says vendors are looking at it in the wrong light. “Better service should be the goal; it should take seconds, not 15 minutes, to find circuit data for resolution so that there are fewer outages in the provisioning process and faster repairs when things break,” he says.
“Don’t look at auto-discovery as a movement toward elusive plug-and-play environments,” says Gartner’s Anton. Carriers want to plug in network elements rather than specific applications, however the issue is the fact that most carriers possess multi-technology and multi-vendor environments. While carriers would like the freedom to purchase network elements in a homogenous manner so they could easily plug into the network and activate, there is a danger that auto-discovery could compound existing problems, according to Anton: “Service providers must evaluate network element interfaces. If they don’t work sufficiently, or at all, then you’ve just compounded the problem, as intervention will be more costly then the processes you had in the first place.” However, he notes that having the proper network element interfaces in place will improve the design capabilities, data quality and integrity.
Networks contain many layers. Layer 1 consists of the pipes over which servicesride, whether IP packets or ATM cells or voice. Layer 2 can consist of the framerelay and ATM that rides on top of layer one. And layer 3 includes the IP packetsthat ride on top of layer 2. “It is important that discovery understand what layer of the network you wantto focus on, as there are different skill sets for each,” says Jeff VanZwol,marketing manager at Elematics, noting that Elematics focuses on layer 1, whichdeals with SONET terminals, digital cross connects and DWDM multiplexers. Providers looking for auto-discovery in layer 2 should, for instance, look forexpertise in frame relay, switching and edge or frame relay core expertise. Forlayer 3, they should look for expertise in things like Cisco and Juniper routers. “While you can discover multi-access service platforms, there are limitationsin that you cannot see who owns the device, or in what building and what roomit resides,” says Sandeep Sharma, director of product management, overseeingMetaSolv’s NRM solutions. Its Evergreen Inventory Practice involves a methodologyfor data migration and synchronizing inventory management views with those ofthe network. The organization has done more than 30 data migrations, which hashelped build a reconciliation framework that consists of data migration, auto-discoveryand integration of EMSes. The organization touts a “bottom-up” approachof finding the physical assets before looking into services. “Even thoughthe discrepancies are highest in the services layer, with stranded service assets,service providers look first at their physical layers,” says Sharma. By going from physical to logical to service layers, the difficulty increaseswith each step. “You start with equipment because it’s the simplest;you can get the success behind you as an enabling step, then you have a foundationon top of which to build the other two layers,” adds Sharma. “Don’ttry to solve world hunger first; build successes step-by-step.” |
