Managing Capacity in IP

On-demand, interactive services have been forcing carriers to over-provision IP cores to assure quality without circuits. However, the strategy of over-provisioning increases exponentially with demands from the edge. "Each new DSLAM can mean many more connections in the core," explains Cramer's Peter Briscoe, worldwide product manager. "That gets expensive, because without knowing where traffic is going, you cannot have focused network build-outs."

For now, carriers are still buying a lot of IP hardware for their core networks.

Companies are continuing to build out a common transport infrastructure for network services, with a concentration on doing so more efficiently than was the case traditionally.

"We are building a common transport infrastructure not just IP networks, but high-bandwidth services such as wavelength services, storage area network extension and perhaps future connectivity for computing grids," says Jack Wimmer, VP for network architecture and advanced technology in MCI's Operations and Technology organization. He believes it's possible to substantially increase capacity for such services without unit cost increases. He says the economics of converged networks are compelling, "especially when you factor together the network, the back office, services and automation of service delivery," says Wimmer.

MCI expects a 50 percent reduction in the number of network elements necessary to meet expected bandwidth demands as MCI replaces SONET across its core (between metro areas) with optical network deployments. Using Ultra-Long-Haul (ULH) Dense Wavelength Division Multiplexing (DWDM), Wimmer says, MCI has significantly reduced the number of network elements needed on each wavelength it adds.

"As much as 70 to 80 percent of electronics in SONET are attributable to regenerators," he explains. "ULH extends to 2,000 to 3,000 kilometers—a huge improvement over SONET systems, which require regenerators every 500 or 600 kilometers." By cutting down on costly electronic regeneration equipment on each wavelength, MCI can connect "city pairs" without regenerators. By laying thousands of miles of ULH fiber across New York to Cleveland, Chicago to Denver, and Salt Lake City to Los Angeles, MCI will decrease the number of costly facilities with electronics and air conditioning in rural areas. Additionally, ULH extends WDM capabilities to as much as 160 wavelengths, which can bring carriers closer to "just-in-time" capabilities for incremental bandwidth increases.

"Traditional SONET deployment cycles for transport bandwidth can extend to nine or 12 months for planning, equipment procurement and construction/installation intervals," Wimmer says. "With ULH, once the infrastructure is in place, adding a wavelength only requires plugging in one card, or transponder at each end of the transport system."

Another plus is that ULH could support 40 Gbps wavelengths on the same infrastructure that supports 10 Gbps wavelengths today. "We have confirmed compatibility and performance of 40 Gbps through both lab and field trials," Wimmer says, "so we expect this integration will significantly reduce the cost of migration to 40 Gbps, because it eliminates the cost of deploying a separate 40 Gbps infrastructure."

Deployments of backbones, transport, and access networks vary, but most aggressive IP strategies today share a common thread of IP/MPLS technology.

A Common Backbone

With converged IP/MPLS networks, multiservice providers hope to better accommodate emerging QoS-driven and on-demand services, as well as services converging from frame relay, ATM, PSTN, Internet and private IP networks.

MPLS is increasingly valuable for aggregating different traffic protocols, thus promoting flexibility in the design of IP networks. Freed from the burden of building dedicated ATM or Ethernet networks, carriers can now aggregate traffic types onto one backbone. That enables traffic to be flagged and segmented as QoS-based for monitoring of jitter, latency and other variables, while aggregating non-QoS-sensitive traffic onto separate MPLS clouds.

AT&T, for one, views IP MPLS as a core enabler to QoS in IP. "MPLS is becoming the foundation for both Layer Three IP VPNs and for Layer 2 Ethernet-based services [VPNs]," says Tom Siracusa, a director in AT&T Labs responsible for VPN strategy. He views MPLS as more than just a tactical VPN service, as it is a foundation for AT&T's core, on which an increasing numbers of services ride. "We are building VoIP infrastructure to ride on top of the MPLS core, which is bridging the gap between intranets and the public networks for applications needing ubiquitous access."

One of the biggest advantages of the MPLS core is scale, according to Siracusa. "It enables you to separate service features and definitions at the Edge from the core transport."

Despite the any-to-any "reachability" MPLS can foster, Siracusa concedes it is still necessary for carriers to monitor endpoints for customers. "There are no specific Layer 2 connections between endpoints, so we have built proprietary tools to monitor that for the customer. "Even when we have automatic re-routing around failures and a self-healing network, we still have to do a lot of monitoring with OSS for performance to make sure jitter and delay meet SLA requirements."

He notes that QoS across the backbone is intended to complement traditional capacity planning, which deals with such capabilities as trending of traffic for new customers. "QoS across the backbone deals with sudden traffic spikes or other unexpected events like worm attacks—anomalies that traditional capacity planning does not support immediately," adds Siracusa, noting that QoS makes capacity planning more proactive in nature. He cites protocols like DiffServ in the IP packets or EXP in the label headers across the core as ways to makes sure voice packets are identifiable and go through in an unexpected event. Indeed, sophisticated traffic engineering will be necessary to accommodate the service-ready nature of next-gen networks.

Engineering Issues and Strategies

Policy control at IP and traffic engineering at MPLS are the means for engineering real-time, on-demand applications so that services correlate to available bandwidth. However, engineering converged IP networks is proving to be much more complex than engineering single-purpose networks.

MPLS IP VPNs must have a design that not only supports the performance requirements of very technically demanding services, but must even anticipate those demands, since VoIP and high-quality video distribution require much more stringent control over latency, packet loss and packet jitter than does ordinary Web browsing over an IP network. "You need superior management, because bandwidth is finite, and we can't own all the fiber in the U.K., nor in the world, so overbuilding everywhere as a quick-and-easy means of capacity planning is not enough," says Mick Reeve, group technology officer for British Telecom. "No matter how inexpensive or how much it is a commodity, bandwidth will ultimately be eaten up by new applications."

Even if carriers could overbuild their cores enough, the fact remains that access pipes will have limits, particularly in technologies like DSL.

Reeve sees a need for more development in the management of the MPLS and Ethernet technologies in the industry as a whole. "More standardization is necessary with MPLS, IP and Ethernet, as we need better management of the network for true carrier-class monitoring and resilience," he says.

Detailed initial information about what an MPLS core should look like, what IP-based services will be and what intelligence layers are necessary can be found in the next-generation network (NGN) work of various standards bodies, including ATIS, ITU and ETSI. OSS and equipment manufacturers active in those organizations are trying to avoid the types of management issues posed by traditional IP and Ethernet networks. They recognize that OSSs will have to be more proactive than reactive in guaranteeing the performance of the network, and more capable of deciphering specific application types and their bandwidth demands.

A slew of protocols and technologies are picking up momentum to facilitate new-age engineering support and capacity management, such as Diffserv (Differentiated Services), FIFO, Weighted Fair Queuing (WFQ), Per-Hop Behavior (PHB) and the Resource Reservation Setup Protocol (RSVP).

For example, BT is using policy control and the IETF's transport protocol for differentiated services (Diffserv), as well as MPLS VPNs—all of which give rise to QoS capabilities for different services traveling over its IP network. By using MPLS traffic engineering in the core and MPLS fast reroute to establish MPLS VPNs per service type, BT will be able to assign classes to its traffic types. "That will enable bandwidth to be distributed preferentially," Reeve says, "with Internet services considered to be 'best-effort' services, and real-time voice and video getting end-to-end QoS."

As BT extends capacity by moving to an IP-based network and an MPLS core with Ethernet backhaul—what Reeve calls its "long-term vision in the Layer 2 transport domain"—it hopes to enhance capacity, resilience and security for IP and MPLS. Thus far, real-time, on-demand capacity planning is evolving out of technologies like Diffserv at the IP layer. "We are using MPLS traffic engineering in the core to foster resilience, as well as the ability for 'session-acceptance' control to prevent the occurrence of too many sessions going through access lines," such as home DSL lines trying to download too much video simultaneously, says Reeve.

BellSouth is another aggressive IP player using MPLS traffic engineering. "So far, MPLS RFC 2547 has been valuable in making the best of our Layer 2 and Layer 3 technologies," contends BellSouth's Nimesh Shah, senior director of data product management. He regards MPLS as a "proven technology" in enabling BellSouth to deploy SLAs that back performance promises. "MPLS creates a networking environment with a security equivalent to Layer 2 services [like frame relay and ATM], while at the same time allowing for any-to-any connectivity by utilizing the routing capabilities of a Layer 3 IP infrastructure," he explains.

To have highly programmable QoS through multiservice switches at Layer 3, Shah also is using Diffserv code points and a number of queuing methods within those, such as Class Based-Weighted Fair Queuing (CBWFQ), Low Latency Queuing (LLQ) and Modified Deficit Round Robin (MDRR). PHB and other routing protocols have also been employed by BellSouth to implement QoS levels based on traffic prioritization from the customer premises all the way through to its core network. "That has helped us to ensure hierarchical routing of data," Shah says. Next he will investigate RSVP and other similar performance-aware routing protocols to further optimize the network.

BellSouth also is making extensive use of modeling to manage multiple layers, such as transport, Layer 2 for ATM and frame, and Layer 3. "Traffic engineering is critical to ensure we have sufficient capacity, so we perform modeling in our core to monitor traffic," Shah says. "We do that to ensure sufficient capacity for each node, as part of our relief strategy." To understand the behaviors of the company's access and to predict optimal deployment, Shah says, BellSouth uses simulation tools as well as capacity management statistics and spreadsheets.

MCI's vision is to have "packet-friendly" access—whether on- or off-net through its Converged Packet Access (CPA), an Ethernet environment for employing strict support of SLAs for carrier-grade service to enterprise and government customers.

"In instances where it is not economically feasible to build out fiber, the company will lease broadband and surround the connection with the CPA infrastructure so that it operates the same as if it were part of MCI's network," says Wimmer. "Many Ethernet implementations today are only supportable where optical fiber has been deployed, but we are banking on the idea that the CPA architecture eliminates this constraint by being compatible with any form of physical transport, including wireless, wavelength, fiber or leased transport from third parties."

In combining CPA with OSS, MCI will add capabilities over the next 18 to 24 months to enable near-real-time provisioning of bandwidth and services, as well as customer self-service and reporting capabilities. "Our aim is to facilitate end-to-end automation that allows remote management and troubleshooting so we can reduce provisioning time and costs," says Wimmer. He notes that fully managed services that extend out to the customer premises interface will require embedded signaling in the architecture for rapid and automated provisioning of bandwidth and services. "When coupled with a customer portal," he says, "it will enhance customer management, provisioning and reporting."

One of the keys to CPA is the use of pseudo-wire technology—the IETF standard defining the mapping of many types of payload over a common logical circuit. As a mechanism for transporting and maintaining separation of multiple services over a common infrastructure, pseudo-wire connects customer POPs and edge services networks to provide a logical connection associated with a certain amount of bandwidth for each service. "We believe it will enable us to guarantee slices of bandwidth, as it enables us to ensure that bandwidth committed to certain types of traffic, such as Internet, don't burst into video or VoIP traffic," Wimmer says.

As MCI engineers its network to enable this type of control, computer modeling and performance simulations are becoming prevalent. "We have to understand how the network and protocols scale as we optimize the design for high performance, convergence and capacity. Up-front planning is essential, as we don't want to 'engineer ourselves into a box,'" says Wimmer. "If you start with too narrow a target [such as a single service], and optimize to that target, then the second service you want to add later may require higher performance, greater capacity or even protocols and interfaces that the more narrow design cannot support." Though he contends it is not necessary to invest in deployment of features or capacity before they are actually required, he believes it essential to design an architecture up front that allows their implementation over time.

Making OSS Proactive

As capacity is built out, the need for application-driven capacity management is pushing a need to augment management of QoS at the path and application level. In the long-run, real-time application control, capacity planning and end-to-end session control must come together to prevent overload situations that break the bandwidth pool or damage premium, enhanced-quality services.

Facilitating that will require a maturation in existing OSS architecture. The reactive fault and performance management of the present day will have to inherit more proactive monitoring for capacity management in real-time applications.

"We see a need for off-line capacity management and on-line capacity management in different applications," says Reeve at BT. Off-line management planning means that capacity is measured according to aggregate traffic flows that reflect when capacity is reaching its limit. "Ensuing action," he says, "can take days or weeks." However, he notes, managing real-time traffic capacity requires more of a network-control approach that yields instant results.

That means carriers need a way to decipher the origin and nature of capacity demands. "There have to be separate mechanisms for fielding capacity requests for more fiber from a keyboard management action in the NOC, as opposed to an HTML or dial-in request from an application that needs capacity on-demand" explains Reeve.

He is working with OSS partners Amdocs, Cramer and Micromuse to facilitate constant monitoring of performance on a per-application basis. The goal is to move the IP network from best effort (with capacity managed on an aggregate basis) to on-demand or QoS-centric performance with proactive, real-time capacity planning.

"We want to proactively manage capacity and fault conditions," Reeve says, "which means OSSs have to be more aware of the performance of the network, and more capable of deciphering one application's bandwidth demands over others." Finance systems, e-mail, video and telephone calls, he points out, all have different capacity demands and different priorities.

While establishing real-time monitoring of bandwidth at both Layers 2 and 3, Reeve sees part of the solution in packet inspection systems. "They have to inspect every packet in a network and show how applications are using capacity on the network," he says. Interesting technologies and tools have emerged from Ellacoya Networks, Cloudshield and Sandvine, he says, which "enable us to implement intrusion detection capabilities as part of performance monitoring and security."

The goal is to take information from packet-inspection tools to feed back to operational systems. "Then, there is a chance to see how capacity splits among applications, which is how policy control is spawned," says Reeve. When that occurs, he says, carriers will be capable of policy control at IP and traffic engineering at MPLS.

Increasingly, the boundaries are blurring between real-time capacity management and an "OSS-enabled" long-term planning approach. "That means setting policy on the fly for services to enable application-driven QoS," says Reeve.

'Discovering' Networks

While carriers work to design "just-in-time" capacity models, they are looking to reduce the lag between customer orders and service activation, as well as reduce OpEx by unifying the network planning and engineering processes.

"It's a period of experimentation. Technologies tend to be very specialized and constantly evolving, so there will be some false starts for those pioneering new IP utilization and management technologies," concedes Ross Munro, a former Bell Canada operating solutions engineer who started Comscient Consulting, a firm advising on network and service planning and provisioning solutions. He believes QoS can be related to sound network architectures that minimize vulnerability to failure. Rather than tie success to "recovery of assets," Munro advises carriers to "discover" networks to improve inventory accuracy so as to better simulate network behavior scenarios (increased traffic loads, node failures, etc.) and ensure QoS proactively.

"The best case is for solutions that open the door for the reconciliation or improvement of inventory records, so that existing assets can be used for service demand instead of automatically deploying additional assets," says Munro. In his view it is less attractive for a stranded asset to be physically recovered and redeployed. "This is a trade-off of the expense of the recovery and redeployment versus the capital cost of deploying a new asset," he says. Furthermore, Munro believes the reusability of assets declines with time as networks evolve to newer technologies. "In the final analysis," he says, "much of the stranded asset base may not reusable."

To truly understand and manage network capacity, Munro believes performance management and inventory systems must have strong linkages. "Because the necessary linkages are not typically found in COTS solutions," he says, "carriers typically perform costly customized integrations, or depend on locally administered work-around processes and solutions." Rather, he recommends that carriers use network modeling tools in the network planning phase of IP roll-outs. "That would enable input results to be extracted from performance management," Munro says. "The effectiveness of the network simulations can be verified and improved by using feedback from performance management tools."

Of course, implementing an automated inventory-reconciliation project requires substantial IT infrastructure. For that reason, carriers should seek solutions with proven track records, or with meaningful pilot projects that reveal any missing functionality and unexpected implementation costs. "Be wary of claims by vendors that they can recover, reuse and revive stranded assets if there is no existing proof of such functionality," Munro emphasizes.

Optimizing Utilization

Some emerging solutions are attempting to optimize network capacity by automating network resource planning and engineering, to help carriers gain capacity by reusing their current networks more efficiently and by homing in on specific areas needing equipment and management.

Among those players is VPI Systems, a start-up founded by John Holobinko, who co-founded American LightWave Systems (later acquired by ADC Telecom as its broadband division). Holobinko is promoting "network resource life cycle planning" through tools that use routing algorithms to model predictable transport networks that can see behavior in ring and mesh multi-vendor networks. The company's strengths are its algorithms that define behaviors at each network layer, as well as a common data model across all network layers.

"Traditional tools, such as WANDL, generally deal with a single network layer, such as IP; they do not take into account the real-world impact of the existing network and the Layer 0–2 capacity requirements that a Layer 3 service needs for meeting its QoS and redundancy requirements," says Holobinko. "There were no engineering support systems that interface directly to existing OSS/BSS systems to determine where capacity can be added in multilayer networks, so we designed algorithms that are used as plug-ins to accommodate the unique ways in which different equipment manufacturers implement the same technology."

To manage multiple layers of a network, he explains, one must model behavior for each layer—whether transport, Layer 2 for ATM and frame, or Layer 3. "The carriers must be able to cross-link the data among those layers to analyze the impact that traffic in one layer has on the other," he says. "You have to be able to pass data critical to other layers from any layer. Only then can you determine what capacity, performance and redundancy is required.

"For example, if a carrier is trying to take advantage of an IP/MPLS backbone to reduce costs and improve efficiency, they need to know that they can support the end-to-end performance of ATM, frame relay and other non-services end to end across the network through the IP/MPLS core. It is essential, therefore, to model the end-to-end network capacity requirements of these services across all network layers from Layer 3 to Layer 0."

In circuit-based networks, engineers and planners could be reactive when monitoring capacity, but now carriers need to preplan capacity utilization, because they will lose that one-to-one relationship with the customer and network resources.

"Now that the goal is a smaller number of servers connected over the core network, it further drives the need for traffic engineering," according to Briscoe at Cramer. He believes that IPTV and VoIP will push companies to add QoS to IP cores in convergent networks with traffic engineering technology, such as MPLS traffic engineering through InterServ or Diffserv that manages the general consumption of the core network. "The by-product will be that traffic will be sent through more efficient paths rather than more obvious paths," Briscoe says. "This ensures all the networks capacity is used evenly and reduces bottlenecks seen with these protocols."

Cramer has systems that plan static traffic flows, and then provide an abstract view of services so operators assign equipment and resources to build new MPLS label switch paths. "We already model this underlying technology in our product today," Briscoe says, "which allows us to easily map services into the core for proactive capacity management."

That is different from the past, where the direct customer-to-network connection made thresholds timeslot-based. "Because infrastructure is shared in IP networks, setting thresholds is more complex, because you are offering multiple IP services—each service possessing a unique bandwidth profile," Briscoe says. For that reason, traffic engineering must mature to augment the planning process. "That can only happen by knowing how many services with each bandwidth profile might be necessary in different geographies," he says.

Briscoe notes that there is not a problem of loss of capacity, since utilization of capacity stands at only 15 to 30 percent. "The issue," he says, "is using monitoring tools to passively validate capacity and not having a proactive approach to IP network consumption."

"Carriers have to manage capacity utilization more closely than before," he says. "Without this, it will be very hard to match the demand from customers to the process of planning new investment in the network." Briscoe warns of the potential for spending millions of dollars without any tangible benefit. "Carriers have to ultimately be more controlled in their equipment spending," he says.