Metro Ethernet Finally Delivers

As carriers are pressed by the financial community to compensate for lost voice revenues and to counter triple and quad play competition from cable, they have to figure out how to move to multi-service offerings without compromising operational efficiency. Service providers are feeling a definite sense of urgency, as they rush to make up-front capital investment without knowing whether volumes will be high enough to make it all worthwhile.

Yet as carriers push to deliver services to multiple far-flung locations, legacy technologies in their metro area networks (MANs) are showing signs of weakness. Because MANs were meant to accommodate circuit-switched voice traffic, SONET/SDH ring infrastructures lack the dynamic functionality and scalability for handling prodigious data volumes. The temporary fix has been expensive network build-outs, but the complexity of mapping data into rigidly defined time division multiplexed (TDM) channels is making it obvious that those channels do not harmonize with the bursty nature of data traffic. Consequently, carriers have to add fiber or make major infrastructure changes.

The cost and time expended is no longer acceptable, as the majority of the bandwidth remains underutilized—as much as 80 percent in some networks.

Even though the knowledge base and resources for SONET are still substantial, current market forces leave carriers little choice but to move to a more efficient model bandwidth allocation and network management.

Enter Ethernet
“Ethernet is beautiful in its simplicity,” says Andy Mayer, principal systems engineer in Telcordia’s Global Services. “There are so many forms of Ethernet, and many chip sets that are mass produced, which leads to a good economic reason to use it wherever possible.”

Indeed, metro Ethernet for layer 2 transport is finally delivering on its promise of efficient and tailored bandwidth allocation, as well as the flexibility to grow out a network at a substantially lower cost than was possible with legacy technologies.

Although the marriage of optical and Ethernet technologies has long been anticipated as a solution, carriers are just now embracing Ethernet as a transport layer for more than just LANs, such as connecting enterprises in metro and even long-haul networks.

Because devices can be reconfigured for different bandwidth needs without truck rolls, Ethernet might be the enabler for bandwidth-on-demand applications in the not too distant future. The technology’s 1 Mbps increments enable more efficient bandwidth configuration than was possible with OC3 and other increments of SONET/SDH data transmissions.

According to a Metro Ethernet Forum (MEF) study based on a three-year build-out of a medium-sized metro area network, a network based on optical Ethernet design principles cost 49 percent less to operate than a legacy SONET/SDH-based network. Those operating costs are also kept to a minimum because carriers can use combinations of multiple 10 Gbps lines to boost bandwidth, or they can take advantage of emerging 40 or 100 Gbps services.

For now, anything over 10 Mbps is overkill for most enterprises, but large financial exchanges and other high-volume entities will ultimately utilize such tremendous trunking capabilities.

Maturing Standards
There are three types of Ethernet, each using different optical transport technology, and each promising different levels of resiliency, flexibility and scalability (see sidebar, “Permutations of Ethernet”).

“Driven by VoIP, and progress with MPLS, Ethernet may trump ATM, SONET or circuit switching,” says Stu Feeser, instructor for global knowledge and president of Alta3 Research Inc. He notes that maturation of standards have really bolstered Ethernet’s viability as a technology for not only short distance, but longer ones as well.

“Indeed, 802.11 type standards, such as the IEEE VLAN standard 802.1Q, enable Ethernet to be carried over longer distances, so that far-flung enterprise offices can be connected more economically over longer distances,” says Eric Barrett, director of product management for Masergy Communications Inc., whose nCONTROL VPLS product allows customers to securely extend Ethernet LANs throughout WANs. Its Intelligent Transport Service—a means of terminating serial connections to foster multiple VLAN connections—converts to Ethernet the T1 or DS3 lines at customer sites, thus simplifying networks by eliminating complex router management. “Ethernet makes DS3 or T1 or native Ethernet all look the same,” Barrett says. “That simplifies convergence of Internet for IP, frame for connections, and ISDN video and voice can evolve to more advanced IP solutions as they phase out frame relay.”

In addition to VLAN standards, class of service (CoS) designations are beginning to emerge, as the IEEE’s 802.1P standard marks frames with priority levels. That standard defines the 802.1Q header for designating services classes, such as “best effort,” platinum, gold or silver.

Foreseeing synergies and possessing a desire to avoid duplicate work, the MEF and MPLS Forum joined forces in 2003 to accelerate the adoption of optical Ethernet in metro networks worldwide. Since then, both organizations have worked to help link sites with Ethernet services to sites with legacy services. In November of last year, the MFA Forum voted on a specification that would guide interworking of Ethernet traffic over MPLS with a variety of other traffic types such as ATM, frame relay, High-level Data Link Control and TDM. The specification maps all of those technologies over MPLS—the aim of which is to form a “glue” for IETF standards, such as frame, ATM and PPP.

“We are pushing MPLS as close to the customer premises as possible, because MPLS using Ethernet gives end-to-end QoS and critical OEM functions for troubleshooting,” says Chad Holliday, director of product marketing for the IP division at Alcatel. The company’s 7250 service access switch extends MPLS to the customer premises for enhanced service manageability and scalability.

With advances in devices, traffic coming over an Ethernet transport layer (layer 2 switching) is more readily converted from ATM or frame into IP right where it enters the network. “That means to customers, the service can still look like a frame relay or ATM service, or like Internet access or TDM services. Everything is converted into Ethernet and transported,” says John Holobinko, managing director of consultancy Northington Systems, which specializes in telecom planning and strategy. “At the core of the network, everything goes through Ethernet switches and is put over MPLS or whatever the core is made up of,” he says. “The customer doesn’t even know that Ethernet is being used in the metro access.”

For that reason, Ethernet is being phased in as other networks like SONET, ATM and frame relay are phased out.

Legacy Technology Will Persist
This is not to say that ATM or frame are going away. Carriers have a huge investment and satisfied customers on those networks.

“As enterprise customers upgrade their infrastructure, they are changing out private-line ATM or frame for better speeds,” says Craig Drinkhall, senior VP of product development and engineering for service provider TelCove. “Going from T1 to DS3 was a big jump, but going from 10 Megs to 1 Gig will be a huge upgrade.”

Carriers will continue to run ATM and frame relay until it’s no longer economically feasible.

“Despite developments, there remains lag time, as we have to wait on equipment manufacturers to build core and customer equipment for delivering QoS capabilities,” says Donna Silvey, senior manager in product marketing for BellSouth. The carrier is focusing on fostering redundancy and failover capabilities by using SONET and DWDM in its Ethernet offerings, and testing Ethernet over copper in areas not served by fiber.

“There’s a lot of hype around Ethernet enabling bandwidth on demand in the future, but we don’t see a demand for that. Rather, customers want to know there is redundancy and failover should another Katrina hit,” says Silvey. “What mattered to our customers after the hurricane was the quick turn-ups we had because of our carrier-class metro Ethernet deployment. Tulane University was thrilled with the quality of the equipment, procedures and protocols, because the servers all started right up, even after they turned down the data center.”

As companies call for carrier-class performance and low latency, demands on current routers may take their toll. “Current routers will not be sufficient, as they cannot handle very high volumes of services while maintaining QoS,” Holobinko says. “In labs where there are gigabytes of data simultaneously traveling over networks, routers fail because of the size of the buffers and inefficiencies in current operating systems.” Provisioning might be simpler with routers, he concedes, “but today’s routers don’t have the internal performance capabilities for buffering, plus there are limitations in the operating systems to support robust QoS in multiple services simultaneously.”

The Pseudowire Alternative
One way manufacturers help their customers get around those limitations is with the use of pseudowire—an alternative to VLANs and pure Ethernet. With pseudowires, companies extend IP MPLS through access networks to customer premises. More and more, carriers are offering pseudowire technology as part of their growing metro Ethernet service offerings as a means to foster integrated support for existing TDM services over the same Ethernet transport network.

"Pseudowire helps give end-to-end increases in bandwidth where upgrades are necessary. The challenge with frame or DWDM rings is pushing the request for bandwidth to the core and access networks. Coordination is a challenge when there are multiple OSSs supporting the core and others for the access," says Ian Roy, VP of product management for Netcracker, whose service management capabilities are designed to help carriers consolidate OSS. "Many providers have the MPLS core with Metro Ethernet access and pseudowire over the core, so we put in inventory management and service provisioning for an end-to-end view across the cloud--customer prem to customer prem."

Because so many customers still require T1 and T3 interfaces to support legacy TDM equipment, they cannot support existing and emerging IP applications through pure Ethernet.

For example, the wholesale side offers opportunities to provide wireless carriers with big bandwidth to cell towers, as they try to reduce costs for backhauling traffic from cell towers to mobile switching centers (MSCs). They still have to contend with technology at cell towers, meaning 2G and T1 circuits for backhaul. Through pseudowires that transport TDM traffic across the same Ethernet link, cellular carriers have been gaining access to high-bandwidth, low-latency Ethernet connections for tower-to-MSC connectivity.

Pseudowire also is prevalent among businesses that used to have internal Ethernet networks. The pseudowire with Ethernet extensions to VPNs is used often; however, some say that is not a sustainable business model. Because using Ethernet in the metro involves fewer elements to manage, some say QoS is easier to control when the problems with routers are removed.

“It’s not practical in the long run to have IP all the way to the customer prem with pseudowire,” contends Holobinko at Northington Systems, who believes using pseudowires for all existing routers will be difficult. “The technology just isn’t there yet, but real-time services with strict QoS requirements around latency and jitter are coming into play.”

He acknowledges that combining pseudowire and MPLS through the network makes for more dynamic provisioning, “but defining a pseudowire throughout a network becomes very complicated, as you have to define it from the customer, to the network edge, through the network to the core,” with a lot of end-to-end mapping necessary.

Despite shortcomings, pseudowire is considered by some to be a key enabler for point-to-point VPNs, or point-to-multipoint VPLS services. “Usually those pushing pseudowire all the way to the access layer to the customer are early adopters. That means the challenge is with education rather than routing technologies,” says Cisco’s Mark Milinkovich, director of product marketing for service provider routing and switching. “Companies like Comcast, Time Warner, Cox and Verizon use Cisco’s 7600 class machines for carrier Ethernet services, and they are able to use operational administration and management functions of multiple services over Ethernet interfaces. It comes down to experience sometimes.”

Cisco has been working to improve the management of Ethernet infrastructure and MPLS over Ethernet to drive multi-service architectures. It has aligned itself with the MEF’s initiatives, such as supporting ping and MPLS multitask traceroute commands, as well as IP SLAs over LSP (label switch path).

Troubleshooting and Control
“One of the core issues carriers want to resolve is trouble diagnostics and correction,” according to Milinkovich. “With millions of subscribers and thousands of Ethernet ports, it is very difficult to identify the root of a problem, whether it lies with a fiber cut, or line card module, or return path.” Cisco’s recently announced software tool, MPLS Diagnostic Expert, is designed to isolate problems through rules-based reasoning and intelligence such as traceroute, ping and health monitor functions for troubleshooting accessibility of devices, status of a host, and packet loss and impact to users.

Cisco also launched its Cisco Service Control Point (CSCP) for dynamic policy control and service environment control. As part of its IP NGN initiative, CSCP offers a service exchange framework (SEF) intended to extend IMS by attaching IMS functionality to the SIP control plane. “There [is] a wealth of applications, like IPTV, video and director services that are not SIP-enabled, so SEF is designed to give control over policy and activation, leveraging the intelligence of the network for dynamic services to the carrier’s customer base,” says Milinkovich.

The intelligence of the network is important in matching customer applications to the right technology for delivering service quality. “You can’t have the same technology at every location,” says Time Warner Telecom’s Mike Rouleau, senior VP of business development and strategy. “Ethernet is the cornerstone for data and IP strategy, as it acts as a point of convergence of networks for triple play. Some companies need point-to-point, others multipoint, some need switches, and some need SONET. You have to be able to tailor your Ethernet solution to the customers’ needs.” Among the company’s 5,750 connected buildings, he says, “we have wavelength-based services for point-to-point, and switch-based for multiple points and VLANs, and Ethernet over SONET for faster restoration speeds.”

As carriers expand the breadth of Ethernet offerings, they will get closer to the sexier application of Ethernet for bandwidth on demand (BoD) services.

“To be able to throttle bandwidth according to your needs, carriers have to get to the point of enabling self-provisioning. That would happen through Web portals that reflect bandwidth availability and usage in each part of the enterprise,” says MetaSolv’s Stephan Pelletier, director of product marketing for IP and Ethernet services. “You’d want a really GUI front end on which a person clicks a button to ratchet bandwidth up or down.”

For that to happen, provisioning environments will have to be very tightly integrated with billing. “The billing has to know immediately when someone has made a change to bandwidth,” says Pelletier. That means the provisioning systems have to tie to Web portals so that orders are collected and the networks are checked to ensure bandwidth is there. And then activation has to contact billing so that CDRs are updated. Then, billing has to send messages back to the Web portal to indicate to the customer whether or not the service can be activated.”

“That will require heavy-duty provisioning and heavy workforce involvement at first. But once the access path is up, only soft changes are needed to turn bandwidth up or down,” says Leonard Sheehan, director of provisioning solutions for MetaSolv. For that to be possible, he says, QoS has to be at the core of configuration strategies. “QoS has to be front and center across layer 2 Ethernet and metro,” Sheehan says, “and the connection to the IP core network as well.”