Service Assurance on Maturing VoIP Architectures

Service providers are learning the hard way that VoIP networks have more components to track and are much tougher to manage than plain old telephone service. The circuit-switched world, for instance, has had decades to automate diagnostic systems to chase down weaknesses in the network. Once a customer complaint about a weak signal or static reaches the phone company, technicians easily conduct a metal test using a probe to determine where the copper or other metal used in POTS lines is degrading.

"The reactive nature of voice testing is nothing new. That's the way it works today," says Adan Pope, principal OSS architect at Cramer Systems.

However, this reactive strategy won't work for entrepreneurs launching VoIP networks or those providers completing triple play launches that include packet voice services. One goal of VoIP owners includes the ability to fix network problems before they affect subscribers.

VoIP networks, though, present some daunting obstacles to that goal. The voice call or data traverses several network elements before it hits the long haul. This also means that the call is quickly out of sight of network managers; the media gateway doesn't retain routing information, so the network administrator's view is obstructed, making service assurance very complex.

So which of those network elements should be the focus of scrutiny? While softswitches provide services for call servers or media gateways, it's the media gateways that provide the VoIP over the data infrastructure, Pope says. The call server is not really burdened with the transport at all. It doesn't have to be burdened with how the packets are being carried.

"From a transport perspective, [the call server] is disconnected from the transport. If you're trying to figure out if this call has been routed over the networks, from the service assurance perspective it's impossible without probes," Pope says.

Using probes, VoIP system administrators pull data from several elements on the network and compile the information on a server linked to the OSS. Once the data is centralized, they can narrow the problem to its source.

More Places for Errors to Hide

For a VoIP call, finding the point on the network that affects quality is a problem.

"Softswitches may be in one or two places; then you've got the session controller and gateway, two more modules, then more hardware in the trunking or media gateway. Add in the signaling gateway and now you have four boxes, with all the connections among them. Not one of them alone can tell you the whole story," says Grant Lenahan, executive director of wireless mobility for Telcordia Technologies.

It is this distributive nature of VoIP networks that masks problems, allowing several small problems to remain hidden throughout the network. System administrators may think everything is running smoothly on the network, while half a dozen immeasurable problems are creating jitter or latency in calls. Individually, however, none of the problems are strong enough to cross thresholds and set off alarms. "You add data into the mix, and you have what we call 'additive problems,'" Lenahan says. "You could have a performance problem coming in that isn't bad enough on its own to trip an alarm. But if you add it to another problem, the two combined together will trigger an alarm."

In short, service assurance changes from looking at the health of the large box (a Class 5 switch) to investigating the end-to-end network and all the components that contribute to its health.

And as if wireline VoIP networks weren't already complicated enough, add in a link to wireless data networks, which are famous for hidden issues that cause jitter and latency. Now you've also got a sensitive translation between the GPRS and the SSGN (the point where wireless signals hit wireline networks). The complexity is immense.

"IP networks have much more complicated performance characteristics than trunk networks," Lenahan says. "Trunk networking is binary—it was working or it wasn't working at all. But in an IP network, there are many shades of gray that generally reveal themselves in latency." Telcordia and others have been working to tackle the complexity of wireless data networks, which perform much more poorly than fiber networks. "You can have dozens of network elements contributing to the end-to-end performance of those wireless packet networks," Lenahan says. "In addition to all the routers, gateways, and soft switches, you have all the GPRS components to worry about."

Finding the Problems

Yet another characteristic of VoIP architecture challenges technicians trying to pin down problems on the network and fix them. Unlike circuit-switched networks, where the phone line is tied to the customer's address, VoIP administrators have to combine several sets of data to pinpoint call problems.

"The challenge is to be able to determine the location of the VoIP subscriber and crack an identifier to find the phone number," says Stephan Pracht, product manager at Agilent Technologies. "It's a location-less system, much like mobile networks."

VoIP providers don't have the luxury of the Loop Facility Assignment and Control System (LFACS) database the POTS world uses to look up customer address/line assignment information. "Now with triple play, we have voice, Internet and TV all over the same line. LFACS doesn't fit the bill for complicated converged billing," says Pope at Cramer Systems. "It also creates blind spots in provisioning and inventory efforts." When such a line is located, latency can be measured on the VoIP network using a toll-grade probe. With such a device, technicians can do an "FTP roundtrip" measurement, determining latency by sending a signal from the customer's premises to the central office, and measuring the time it takes the signal to return.

Pracht recommends that VoIP providers understand the baseline quality of each part of their network, so potentially weak areas can be watched more closely. Agilent's product, called NetExpert, sets baseline measurements and alarm thresholds. By creating a span of key indicators on the network, service providers can create a baseline performance threshold for measuring degradation.

For instance, by seeing how long it takes for the VoIP network to perform call setup under ideal conditions, a VoIP provider can use that time as a baseline for judging other call set-up times. If Agilent's SS7 monitoring system records a 15-second call setup, and the baseline is 2 seconds, then something is obviously wrong. The service provider would implement an alarm at the call setup and set the threshold at 4 seconds, so when it slows, the staff in the NOC knows there's a problem. Baselines are set up at other parts of the network to alert network administrators that there's delay or jitter.

Finding the Caller

Agilent also markets its NGN Analysis System for managing VoIP networks and ensuring the quality of voice and data traffic. Pracht describes how a service provider might locate a hypothetical subscriber making a VoIP call from a London hotel room. The subject plugs his SIP phone into the hotel's broadband hookup owned by BT or some U.K. cable broadband provider. "Physically, we don't know where they are," says Paul Capozzoli, product manager for OSS for Agilent.

"One needs to go a step further, beyond looking at the IP address and phone number, to crack the signaling information," he says. "The IP address is not fixed; they're handled by a DHCP server." As its name implies, the DHCP "dynamically" assigns IP addresses randomly as one call ends and the IP address for that call is discarded. The DHCP server sends up another IP address when a second VoIP subscriber launches a call. This is repeated so there is always an IP address available for VoIP calls.

"You need to have the lookup. You have to tie yourself to the DHCP database—that's where you get the mapping," Pracht says. "There's not even a base station identifier like there is in mobile networks."

The next step is to determine where the call degradation is occurring. For discussion purposes, we'll call the VoIP provider "ViPcall."

"One of the first questions the service provider wants to ask is, 'Is it on my network, or someone else's?" Pracht says. "By monitoring both the VoIP and SS7 data and putting all those pieces of the call together [and linking into its DHCP database], we can locate the IP address for the specific signaling gateway or media gateway the call is coming through."

ViPcall, using Agilent's system, tries to determine the point at which its customer's call hits its home network, also known as the demarcation point. ViPcall is interested only in the carrier that hands off the VoIP call to its network; previous carriers in the chain don't affect ViPcall's network. Once it determines that the IXC handed over the call at the demarcation point, ViPcall's probes go to work, reporting on whether latency or jitter was affecting the call before the IXC handed it over. If the call was fine when the IXC handed it off, ViPCall searches its own network to see if routers, softswitches or media gateways are malfunctioning, degrading network performance.

SLAs: Proving Who Is Right

Which brings up another important aspect of service assurance: How do QoS problems fit into SLA management when SLA performance is at stake? Telcordia's product, called ServiceDirector, performs service quality management as well as SLA management, Lenahan says. When a service provider's network interconnects with others, cooperation between players may be hard to come by.

"They just want to understand if it's their network or someone else's," Lenahan says. "They have SLAs with other companies, and what bothers them the most is they don't know day to day what to fix. It's a perfect example of what we're seeing." Products like ServiceDirector can map all the network performance data end-to-end, Lenahan says, and if one wants to know how his VoIP network is performing, ServiceDirector can identify every component that contributes to the network's health and map it out using algorithms that translate into color codes: green, orange and red, much like a stop light. "And you can tell if you have a problem and its underlying cause," Lenahan says. Armed with customized reports, the service provider can determine fault and decide whether to provide credits or other refunds on the customer's bill.

Not only that, but as in the circuit-switched world, the VoIP provider may have to gather proof when confronting an interconnection partner whose traffic is hurting your network's performance.

IP and VoIP networks are closed to the prying eyes of interconnect partners; one reason is the presence of IP network firewalls. Though they work well for security, they hinder the performance of VoIP activity, which is a complex set of communications designed for far more fluid interaction.

"When IP providers had beta networks, they used firewalls to protect them," Capozzoli says. "But in VoIP, firewalls don't work because they're static [obstacles] that block traffic." In VoIP networking, a port is opened based on the signaling request. "There's the signaling that sets up requests and handles the negotiation between the two endpoints. For instance, there's a request that port 255 be opened in the firewall; seeing the signaling, the voice gets through, but closes right away after the call," he says.

Session border controllers are another brick in the wall between interconnect partners. These increasingly popular devices, which sit on the edge of the network, are designed to make the networks more secure, but they prevent interconnect partners from seeing into protected networks.

"The carriers want to make sure the quality in their networks is up to what they have in SLAs," Agilent's Pracht says. The way carriers see it, giving information at the hand-off point is OK, but they're not going to supply their interconnect partners with information on the entire path of a customer's call.

"There's a clear expectation set when they engage in that contract," Pracht says. "Cooperation should mean that carriers actually provide visibility of the quality they have in their networks, but no carriers we have talked to have agreed to provide that much visibility."

But it's no good to have interconnection agreements or SLAs with partners that consistently send calls gnarled by latency and jitter. After all, someone has to give the customers a refund for dropped calls, not to mention taking the pain from losing customers who flee because of poor call performance.

"Which carrier do I really want to have peering with?" Pracht asks. "It can save me millions of dollars just having the right peering partner."

Enterprises Seek 'Tailored QoS'

Synchronoss Technologies provides OSS software to manage QoS, as well as trouble-ticketing, provisioning, order entry and the other staples of the OSS. It has been developing ways to measure and provide high QoS for service provider customers who want to set aside quality bandwidth for executive conference calls and other "must succeed" communication events.

Ensuring service quality during hundreds of such requests is difficult, but service providers are finding ways to do it. So service assurance means more than just having a healthy network overall; it also means differentiating service assurance levels at different times of the day.

"The enterprise customer is deploying business services, and service providers are standing up to support that QoS. It wasn't too long ago that all they could provide was best-effort service," says Stephen Waldis, Synchronoss CEO. "Now they're like air traffic controllers. They'll get you within these ranges in this time period."

If service providers could only consistently provide best-effort service 18 months ago, what's improved to today's reality, where VoIP providers are more successful providing higher classes of service? "The infrastructure is getting better," Waldis says. "Providers are getting more comfortable with their abilities; they know there are [service assurance] vendors out there they can rely on."

There's a growing demand from enterprises, such as banks, credit card companies and other financial institutions that want to reserve gold-level QoS at certain times each month. It may be a time during which a credit card company transmits critical financial data or executes an international conference call between executives.

"Those kinds of call quality requests drive a lot of innovation on our end, especially around the ability to order this event," Waldis says. "We've been working on our software to handle such requests so we can provision that, as well as track and bill for it." He says the number of enterprises seeking QoS bundles has grown to about 10–15 percent of its 211 enterprise customers. "It's up about 2 percent from a year ago."

Vonage: Adapter Aids in Locating Calls

Vonage makes a point of not providing the last mile; it doesn't own its network per se. It gets its calls out and into the world through peering agreements with RBOCs and other carriers, including international carriers.

"The agreements we have are carrier-standard SLAs that ensure that we have the proper bandwidth at the peering points," says Michael Tribolet, executive vice president of operations for Vonage. Though Vonage doesn't own the last mile, it owns other elements such as calling gateways, IBM servers for application management and other hardware.

Vonage also uses SS7 probes to pull performance and customer identifying data from its network; its OSS was built in-house under the guidance of CEO Jeff Citron, who helped found Vonage after a stint designing back-office systems for an online stock-trading firm. "We know where the call originates, because we track it from the IP addresses that are mapped to the subscriber's home phone," Tribolet says. "Once the customer's [IP phone] is plugged in, a Linksys analog telephone adapter Vonage supplies its customers emits information across the call path that tells us where the calling device is located."

The call location ability makes it possible for Vonage to manage one of its most popular services that deploys "virtual numbering." The service works well for businesses that operate in a large metropolitan area with multiple NPA-NXX exchanges. A plumber with an office in San Francisco, for instance, can order a second number for about $4.99 that makes it look as if he has a second office across the bay in Oakland. Residents in Oakland looking in the phone book for a local plumber call the virtual number, believing the plumber to be in the neighborhood. Subscribers can also order international phone numbers, which cuts down on long-distance charges, too. Vonage has to be able to determine where those calls actually originate for both service assurance and customer service reasons. The analog telephone adapter lets them establish where those calls originate.

"We 'architected' our back office so the customer care rep could see all device properties from one console," Tribolet says, which lets technicians fine-tune the network. "The rep can change CODECs' firmware versions, audio gain settings, as well as other factors that affect QoS."

Where It Is All Headed

As Synchronoss CEO Waldis has pointed out, VoIP network managers have improved in the past 18 months to where they can take orders from enterprise customers that want to set aside gold-standard specific days of the month, with top-notch broadband service at pre-determined hours on specific dates a month or more in advance. It's a mark of growing maturity in the technology that mastery of VoIP traffic may be at hand. The signs are that VoIP providers are getting a handle on the causes of degraded service, which lets them spend time on other goals, such as designing those neat new services for customers.

The winners in the VoIP space will have the best quality control mechanisms in place. That will make them the most reliable VoIP providers, and therefore the ones with the most subscribers.

"VoIP is not only a new network. In the age of economic status, the VoIP provider that keeps things running smoothly all the time will have the advantage," Waldis says. "It's just like the way trains or the airline industry has to work. It's not until air traffic control systems got going correctly did the public see airline travel as a viable way of life."