Testing the Claims of Test Automation

While IP can facilitate testing by making it easier to develop a common platform around it, the protocol also can make test automation more difficult. Case in point, said Rainge, is that incompatible SIP stacks, questions about the nature of IP traffic flows and optimization, and the inherent openness of IP all drive complexity.

“IP also makes testing more complex because multiple services are converging on top of the common IP core and sometimes a shared IP edge or access infrastructure, Rainge said.

Ajit Sancheti, co-founder and vice president of product management at Mu Dynamics, said service providers are driving the automation efforts of test companies. However, as much as they desire the ease of automation and its subsequent time and cost savings, they may not want to be married to a single vendor — even within an alliance — to get it.

“Some equipment vendors and service providers have their own automation solutions. They are pushing back because they don’t want to be too closely tied with any one company,” Sancheti said. “They want the flexibility to change platforms if they feel the need to.”

Mu Dynamics is part of TesLA, and like Fanfare, it sees no conflict potentially being part of both it and the TAA. “Our goal is to be a part of all these environments because that makes us a component of a complete solution.”

He said it is not clear whether Ixia or Spirent will win this battle of the alliances and he isn’t convinced one has to. “It's all about how good the automation is and how easy it is for any customer like Verizon or AT&T to walk in and use it. And there’s the cost,” Sancheti said.

Mu Dynamics had a Tier 1 cable operator whose three staff security experts were spending only 20 percent to 40 percent of their time doing security testing. After deploying Mu test products that automate that process, they were able to spend 100 percent of their time actually testing. Another customer, SonicWALL, an infrastructure company, claimed a one-month payback on a Mu purchase based on the cost savings from test automation and identifying potential threats earlier in the process.

The cost of automated testing may be obvious, but the cost of not catching problems early is not so obvious. According to the National Institute of Standards and Technology (NIST), a bug found in the coding and unit test stage is five times more expensive than if it had been caught in the architectural design stage. Catching it in the integration and system test stage is 10 times as expensive to fix. Discovering it during a beta test is 15 times as expensive to fix. And when a bug is found in a production network, it is 30 times as expensive to fix than a bug found during the architectural design stage.

Sancheti sees another way for end-to-end automation to be accomplished. He said a company like Mu Dynamics can be a neutral, third-party orchestrator of all the testing.

In its bid to provide proactive service analysis, beginning with security vulnerabilities and extending to service assurance, the company has developed a substantial test automation portfolio. It provides automated testing for its three primary products, a service-level traffic analysis solution, a published vulnerability analysis tool and a denial-of-service assessment tool. Its message is that it eliminates downtime in the network by finding vulnerabilities in the development cycle as well as throughout the product lifecycle.

Whether the best solution for test automation is a fixed alliance of lab and test solution companies, a more flexible and growing version of the same, or an open-source solution, the benefits of automation can’t be ignored. Given the complexity of achieving true automation, it may likely be a combination of all three and then some.

** For more stories on test automation, read the full eBook “Uptime Guarantees Begin with Upfront Quality Assurance,” sponsored by Mu Dynamics, at www.xchangemag.com/mudynamics. **