The world of software development traditionally was a clandestine one, where developers in the bowels of their institution were entrenched in writing thousands of lines of code to meet ever-changing business requirements.
That isolated environment is increasingly giving way to open, collaborative development involving tens of thousands of engineers and developers around the globe through the worldwide open source community. Through this ubiquitous Web-based community, end-users' needs are posted, and programmers—hungry for recognition or just the betterment of technology as a whole—write an application, after which an army of individuals rewrites and debugs it.
Open source software's potential for enabling rapid development of IP applications could be important, as the Internet drives new services in the fixed and mobile worlds. Whether MySQL, Apache, Open Forge, GNU, Linux or any number of open source projects—the open APIs by which component architectures talk may lay the groundwork for interoperability among applications.
How and If Open Source
While Sprint's CTO was waxing poetic on a panel at Linux World, and AT&T has stated that open source development helps ease the technology burden and improve infrastructure quality, most carriers are reluctant to talk publicly about adopting an open source approach. One reason might be the publicity around the SCO/IBM lawsuit (see Standards Watch, pg. 34); another might be fear of giving up competitive advantage.
So far, it has been the switching equipment vendors to first take the plunge. As companies like Cisco, Nortel, Lucent, Alcatel, Nokia, Avaya and other leaders in their spaces work on Linux, the infrastructural nitty-gritty will sort itself out, and carriers may then make a move toward open source software for their back end. While specific projects aren't yet known in the mainstream, there are constant rumblings, as most of the Tier 1s have either inquired about support from OSS/BSS vendors, or joined the Open Source Development Lab (OSDL, at http://www.osdl.org) or other similar forums to explore how to link together third-party modules for developing back-office packages. Also see sidebar "Open Source for Telecom" for information on initiatives in the billing space.
Thus far, innovations have fallen into three categories. First, development tools (such as JUnit and Eclipse) are used by a sophisticated user community of developers. Second is embedded open source software (Perl, JFlex, Jackyl), which, transparent to users, is embedded in products they buy and guaranteed in SLAs that cover all the software, no matter what its source. Third is open source infrastructure (operating systems, such as Linux), which competes with proprietary operating systems, mostly at the low end but becoming more common in multi-CPU environments.
Multiprocessor scalability has been the most talked-about issue, which has led to three Linux initiatives: one involves scalability, another is carrier-grade Linux, and the last is desktop Linux. Of the three, carrier-grade Linux has attracted the most attention in terms of its potential for the telecom space.
Preparing Linux for Prime Time
The strongest area of Linux penetration thus far has been carrier-grade Linux. This effort centers on tailoring Linux to carrier environments that are provided to platform manufacturers so they can establish APIs around fault response, availability, scalability and so on. Carrier-grade Linux is not a standard but a foundation for manufacturers to show conformance through the "registration" process. OSDL members can register through either the technical or marketing working groups. Then, there are two grass-roots mechanisms for gathering requirements: a Linux Users Advisory Council that crosses international boundaries (Latin America, Asia, Europe, United States), and an ISV forum where independent vendors are invited to port Linux and share requirements with the working groups.
"Thirty-two equipment vendors and service providers are signed up to participate in the creation of requirements for carrier-grade Linux," says Sam Greenblatt, senior vice president and chief architect of Computer Associate's Linux Technology Group, as well as a board member of the OSDL. He notes that carriers such as NTT DoCoMo, Verizon, Sprint, MCI and AT&T are participating in the effort in one way or another.
Now in version 2.0., the biggest network equipment providers (NEPs) have been releasing and enhancing carrier-grade compliant platforms that adhere to specific requirements dictated by the Linux kernel. "For example, every Cisco network appliance sold sits on Linux," according to Bill Weinberg, open source architecture specialist for the OSDL, which is essentially the "center of gravity" for the Linux movement. "There usually exists a lag between back-office equipment and that for switching," he acknowledges, "but that lag is closing quickly."
On the embedded Linux side, companies like MontaVista, TimeSys, Novell, Nokia, Alcatel and NEC, not to mention Suza and Red Hat on the enterprise side, are all prototyping Linux for softswitches and application servers for next-generation network services. These companies will combine efforts with IBM under OSDL to roll out products in volume by Q4 of this year. Embedded Linux is expected to proliferate as vendors build carrier-class switches, media gateways, and CORE and EDGE equipment on top of carrier-grade Linux implementations.
To tailor equipment for telecommunications networks, development of carrier-grade Linux is raising some eyebrows.
"We believe version 2.0 of carrier-grade Linux offers a secure architecture that offers very high performance with a natively load balancing and high availability mechanism," says Herbert Mittermayr, director of strategic and operational marketing at Alcatel Mobile Solutions. In 2003 the company announced a partnership with MontaVista Software for its Linux Carrier Grade Edition as well as the product description for the Alcatel 5301 Proxy Platform. Mittermayr contends that carrier-grade Linux has been powerful in driving the development of Alcatel's proxy platform, as Linux has enabled support of key proxy applications such as protocol gateways (WAP, SMS, MMS, PTT), network optimizations and adaptations, user identification, authentication and access control, network information sharing with Web services (location, QOS, capacity), content adaptation and transformations.
"We are a huge advocate of Linux for telecom and next-gen networks; what you see now is only the tip of the iceberg in terms of what will be coming out in the next year," says Doug Dreyer, IBM Global Services director of Linux strategy and business development. Recently, IBM Corp. at LinuxWorld announced that it is contributing the source code for the Java-based Cloudscape database (which it acquired in 2001, when it purchased Informix Software) to Apache Software, an open source foundation.
With the vision of carrier-grade open frameworks cobbled with tools like Parlay, SIP and the like, vendors could ultimately create a framework with tremendous flexibility. Currently, IBM and a dozen or so NEPs and service providers are defining and building a proof of concept using open standards in a framework for next-generation services. "We want to demonstrate how service providers and operators should go about specifying requirements in their RFPs for PTT, IP Centrix and other next-gen services," says Bruce Anthony, CTO for IBM's Blade Strategy Group.
That desire, in August, yielded the Open Communication Architecture Forum, out of which standards and solution maps will be created for the telecom industry. The forum will publish its first set of requirements in Q1 of next year.
Another ecosystem of partners will try to have a "bake-off" at Supercomm next year involving billing, signaling and switching vendors in the Service Availability Forum (http://www.saforum.org).
Most companies in the bake-off build carrier-grade Linux platforms in one of two ways: the embedded companies develop from the blade up, and the enterprise folks from the cluster down.
With the cluster-down approach, usually geared to ISP-type failover for Web servers, the focus is more on scalability and hosting. The bottom-up approach focuses on failover speeds and policies so that time-sensitive services can rely on blades that instantly reboot.
OSS/BSS Buy-In
For companies in customer provisioning, fulfillment and billing to increasingly use open source software, it must be able to handle mission-critical applications. Currently billing, CRM and mediation still rely on off-the-shelf systems; however, some believe Linux will be ready if and when it can outperform Solaris or UX. "More and more, we see some pretty quick evolutions taking place. We were adamant not too long ago about the performance threading of Solaris or Intel over Linux; however, the new Linux threading model came out and it proved faster than Solaris, so we went with Linux for that development," admits Eric Burger, vice president/ and CTO for next-generation communications at Brooktrout Technology. The company creates environments on the media processing and call control side for interfacing applications into SS7 networks for companies like Avaya and Nortel.
Most vendors do not anticipate a problem in porting to Linux, as IBM offers heavy support of Linux, as does Oracle, HP and Sun. Because most infrastructure in telecom is built with vertically integrated proprietary platforms around Unix—an open ecosystem comparable to Linux—migrating code is supposed to be straightforward.
Until system management issues are hashed out for managing myriads of smaller boxes, the NOCs will still be based on monolithic systems. At most operators, the NOC has a lot of influence in system selection, so if mediation or billing wants to try Linux, they will have to prove to the NOC that failover is adequate.
Right now, there is a lag in finding reliable hardware to run Linux for big servers. PCs aren't reliable, so most carriers will stick with the native operating systems on their Sun and HP machines. But because applications and compilation systems based on Linux are Unix-compatible, that could change in a couple of years.
"We have seen an increasing amount of customers asking if we support Linux," says Openet CTO Joe Hogan, who has used GNU open source for development of its debugger and compiler. Openet has moved to develop some of its software partly on Linux; however, it still ships on HP and Sun. "When we sell mediation and charging platforms, we have to target Sun and HP, as well as IBM," Hogan concedes, quickly adding that change is imminent. "We haven't seen mediation or charging running on Linux, but large operators are starting to ask about it," he says, noting that his development group has already worked on it in a lab environment. "We even have a large equipment manufacturer asking us to do software on Linux, which means down the road, hardware companies may look at embedding Linux."
While Sun and HP are still in a dogfight to dominate on large implementations (100 CPUs or more), they are still deeply investing in their operating systems. Linux, for now, is more viable as an OS for Tier 2s and 3s, where systems with eight, 64, or 125 CPUs can be deployed more cost-effectively than multimillion-dollar Unix systems. That means small carriers or perhaps departments of Tier 1s will dabble with Linux first.
The IBMs, HPs and Suns have plans for making Linux scalable as a standard infrastructure that will compete with Microsoft, the de facto standard. It will be a choice between Microsoft or the open source community," says Mark Nicholson, CTO of Syndesis, whose engineers have spent time in Linux groups within HP and IBM. "You have to have a long-term vision about where you are going and how you will combine hardware, middleware and vertical applications." Nicholson thinks the structured philosophies of the IBMs, Vitrias and Ciscos of the world will provide for a "good marriage."
Intec has a couple of production installations running on Linux for testing new services or performing proofs of concept for business practices. "While no one is asking for it now, we think that will definitely change with all the work going into making it scalable and open," says Intec's Rick Woods, vice president for product management and business development. "All our tests indicate Linux is ready for prime time, so it's just a matter of time before someone adopts it in a big way."
He notes that ISPs and other industries have accepted Linux as mainstream and acceptable for large-scale processes. "Every Google search goes through a Linux engine, and they handle an impressive number of searches," says Woods, who has used Linux for years in the original code and testing of code in Intec products.
"When you need a system with high availability, it's the hardware that has the significant cost, and then the application, then the OS. Of the three, the OS is the cheapest component, so substituting HP or Sun with Linux on the OS side only saves a small amount of money. That won't be the driver," says Woods. "What will drive it will be the fact that Linux will be supported by every developer in the world, so you can pick up tremendous amounts of ideas and experience from open source—not to mention freedom."
Freedom to Choose
A main impetus for the open source movement is to attain independence from hardware vendors over decisions to upgrade hardware and software, as well as planning for long-term interoperability with open assets—important for carriers that ultimately want to move away from large machines toward grid computing environments.
"For years, the vendors dictated that you'd have to upgrade software on their [HP, Sun, IBM, Microsoft's] clock, or every time they'd change their OS," says Dominique Le Foll, chief architect for Acterna's SAS (Service Assurance Solutions) group, explaining that upgrades to Solaris, UX or AIX would force engineers to recompile millions of lines of code to update compilers and software.
Four years ago, Le Foll decided to try and break away by making a move to an open source approach in Acterna's OSS and Test OS solution. "We were emerging from Chapter 11," he says, "so we focused on reuse rather than purchasing for a new DSL product we were creating for France Telecom." Le Foll cited 15 points of reuse and first addressed performance issues: "Our Sybase database needed to improve performance from a daily rate of 100,000 alarms in collections devices to 1.5 million per day." Since Sybase couldn't do it, Le Foll contacted Oracle, which guaranteed a 20 percent increase in performance. "That was not enough, as we needed a database to perform 10 times faster than what [the customer] had in place."
He heard a database expert contend that MySQL could beat anyone in terms of performance. "I was skeptical, but I got buy-in from the CO to try something innovative, so we went with MySQL." The result, Le Foll says, was a database that performed 20 times faster than what France Telecom had had. "The customer wasn't too upset that most of it was developed with free software," he says. "The customer doesn't care as long as it works and is supported."
The influence of MySQL is that it wipes out the low end for Oracle, which offers no application suites there. Rather, Oracle now builds and ships its software on Linux, and it ports to Sun and HP.
Open source software accounts for 95 percent of Acterna's critical components running at Tier 1s throughout Europe, Le Foll says. "We resolved reliability issues on COBOL implementations for interconnect processes—replacing IONA, which wasn't re-synchronizing well in extreme conditions, with a free implementation of Omni Orb."
Now Le Foll is working on a video test facility for "triple play" services. "It took three weeks to get a product for video over IP running with open source," he says.
Acterna also installed a few thousand Linux mini-server/test heads (OS software running on servers controlling test heads for triple play) at BT in the last 12 months. "Success there is galvanizing us to replace Solaris with Linux boxes in our testbeds for triple play, as well as in new handhelds," Le Foll says.
"Stability in the software we offer is the key; with open source, an upgrade in the version of Sun's Solaris OS or HP's UX doesn't automatically mean you have to touch the compiler. Rather, you recompile on the new machine and it works," says Le Foll, citing increasing support for open source and Linux by IBM, Sun and HP. "Ultimately, carriers benefit because they can use and maintain the same software product for years, regardless of hardware changes."
Demystifying Open Source
While open source has its benefits, it has its challenges as well.
For one, momentum on certain projects is not guaranteed. "The development teams are sometimes interested in art, not business," notes Andrew Updegrove, partner at Gesmer Updegrove LLP, a high-tech boutique law firm in Boston that represents more than 65 standards-setting consortia (http://www.consortiuminfo.org). "Because there can be an indifferent attitude toward user friendliness, adaptation in real-world businesses can be a challenge. Also, it's a totally volunteer effort, so there is no preexisting maintenance infrastructure for actual end users."
Indeed, development work can become lethargic in some areas, which is one of the risks when looking for support. Look at open source work for IVR applications or voice browsers, and you see updates from two years ago.
However, on the flip side, the infrastructure work for collaborative computing reveals a ton of work and energy that never ceases, as with the Linux kernel, which involves more than 750,000 people worldwide.
CIOs, CTOs or others who try to tailor open source software to business-specific needs take the risk of having no support down the road. It's widely agreed, though, that most open source projects have superlative support through various forums and consortia, some of which surpasses what is available even in the commercial world. Also, open source vendors offer full support for all delivered software, components, and operating systems.
Another common misconception is that open source software is "cheap" or even "free." Unless software posted on the Web is written exactly as engineers want, the "free" in freeware becomes a misnomer. While open source code can be up to 95 percent complete, it takes only a few lines out of tens of thousands to make debugging or tweaking a major project.
Even the biggest advocates of Linux concede that it is 10 times more difficult to debug code than to write it, which is why the open source community tries to write it as completely as possible the first time.
While that is the goal, the 80-20 rule is magnified with open source projects. "The joke in software development is that it takes 20 percent of the time to do 80 percent of the project, and 100 percent more time to do the last 20 percent," says Burger at Brooktrout Technology.
Open source development ostensibly becomes an exercise in debugging and reengineering. Open source applications written for Linux will have to be tweaked for Solaris shops, or open source software written for a Pentium processors will need tailoring if implemented in a PowerPC environment. That means vendors and carriers that use open source products must change the shape of engineering within their organizations.
"While open source may require fewer engineers to complete a job, it requires aggressive thinkers who can quickly assess the open source code and manage the risk of what is out there in terms of releases and technical support," says Acterna's Le Foll, who concedes he had to make drastic changes in moving to open source software. "I had to change out half of the engineering team, and there was a lot of attrition, as some people just couldn't work under that type of model."
"Since so many telcos are really 'billing systems with phone networks attached,' there already exists programmers for maintaining computer systems; you really just need a couple of really skilled staffers to realize the benefits of open source. You have to be willing to pay for that engineering though," says Burger. He believes another weakness that will resolve itself is the lack of business-specific applications: "Ultimately, end users don't buy the OS; they buy applications, so the fact most open source applications are 'university code' can be an inhibitor." However, with increasing support from IBM, Sun and HP, that code is being polished.
Concerns about licensing and intellectual property have also come to the forefront with the SCO lawsuit.
Open source is issued under General Public License (GPL) agreements, so when software is developed with it, modifications are expected to be posted back to the community.
Some get around that by writing applications on top of Linux, the open source operating system. Under GPL guidelines, dynamic linking opens up ways to make a proprietary application that doesn't have to be put back into the community. There are ways, in other words, to use open source with dynamic linking. However, if you modify open source code, it is recommended that you return modifications to the community so it derives benefit for the continuing pyramid of development.
"If you create proprietary extensions, then you remain proprietary, which is how businesses sometimes incorporate open source while maintaining differentiation," says Burger. "You just have to offer all open source you use to anyone to whom you sell your equipment." It used to be the case you had to automatically ship the open source software to customers—whether they wanted it or not. "That meant if Linux was running a residential router," Burger says, "you had to send it. However, now you don't have to send it unless the customer asks for it."
While the open source requirements can be onerous for equipment manufacturers, they really aren't for in-house developers.
"We make changes at the application layer to maintain strategic differentiation while using open source," says Unibill's David Guggenheim, senior vice president of sales and marketing. "We've come up with a program to provide open source that is not a general open source software license, which would require than any changes to Linux be propagated back to the open source community."
"The back office," he explains, "wants to maintain differentiation while gaining more control over applications." Providing it with open access to source code in its software, he says, enables it to overcome two common complaints in billing and customer care service bureaus: that differentiation is a problem, and that time to market is too long, as roadmaps for features have to be synchronized between the vendor and customer.
"We want applications to be hosted in our bureau, so we perform all the traditional tasks with bill runs and cycles, while the customer focuses on IT resources and development work around differentiation," Guggenheim says. The risk is that the customer's IT organization has to follow a set of guidelines in order to remain synchronized with Unibill's product roadmap.