Mediation in a Multi-Service IP Network

The Internet is moving beyond a web, remote access, and email network, to a robust platform for delivering sophisticated, value-added services and applications. What has changed? First, the massive fiber build-outs from Level 3, Qwest, Williams, and the existing fiber from AT&T, MCI, and Sprint hold promise to solve bandwidth constraints. Second, xDSL and cable modem technologies will offer residential and SOHO users inexpensive, high bandwidth access options. Finally, Internet QoS protocols (RSVP, MPLS, and IP over ATM/FR) and security protocols (IPsec, PPTP, and L2TP) have matured, enabling ISPs to eventually offer secure, dynamic QoS on a per-connection basis.

With a QoS and security-enabled network, ISPs have a platform (or convergent network) for offering services traditionally provided by circuit-switched networks. Opportunities for startups are enormous, and the threat to circuit-switched carriers is real. Already we see services such as telephone, fax, video-conference, pay-per-view movies, secure corporate network interconnection (VPN services), secure remote access, electronic commercial transactions, network games, TV, and radio broadcasts carried over IP networks.

What is required to bring these services to market is a comprehensive IP metering system, since the existing flat-rate pricing model simply does not apply to the resource-hungry, performance-oriented applications described above. Further, customers need to control their telecommunication expenses, and will select providers based on cost, QoS options, available support for both stationary and mobile users, and the provider's ability to measure and rate services under various usage-based metrics.

Before discussing IP metering technology, consider Table 1, which shows some typical corporate applications, their QoS requirements, and pricing structure in a multi-service IP network. (Note: prices are for illustration purposes only).

Application
QoS
Cost

Bandwidth
Latency
Directional
Local
Long Distance
International
Inter-Carrier
Off-peak discount
Bad effective QoS discount
IP Video
> 384 KBPS
< 200 ms
Bi
10¢/min
20¢/min
30¢/min
40¢/min
10%
10%
IP Voice
< 64 KBPS
< 200 ms
Bi
2¢/min
5¢/min
10¢/min
15¢/min
10%
15%
Resource access
High
< 1 sec
Uni
1¢/access + 3¢/megabyte
1¢/access + 6¢/megabyte
1¢/access + 8¢/megabyte
2¢/access + 10¢/megabyte
20%
5%
Fax and e-mail
Low
< 1 hr
Bi
1¢/access + 1¢/megabyte
1¢/access + 3¢/megabyte
1¢/access + 3¢/megabyte
1¢/access + 3¢/megabyte
30%
0%

Table 1: QoS and illustrative pricing of typical corporate applications.

In contrast to today's dedicated networks (e.g., PSTN, FR, and cable networks), which carry a single application type at a roughly fixed price, Table 1 shows that a multi-service network requires a flexible pricing model, where price is a function of the QoS provisioned. For example, voice and video applications require bi-directional latency and bandwidth guarantees, whereas resource access and fax/email applications do not have the latency constraint, but potentially require significant bandwidth. Thus, we can expect voice and video to have a flat per-minute rate, whereas resource access and fax/email applications have a setup plus a cost per MB of data transferred. A second key distinction, compared to traditional billing and rating systems, is the QoS discount. Since the network may fail to meet the QoS level specified by the application (which is likely, particularly in early network deployments), the customer will require a credit/discount for that session (similar to a SLA found in FR/ATM networks).

Fortunately, today's convergent billing systems from Saville, Amdocs, Kenan, and Portal are capable of providing usage-based billing support for the pricing structure discussed above. What is missing is a mediation system interfacing the IP infrastructure to OSS, BSS and CCB systems. Such an IP mediation system must be able to collect network usage data, produce a single service detail record (SDR) for back-end billing systems, link billing systems to network authentication, authorization and QoS systems, and seamlessly integrate with the carrier's CCB.

Why is IP mediation a challenging problem?

Fundamentally, the problem with IP mediation is that there is no single network element to exchange control and accounting information with OSS, BSS, and CCB support systems. For example, consider a voice call in the PSTN. Here, the central office switch can provide AAA (authentication, authorization, and accounting), generate a CDR and provide a single interface to support systems. In contrast, an IP network is composed of a substantial number of heterogeneous network elements, which provide and provision the service.

To illustrate the complexity of a carrier's IP network, consider Figure 1. Here, a remote user located in New York is connected to the corporate network in Atlanta via an ISP. Within the ISP cloud, the NY remote access server (NY RAS) provides ISDN or PSTN access. Routers (assume Cisco 7000s) are used at the ISP's New York and Atlanta network access points. Finally, a number of support systems (i.e., RADIUS authentication server, RSVP manager, H.323 Gatekeeper, network management server, web proxy, and a traffic shaper) are used to manage the network; that is, provide AAA service and guarantee QoS. (Note: a similar paradigm is applicable for a stationary remote office PC.)

In the network illustrated in Figure 1, consider a remote user would like to access corporate resources in Atlanta and perform the following network transactions:
* attend a video conference call with two people from the Atlanta office;
* search for an item in the corporate inventory database;
* store a presentation on the Atlanta central office file server;
* send multiple e-mails and a fax to a business partner in another company;
* browse through Internet web sites; and
* view an IP video news broadcast from CNN.

For these applications to work to the satisfaction of the laptop user (or, for that matter, a corporate desktop or home user), the carrier has to provide suitable QoS for each transaction type. Exactly how the network determines the QoS requirements depends on the application. For example, the video conferencing application would likely issue a connection request to the remote PC through an H.323 gatekeeper - which, in turn, allocates the bandwidth. The database inventory access and remote storage application can contact the known peer address directly, and request the bandwidth allocation directly from the RSVP policy server. (RSVP is a protocol that, for a designated flow, enables the allocation of bandwidth on each router/network element between two hosts.) The fax application would indicate its QoS requirements to a border traffic shaper that molds statistical parameters (i.e., priorities) of packet flows. Alternatively, depending on corporate policy or the type of SLA in the service contract, the bandwidth or priority reservation might be given automatically when the user logs into the network. Finally, e-mail and other low-priority messaging applications can be given low priority on the routers and traffic shapers so as not to disturb time-sensitive traffic.

After the network service is provisioned, the mediation system must obtain usage details from each of the network elements, and consolidate them into an aggregate billing record (or, aggregate CDR). In the examples shown above, the aggregate CDR should contain at least the parameters illustrated in Figure 2 (other parameters may be needed to accomplish other types of billing schemes).

Initiator customer ID
Terminator customer ID
Initiator and terminator IDs identify customer and supplier accounts within the billing system database that must be billed for the transaction.
Debitor/creditor ID
Transactions can be billed to the initiator, terminator, or both depending on specific business logic or other transaction parameters. If the transaction goes beyond the boundaries of the IP provider, there may be the issue of inter-carrier settlements. This is especially true when QoS reservations span across multiple IP carriers.
Initiator Location ID
Terminator location ID
Distance ID
Initiator and terminator location IDs are associated with the geographical location of transaction end-points. This information is used to produce a distance identifier, which can be used to charge differently for local, long-distance, international and inter-carrier service.
Application ID
The Application ID identifies the type of application used in the transaction (e.g., web, voice, and mail). The application information can be obtained from the port numbers within the TCP/IP header (e.g., web traffic is sent on port 80), or can be obtained from the gateway or RSVP policy manager.
Reserved QoS ID
QoS (i.e., traffic specification) reserved for the transaction.
Outgoing bytes
Number of bytes generated by the station.
Incoming bytes
Number of bytes received by the station.
Start
Transaction start time.
End
Transaction end time.
Figure 2. Aggregate billing record (CDR).

To generate an aggregate CDR, the mediation system must first collect and correlate data from the various network elements. In the remote corporate user example illustrated above, the router, RADIUS server, RSVP policy manager, network management node, and routing tables each produce various usage records. The Detail Records Sidebar explains what each network element does, and the usage information produced. By carefully correlating the usage records, the mediation system can derive network usage details for each network transaction. The final step, however, is to consult the CCB database to add account identifiers that are meaningful to the billing system (e.g., customer, debitor, and location IDs). After incorporating the CCB information, the mediation system can forward the aggregate CDR to the back-end billing system.

The aggregate billing record can now be rated and billed by the BSS depending on the type of contract with the customer. In some cases, based on the type of peering agreement with neighboring carriers, this transaction may require settlement, transit, or access charges. The financial model in this case can be taken from reconciliation terms from interexchange carriers.

IP mediation system design issues

Generating network usage accounting information to be used by a convergent usage-based billing and customer-care system isn't an easy task. Let's look at some of the challenges confronting an IP usage data collection system:

* As illustrated in the Detail Records Sidebar, there is no single point from which to collect basic network transaction data. The data exists in many network devices and application servers such as routers, IP switches, firewalls, mail and web servers, IP telephony gateways, and QoS facilitators. To further complicate matters, network usage records might also be obtained by monitoring packet streams at various locations of the network and reconstructing application events and usage.

* Each information source has a proprietary log file format, CDR format, and access protocol.

* By themselves, the basic transaction records generated by each network element are not sufficient for billing. The missing information exists in other information sources and, in most cases, must be correlated in real-time because of the dynamic nature of the reference data. For example, IP address are often allocated dynamically (e.g., by remote access devices or using DHCP on a corporate LAN) and network paths can vary depending on network load and availability.

* In a typical production IP carrier network, network elements generate millions of transaction records per hour. Sending these accounting records to a central data collection host (for batch processing) can degrade network performance and create the potential for losing usage records. In a large-scale environment, it is essential to distribute the collection process and to filter, aggregate and compress the data before sending it over expensive long-distance line segments.

* Various network elements can record the same event. For example, a web transaction may be accounted by a web-proxy and by a router. Duplicate CDRs must be detected and consolidated.

In addition to creating CDRs, an IP mediation system must also assist in provisioning network service, and provide a simple seamless API to the CCB for configuration of elements in the IP network.

Table 2 gives typical customer and system provisioning requirements that an IP mediation system must support. Customer provisioning involves enabling the network services available to the customer; for example, enabling a mail account, personal web site, security access codes, QoS restrictions, and more. Customer provisioning mechanisms must also handle the user's various access modes. For example, provisioning a dial-up user has different security requirements than a dedicated customer. System provisioning, on the other hand, involves authenticating the user, verifying their account is in good standing, and configuring information sources to provide usage accounting data (AAA). Both customer and system provisioning in an IP environment is often a complex task. Therefore, a simple API is critical so that configuration details are hidden from the CCB and support systems.

Customer configuration

System configuration

Dial-up Users
Leased-lines

Authentication
Firewall service
Configure routers to account

Mail
Mail, DNS
Configure servers to log

Personal web site
Web hosting

QoS
QoS
Roaming

Table 2: Example provisioning requirements from a mediation system.

To illustrate the challenge of provisioning IP services, Table 2 shows some of the network elements and their unique user resource database. In general, the CCB must be able to access these different user-related databases through the mediation system. For example, when a user dials into the network, the remote access server may query a RADIUS server in order to permit a login or access to restricted information. Based on CCB request, the IP mediation system must also have the capability to add, remove, and update user-related information at the RADIUS server in real time. Likewise, since a router contacts a QoS policy server in order to grant QoS reservation for a given user session, the mediation system must be able to inject user-related information to the QoS policy server. The same applies for a mail, web server, and firewall that have user-related databases granting access privileges.

Integrating mediation and CCB systems

The most difficult challenge in IP mediation is reducing the number of raw data records that flow through the collection system. By aggregating usage-information, the mediation system reduces record data flow considerably. However, there is a trade-off. If the system over-aggregates raw data records, the usage information presented to support systems may not be able to offer flexible, competitive pricing models for different market segments. For example, the IP usage collection system might be configured to produce daily aggregate usage records for each user, thereby substantially reducing record flux. However, records ending up in the CCB will not contain the time of day when each transaction took place, therefore making it impossible to discount off-peak usage.

A way to cope with this problem is to have the CCB provide business logic to the data collection system in a way that the aggregation will not reduce the granularity of billable parameters. In the above example, the business logic will demand that instead of having daily aggregation, the collection system should aggregate the records for the segments of the day that define peak and off-peak hours. Note that the batch-oriented approaches in traditional networks (where raw CDRs are centrally collected from switches and fed into the CCB without aggregation) will not work because a typical IP network might generate hundreds of millions of raw records per day. This volume of CDRs can not be stored cost-effectively into a database, but they can be aggregated in real-time to a manageable volume.

The second mediation challenge is that two levels of CDR details are needed: one by the CCB to produce a bill, and the other for customer support. For billing purposes, the system should collect an order of 10 to 100 records per customer per day because a typical customer prefers a simple bill without detailing each network transaction's duration, number of bytes transferred, etc. With 10 simple records a bill can summarize usage activity to which the customer can relate, such as "web activity at $0.01/MB", or "VoIP during work hours at $0.03/hr".

The second type of accounting information is needed for the purpose of customer care. A customer may call up the operator and say: "I just placed this long-distance call, and the quality was terrible." In real time, the operator must be able to look at the customer activity database, verify the story, and possibly say, "Indeed, you spoke to your office in Australia. Currently our Australian fiber lines are out, so we routed your call over satellite. We will credit your account." This emphasizes the necessity for real-time access to high-resolution customer-support data.

Supplying customer-support data is not an easy task since detailed usage information is filtered by the collection process. Therefore, it is necessary to distribute the collection process and produce at least two different levels of aggregate tables. For example, consider Figure 3. Here we see an IP carrier that has point of presence in New York, Boston, and Atlanta. Mediation mechanisms are located in all three sub-networks, collecting, filtering and aggregating billable usage records. The full bulk of accounting records are stored in a separate detailed accounting database in each of these sub-networks, whereas the billing database and CCB is in New York. Remote systems forward aggregated billing records to the central billing database without flooding expensive inter-state communication lines. The customer care database, including the full detail of user activity, is located close to the information source. If necessary, as in the above customer care example, the central CCD can retrieve each NE's CDR on-demand by querying the respective remote database. The Mediation Architecture Sidebar gives an overview of a production IP mediation system from Xacct Technologies.

Conclusion

Usage-based billing models are the next logical progression in the evolution of Internet commercialization. As IP technology and economics improve, customers will continue migrating applications traditionally provided by circuit-switched carriers to IP networks. For an ISP to compete successfully in this market, they will not only have to offer on-demand, secure, QoS-based service, but with meaningful price metrics.

IP mediation systems will play an instrumental role in the Internet's commercialization process. From an engineering perspective, mediation systems will determine network usage, provision services, and insulate support systems from the complex nature of IP carrier networks. Mediation systems will also play a critical role facilitating the integration of future value-added services as well as next-generation IP transport technology.

In the end, however, the pervasiveness of mediation mechanisms will depend on the customer. According to Andrew Campbell, Senior Vice President of Products at Saville, "for the purposes of interconnection, there is a need to measure bits and bytes, this is where IP mediation is critical." However, "small business and residential customers don't want to know about it {bits and bytes}."

Campbell's point is important. Concerning their VPN service, sophisticated corporate users want specific network usage and performance details on a per-user, per-department, and per-application basis. Likewise, such details are critical for inter-carrier settlements. However, in the residential market, Campbell says, "if I couldn't explain it to my mother, I couldn't sell it." Figure 1 illustrates Campbell's point. Here we see pricing as a combination of session duration, access, data flow (MB of data transferred), distance, and time of day. Campbell believes the residential market will not be able to understand such metrics. Instead, residential billing systems must stay away from complex pricing schemes and identify usage at the application level (e.g., "$40 for X hours of video.") Campbell notes that for such a pricing model to succeed, QoS must disappear as a problem and become a standard. Although many IP experts believe standard QoS is not in the cards anytime soon, we already see announcements from IP carriers such as UUnet guaranteeing network latency.

Mediation architecture sidebar

A common mediation architecture is to distribute data-collection and provisioning mechanisms among the network elements and support systems. For example, Figure 4 shows the distributed architecture of an IP mediation system developed by Xacct. The central event manager coordinates, manages, and controls the operation of the system and its related components. Accounting records are sent to the CEM and are stored in a database, sent to another CEM, or passed to the CCB. A hierarchy of CEMs can be achieved in order to organize billing information in one database, and detailed customer care information in distributed databases.

Gathering agents collect network session data from the individual information sources (e.g., RADIUS server, netflow router, and H.323 gateway). The Gatherers are strategically located close to the information source(s) to minimize traffic on the network. Typically, Gatherers are hierarchically organized, sending network transaction data either to other gatherers for further aggregation, or directly to the CEM. Using special interfaces, called ISMs, Gatherers access an information source without interfering with its operation. Gatherers capture and process data from both asynchronous (or "trigger" sources) and synchronous (or "enhancer") sources. For example, a router generates session triggers whereas the DNS server helps enhance the session record. Gatherers are remotely administered and controlled by the CEM.

Detail Records Sidebar

Figure 5 illustrates the process by which a mediation system collects usage information from various network elements (NEs). The approach is to communicate with each NE, obtain the CDR, and forward an aggregate CDR to the billing system. In the network shown in Figure 1, the mediation system must coordinate with various network elements to capture the network transactions correlating with the corporate user's application. The CDRs generated by the NEs are discussed below:

Netflow accounting record: Routers are network devices used to forward IP packets across the ISP network. Cisco's routing equipment has the capability to produce per-connection usage records on a per-connection, unidirectional flow aggregate. These types of records provide the basic raw material from which billing information is built. Some of the useful parameters given in a NetFlow record include:

Source and destination IP Address
Initiating and terminating IP addresses
Destination port
Terminating IP port (in some cases representing the application used)
Protocol
TCP or UDP
Sent and received bytes/packets
Number of bytes
Start/end timestamps
Time that the flow began and ended
Netflow usage record.

RADIUS (remote authentication dial-in user service): When a user dials into a remote access server, they are transparently assigned a dynamic IP address as part of the login procedure. The IP address is valid until the user log-out, when it is assigned to the next user logging in. RADIUS servers authenticate remote dial-in access, log authentication request and results, as well as account for login and logout events. Mediations systems can use RADIUS records, given below, to associate network transactions with customers.

IP Address
Dynamic IP address that was given to the user at login time
User name
Name of user
Start (login), End (logout)
Times of login and log-out
RADIUS usage record and reference data.

RSVP (A resource reservation protocol): An RSVP policy manager, such as Class-Data, accounts for QoS request events. By correlating this data with the netflow records, it is possible to determine the connection's requested and QoS provisioned.

Source and destination IP address
Initiating and terminating IP addresses
Start and end
Timestamps of the QoS session
Requested QoS
Level of QoS
RSVP usage record and reference data.

Network management records: IP Network management stations such as HP-Openview, among other purposes, map network (IP) addresses to machine locations and user groups. By accessing the network management system's database, it is possible to associate IP addresses to billing entities and geographic locations. The following information can be obtained:

IP group
Groups of IP addresses and network masks
Real-world entity
Name
Geography
Location associated with entity
Openview reference data.
Routing tables: Routers use dynamic IP routing schemes to balance network load among several paths, and to allow for redundancy and fault tolerance. In some cases, to find the path by which network transaction passed and identify external billing entities, it is necessary to collect the following information from routing tables:

IP group
Groups of IP addresses and network masks
Path (IP) Identifier of the path to be taken by IP packets destined to IP group
Routing tables reference data.

About the authors:
Matthew Lucas is an IP billing and multicast system consultant. He holds a Ph.D. in computer science from University of Virginia and a B.S. in math/computer science from Carnegie-Mellon University. Matthew can be reached at matt@telestrategies.com.

Limor Schweitzer is co-founder and CTO of XACCT Technologies, Inc. Limor is the architect of the XACCT usage Network Usage Metering solutions. He can be reached at limor@xacct.com.