Team 3 Webpage

TEST TOOL DEVELOPMENT FOR ACCESS CONCENTRATOR DEVICES

CSC402/ECE470 - (Inter)Networking Technology & Projects

TEAM 3

Dalat Bui
Chris Brezovec
Shadi Eskamaei
Tyler Humphrey
K. Fritz Lehr

North Carolina State University

November 20, 2002

EXECUTIVE SUMMARY
(to Table of Contents)

The following document provides information pertaining to our project on Access Concentrators. This report includes a brief introduction of our project, a description of the problems we are dealing with, our approach to solving these problems, our results from implementing our solutions, and a verification of our results. We have also included a schedule of our activities throughout the semester. In this schedule we have included our preliminary estimates on start dates, estimated completion dates, and actual completion dates.

The overarching goal of this project is to develop a test methodology to interrogate the operational ranges of different Access Concentrators (AC's). Through our research, we have found that many AC's are evaluated based on their session capacity, session establishment rate, throughput, and session stability. For each of these major aspects, we develop a test to determine its operational range. To make sure that these tests are valid, we run them on many different brands of AC's that are being used in the field today. While developing the tests, we come up with a methodology. At the same time, we also look for new features that many vendors are advertising. Based on our test results and the research, we will make recommendations for future tests.

Also in this report is an appendix which is an RFC written by our team. This RFC is a document containing our detailed methodologies. It is basically a template for the testing and comparing of access concentrators. Our original plan included using our test methodologies to actually test some access concentrators. Because we were not able to get the equipment to test, our methodologies were reviewed by our mentors from Spirent Communications who can be considered experts in this field. These tests therefore are ready to be implemented by our peers and other professionals to test the specifications on any access concentrator.

TABLE OF CONTENTS

Cover page
Executive Summary
Table of Contents
Introduction
Problem Description
Approach to Solution
Results
Verification and Validation
Schedule
References
Appendices

INTRODUCTION
(to Table of Contents)

Owing to the success of the Internet, the number of connected hosts is growing at an exponential rate [10]. As a result, service providers and carriers continually have to respond to the increasing demand. They look for ways to expand their network while reducing the operational costs. One solution these service providers and vendors will most likely consider is an Access Concentrator that can handle their current needs and allow them to expand their network in the future.

An Access Concentrator is an intelligent networking device that sits at the edge of the IP cloud between the Internet Service Provider (ISP) backbone network and the end users (or end user private networks). Its main job is to handle all the connections between the two parties. An AC, which can act as both a router and a switch, is built to handle many connections and to provide many IP services such as Virtual Private Networks (VPN), management control, and security services.

The popular protocol used by many AC's to set up and maintain connections is the Point-to-Point Protocol (PPP). This protocol is used as a result of the early dial-up connections. Originally, Serial Line Internet Protocol (SLIP) was used to network two computers by using a telephone line. However, as networking technology developed and the medium bandwidth increased, the SLIP was replaced with the PPP. These are some of the reasons why PPP is now the standard protocol used by ISP's in a point to point connection. To expand their service and increase the number of customers, PPP is run over Ethernet (PPPoE), over ATM (PPPoA), and over both Ethernet and ATM (PPPoEoA). In this project we are mainly dealing with PPPoE because it is more popular and simpler than the other two.

The Point-to-Point Protocol is designed for simple links which transport packets between two peers. These links provide full-duplex simultaneous bi-directional operation, and are assumed to deliver packets in order since PPP runs over a logical serial link. It is intended that PPP provide a common solution for easy connection of a wide variety of hosts, bridges and routers.

The PPP state diagram consists of four main states. These states are:

Establishment State
Authentication State
Networking State
Termination State

The Establishment State is the state where the sessions is being setup. In this state Link Control Protocol (LCP) traffic is being sent between peers. LCP traffic consists of a number of handshakes where a session is requested, acknowleged, resulting in a Session Identification (Session ID) being created. This session id is unique to all sessions, and it is the way that a PPP session is referenced. Once a session id has been given to a session, PPP then moves into the Authentication State.

In the Authentication State, LCP traffic is still sent, but now the user has the option to authenticate with each other. There are two main forms of authentication. These are PAP and CHAP. Our test use PAP since it is the simpler of the two authentications. This Authentication State is optional. Once the Authentication, if any is done, PPP then transitions into the network state.

In the Network State, LCP traffic is still sent to maintain the session. Here is where the different types of traffic are agreed upon and sent. This is done using the Network Control Protocol (NCP). PPP will stay in this state until it is time to terminate the connection. At that time it will transition into the Termination State.

In the Termination State LCP traffic sent to teardown the session and release the session id that was being used. Once the session is torn down, PPP is finished, and transitions into what is called the Dead State.

For more information on this please refer to RFC 1661 [9].

PROBLEM DESCRIPTION
(to Table of Contents)

Through the examination of what the purpose of an Access Concentrator is, its location in the network, and various known attributes, we have derived four main aspects that should be tested on the network device. These four categories of tests focus on the Access Concentrator's Sessions Capacity, Throughput, Sessions Establishment Rate, and Device Stability. We also found the need to determine what other test categories should be added to the list that test for specific functionality that vendor’s think are important and advertise.

These four tests are the backbone for many other tests that might be created later on.

Vendor Specific Functionality

Many AC vendors such as Redback, Cisco, netCom, Nortel, Cosine, and Juniper are working competitively to claim a larger share of the market. This sometimes push them to over advertise their products. For example, Redback may claim that it's SMS10000 to be able to 100,000 concurrent connections [12]. In reality, it may never be able to handle that many concurrent connections unless some special techniques are applied. Through our research and a close examination of an Access Concentrator, we derive four major aspects that should be tested: session capacity, session establishment rate, throughput, and device stability. The test on each of the aspects will be designed to simulate situations that are as close to real life situations as possible. In addition, these tests will determine the operational ranges of each of these aspects. Many other attributes of each vendor's Access Concentrator are advertised that are rather difficult to test. However, the problem is the aforementioned quantitative attributes can be either overestimated to make their particular product seem better than it actually is or underestimated in order to assure that an unstable state will never be reached. Tests will be run to test the actual capabilities of each vendor's product to determine how close the advertised specifications come to the actual values.

Session Capacity Problem Description

The first test that needs to be performed is the session capacity test. Session capacity is the maximum number of concurrent sessions an Access Concentrator can establish and maintain. This is important because if the maximum capacity is exceeded, the Access Concentrator drops sessions . For example, an ISP planning to purchase and use an Access Concentrator knows the typical maximum number of sessions that need to be established on their network. If they purchase an Access Concentrator that can not do this, they will drop sessions and upset their clients. This is why a test methodology to determine the Session Capacity for an Access Concentrator is needed. It is also needed to correctly perform the following tests, which including Session Establishment Rate, Throughput, and Device Stability.

Session Establishment Rate Problem Description

After the Session Capacity has been determined, it is necessary to determine that maximum rate at which these session can be established. When an Access Concentrator has a large maximum session capacity, it must also have a high session establishment rate. Session establishment rate is the rate at which an Access Concentrator can process connection requests. In the "real world" when an ISP has a large number of subscribers, it must also have an Access Concentrator(s) that is evaluated to have a high session establishment rate. This is mainly for reliability purposes. For example, when the AC is brought back up as a result of hardware failures or attacks, at this instant, it is very likely bombarded with large a large number of connection requests. If the AC is not able to handle these requests at a high enough rate, it will crash or freeze. This will result in customers' complaints and further loss of revenue.

Throughput Problem Description

Once all the necessary connections have been established and the session capacity is known, a throughput test can be run to check how well an AC network device transmits and receives data. Vendors are able to report a single value that has been proven to have use in the market place from the throughput figure determined by the tests. It is useful to know the actual maximum data rate that a device can support, since even the loss of one frame in a data stream can cause significant delays while waiting for the higher level protocols to time out. An example of where this is important is, if the throughput is low, then clients of an ISP might switch ISPs so that they can get more things done, like surfing the web or downloading files, on the network quicker.

Session Stability Problem Description

Now that the throughput, session capacity, and session establishment rates are known, some device stability tests need to be created and run to see how an Access Concentrator might recover when traffic patterns exceed determined limits. The traffic patterns that cause the Access Concentrator to reach an unstable state need to be defined and measured. After this state is reached, the severity of the state needs to be determined. This is done by measuring the time it takes for the Access Concentrator to become stable again on its own. The longer it takes, the more sever the instability.

Many Access Concentrators may pass individual tests in the lab, but in a “real world” traffic setting, the Access Concentrator may reach an unstable state due to a combination of requests. Since the goal of the Access Concentrator, is to be installed, and never be touched again, unless some new configuration is wanted, the Access Concentrator needs to be stable. If an unstable state is reached, the Access Concentrator must become stable in a very short amount of time. Anytime the Access Concentrator is unstable in the field, it costs the buyer money.

APPROACH TO SOLUTION
(to Table of Contents)

The general approach to the solution that our group took started out with everybody reading and investigating exactly what an Access Concentrator was. We looked at what makes an Access Concentrator different from a router or a switch. Based on this definition we, with the help of our mentors, determined the certain key elements that make and Access Concentrator and that should be tested.

The five, original, major elements that we decided on were session capacity, throughput, sessions establishment rate, Session Stability, and Session Forward Stability. Each team member picked a topic, researched it, and wrote a section of the RFC based on that topic. Once we met with Kumar to go over the first drafts of the RFCs, we decided that based on the research that was done, and based on general PPP session knowledge, that Session Stability and Session Forward Stability were not really important tests. The work that was done for those tests was scratched, and we replaced the both of them with Access Concentrator Stability as of November 7th, 2002. It was also decided that a definition section of the RFC would be required..

Vendor Specific Functionality

In order to test each vendor's product, test methodologies and algorithms to run these tests have been designed. The following details each individual test and a way to quantitatively compare the vendor's "estimated" specifications to the actual specifications found by running these tests. In order to test the device, the AC is set up to send and receive data in each of a number of sessions. See Appendix A Section 0.3 for more information on Test Setup.

To find specifications on each vendor's access concentrators, their corresponding websites were referenced. The various access concentrator vendors included

Redback [12]
Nortel [13]
net.com's [14]
Cisco [15]
Cosine [16]
Juniper [17]

Each vendor's website included information about their access concentrators. Most of these companies provided the specifications that we are testing. There were other specifications such as security and recognized protocols which will not have tests outlined in this report.

Session Capacity Test

Several parameters must be controlled and clearly defined to develop a test methodology that accurately tests Session Capacity. This methodology must be consistent and repeatable. This does not mean that the Session Capacity determined by this methodology will be the same as that listed in the Manufacturer’s Access Concentrator specifications.

First, start each test with a clean boot of the Access Concentrator. This ensures that the Access Concentrator is operating correctly and provides accurate results. Also, measure the amount of memory currently available in the Access Concentrator. Then, determine how authentication will be handled. Authentication can be done on the Access Concentrator or on a radius server connected to the Access Concentrator. This is important because it affects the amount of memory used per session. Finally, control all the traffic between the test machine and the Access Concentrator so it does not affect the results.

Begin test by trying to establish and validate the initial number of sessions. If this is successful, then increment by test by trying to establish an additional 1000 sessions. If you validate that all the sessions are established, repeat this process, and validate that all the sessions are established. When you reach a number where you cannot validate that all the sessions are established, record the last number where you could validate all the sessions. Use this as the starting point and set the increment to 10 and repeat the above process until you again cannot validate all the sessions.

The final step of test is to set the number of sessions to the last number where you could validate the established session. Use this as the starting point and set the increment to 1 and repeat the process in the above paragraph of this section until you again cannot validate all the sessions. The last number of sessions that could be established and validated in the test is you session capacity.

After this final test, re-measure the memory. This information is used to determine how much memory is used for each session for comparisons with other machines. Determine this by dividing the memory used by the number of sessions to get memory per sessions.

Session Establishment Rate Test

To measure the session establishment rate, two tests are done. One is to send out a finite number of PADI packets at a constant rate. The other test is to send out a number of PADI packets at a bursty rate. As soon as the last PADI is sent, a time-out clock is started. When the time is up, we will measure how many connection acknowledgement packets come back. Based on the number of connection acknowledgement packets that come back comparing to the number of PADI sent, we can tell whether the Access Concentrator is able to establish the connection requests packets at the rate. By adjusting the rate at which the PADI's are sent, we can determine the maximum rate at which the access concentrator can handle.

The reason we will do two tests is that different types of connection request traffics can have a large impact on the establishment session rate. In the real world, the connection request traffics can be unpredictable. However, most of those traffics can be categorized into two ideal rate patterns, either constant or bursty.

Throughput Test

Conceptually, measuring throughput is a simple task that involves issuing a certain number of requests, counting the number of replies (bits or bytes) received, and dividing that number by the time (seconds) it took to complete the test. Since the layout of a wire is composed of two different pipes, one upstream and one downstream, then the replies should should be counted in each direction along with the traffic in that direction. To simulate real data transmission accurately, we test for both sequential and non-sequential data transmission. Therefore, the size of individual IP packets as well as the size of bursts during non-sequential traffic transmission should be accounted for as set-up parameters.

The software being used to count the number of packets received also evaluates each packet for content and bit count, making sure that no faulty packets have been received.

Also, as with any other test, in order to get a quantitative idea of the robustness of a particular measurement, it is necessary to run the same test several times. Especially with throughput if only one throughput estimate is computed for the entire test, the variations that may occur at time scales shorter than that of the entire test will be hidden.

The algorithm devised allows many packets to be sent in a sequence without synchronizing between the sender and the receiver. This approach to a solution was taken because the algorithm ensures getting a quantitative idea of the robustness of a particular measurement resulting in accurate calculations of throughput measure on the Access Concentrator device.

Device Stability Test

In order to write a test methodology for Access Concentrator Stability, one must know how to measure instability. This was the first thing that was looked at was what it meant for an Access Concentrator to be in an unstable state. Eventually it was decided that the instability of an Access Concentrator should be measured based on how long it takes for it to reach a stable state. Once an unstable state is reached, this is defined as a state where things are happening that should not be, the Access Concentrator is configured in such a way that it backs off to a known stable state whatever was being done to push the Access Concentrator. The severity of the instability is measured by the time it takes for the Access Concentrator to recover without outside interference. An example of an unstable state is where sessions are being torn down that should not be. Stability would be regained when the Access Concentrator can once again accept and maintain all of its connections and requests.

Once instability and stability were defined and quantified, one must now consider how to get the Access Concentrator into an unstable state. In order to test the stability, one must be able to get into an unstable state. Nothing is perfect, the Access Concentrator does not have infinite memory, or infinite bandwidth at its command. The Access Concentrator, just like all systems have their breaking points. The breaking points need to be found and used to test instability.

These breaking points are defined as:

Sessions are no longer being created.
Sessions are being torn down early.
Session throughputs have been reduced below an acceptable range.
Jitter associated with each of the existing sessions becomes outside of tolerated limits.

For more on how to test the instability and the various ways to get the Access Concentrator unstable, please refer to Appendix A and the section pertaining to Access Concentrator Stability for a more in-depth examination on how to do all of this.

Another situation an Access Concentrator may reach instability is in an internet based traffic model. This is where the traffic being simulated is being simulated as if it were on the internet. Many Access Concentrators may be able to pass through the lab with no problems, however, the internet based traffic is much different then the traffic patterns normally used. One of the key differences is the length of time it takes to run a test. Normally these types of tests run for at least a day. Testing various traffic patterns that may be experienced during the day at the time they might be witnessed.

For more information of this, please refer to Appendix A and the section pertaining to Access Concentrator Stability.

RESULTS
(to Table of Contents)

Our results after following the previous approach are listed in Appendix A. We created a test methodology document that consists of these tests and a written set of procedures found in the rfc in the appendix. This written set of procedures is used as a logical frame work on how to run the tests. Our group did not get to actually write the scripts for the test tools themselves because the necessary equipment was never made available to our group. Our time was spend defining and fine tuning this methodology document. To verify that these results and assumptions that are made are valid, we had a few draft review sessions with Ralph and Kumar of Spirent, who can be considered experts in this field.

One of the tasks was to research different vendor specific functionality and make recommendations on what tests might be useful to create in the future. Many of the vendors give a quantity for attributes of their access concentrator such as net.com's claim that their product (SCREAM 100) has a "maximum throughput of 5Gb per second per Network Data Plane (NDP)" and "can handle up to 20 Gb per second of bursty traffic per NDP" [14]. Depending on which devices are available for testing, many of these company's products will be tested using the following algorithms in order to compare actual findings to vendor advertised quantities.

The Access Concentrators to be studied included:

The Redback SMS 1800 and SMS 10000
The Nortel Shasta 5000
net.com's SCREAM 50, 100, and 200
Cisco's 10K and 6400
Cosine IPSX
Juniper

All of the tests that were performed by our group were reasonable and made good test items. Testing device stability is somewhat more advanced and difficult to perform. The reason for this is that it is impossible to simulate "real-life" traffic. Different aspects of the Access Concentrator could be put to test in an attempt to cause an unstable state, however, when placed in a real world situation the Access Concentrator is put through a much wider variety of traffic than can be somewhat reasonably tested.

Other tests that should be included could be issues such as the Access Concentrator's security features and it's routing protocols. Adding new IP services such as voice over IP , virtual private networks , and IP Multicast create a considerable strain on existing resources and network designs. These tests could later be added as test cases to our device stability methodology.

Often, users make significant trade-offs with throughput, capacity, or service mix. These choices can lower expected operating margins and create inconsistent service characteristics. This may be difficult to test but again would provide the most information as to how the Access Concentrator would react in a real world situation.

VERIFICATION AND VALIDATION
(to Table of Contents)

This part of the project determines the validity and accuracy of our methodology. It relies heavily on the input from of our mentors, Ralph and Kumar, at Spirent. They reviewed our drafts and discussed any changes or modifications needed based on their experience and knowledge of how to test network devices. Once we received feedback on our drafts, we made revisions based on their recommendations. This process was repeated several times until they verified the accuracy of our test methodology.

Vendor Specific Functionality

Through the use of Spirent Com’s Adtech Network Tool v4.31 and the algorithms designed for each attribute to test, the actual specifications can be found. These specifications should be compared individually to the specifications advertised by the vendor for the Access Concentrator. The comparisons and any conclusions drawn from this comparison should be reported when completed. Because we were not able to test our methodologies on an actual access concentrator, we gave our tests to our mentors at Spirent Communications to review and speculate on our results.

Session Capacity Verification

To make sure the methodology is valid, a draft presented to our mentors to check for accuracy and consistency. The draft methodology is revised several times and a final copy is validated by our mentors. The session capacity tests will look for:

The maximum number of sessions established and maintained
How much memory is used by each session for comparison

Once the methodology can be tested on an actual Access Concentrator, the results can be compared with the assumptions made by the test. The comparisons can then be used to make recommendations or modifications to the methodology for future testing.

Session Establishment Rate Verification

To make sure that the establishment rate part of the methodology is valid, we will need to run a sanity test on many different Access Concentrators. Due to the current unavailability of the different Access Concentrator devices, we are left speculating the results of the test. There are several parameters we will examine:

Number of PADI packets sent.
The amount of time sending out the PADI packets.
Number of connection acknowledgement packets.
The time it took to set up each connection.
The number of packets dropped.

We would run the test several times to make sure that the result we get are coherent. In addition, the results would have been in the form of graphs. There would have been two separate graphs: One for constant back to back PADI rate and the other one for the bursty PADI rate. The x-axis is the Establishment Request Rate, and the y-axis is the corresponding percentage of successful established sessions. The point where the Establishment Request Rate is highest and the percentage of successful connection is still 100% is the maximum establishment session rate.

The result of the test will be done later by future groups when the devices become available.

Throughput Verification

In order to verify that the throughput test methodology that was defined for Access Concentrator is correct, the test should be run, using Spirent Com’s Adtech Network Tool v4.31, on many different Access Concentrator brands including, Redback SMS 1800 and SMS 10000, Nortel Shasta 5000, net.com SCREAM 50/100/200, Cisco 10K and 6400, Cosine IPSX, and Juniper. Exactly which brands will be tested depends on machine availability. More importantly, when running the test methodology the following key points of interest are studied to determine if tests are valid:

Successful generation of a pre-determined data stream in accordance with specific application’s encoding rules.

At the receiver end, protocol analysis software can correctly assess the throughput.
Software correctly counts the number of packets received.
Evaluates each packet for content and bit count.
Returns error if defects are found in packets received.

Also, modifications to the algorithm that simulate the effects of adverse operating conditions such as signal jitter, noise, delay, and amplitude changes can be programmed to further evaluate and assess the system under study.

Stability Verification

In order to verify that the methodology that was defined for Access Concentrator stability was correct, we need to test out our results on many different Access Concentrators. Unfortunately due to the unavailability of the devices, the actual running of the tests will need to be done by future groups.

When running the methodology the following are things that we looked for to determine that everything was going right:

Run a sanity test.

Created 10 sessions.
Ran traffic through these 10 sessions at 5% line rate.

Looked at the number of sessions created and the memory associated with each.
Looked at the keep alive responses.
Looked at the packets on the other side of the throughput sanity test.

Teardown all of the sessions and wait a few minutes for memory to free up.

Started to run the specific methodology.

Viewed the packets going out of the test device.
Viewed the packets received on the other side.

Collect results from the test.
Re-run the test the recommended number of times.
Created graphs based on data.
Analyzed these graphs based on expect data.
Reported any differences to be looked at later.

The methodology and the expected results are in the Access Concentrator Stability section of Appendix A.

RESULTS

As a result of all the research and thinking that went into this project, our group created the RFC that is in Appendix A. The rfc contains five sections. The sections deal with definitions used in throughout the rfc, session capacity, session establishment rate, throughput, and device stability. The rfc, however, is not completely finished. There are some small changes that need to be made in the future before it is ready for approval.

SCHEDULE
(to Table of Contents)

REFERENCES
(to Table of Contents)

[1] Bradner, S. "RFC 1242: Benchmarking Terminology for Network Interconnection Devices." Network Working Group. July 1991.

[2] Bradner, S. "RFC2026: The Internet Standards Process -- Revision 3." Network Working Group. October 1996.

[3] Bradner, S. and McQuaid, J. "RFC 2544: Benchmarking Methodology for Network Interconnect Devices." Network Working Group. March 1999.

[4] Gross, G., Kaycee, M., Lin, A., Malis, A., and Stephens, J. "RFC2364: PPP Over AAL5." Network Working Group. July 1998.

[5] "Journal of Internet Test Methodologies." RouterTester Solution. Agilent Technologies. November 2002.

[6] Mamakos, L., Lidl, K., Evarts, J., Carrel, D., Simone, D., and Wheeler, R. "RFC 2516: A Method for Transmitting PPP Over Ethernet (PPPoE)." Network Working Group. February 1999.

[7] Mandeville, R. "RFC 2285: Benchmarking Terminology for LAN Switching Devices." Network Working Group. February 1998.

[8] Mandeville, R. and Perser, J. "RFC 2289: Benchmarking Methodology for LAN Switching Devices." Network Working Group. August 2000.

[9] Simpson, W. "RFC 1661: The Point-to-Point Protocol (PPP)." Network Working Group. July 1994.

[10] Sun, Andrew. "Using & Managing PPP." California: O'Reilly & Associates, Inc., 1999.

[11] Zacon, Robert. Hobbes' Internet Timeline v5.6. 1 April 2002. 11 October 2002.

[12] www.redback.com Provides information on Redback AC's.

[13] http://www.nortelnetworks.com/products/family/shasta.html Provides information on Nortel's AC's

[14] http://www.net.com/products/broadband/index.shtml Provides information on net.com's AC's

[15] http://cisco.com/warp/public/44/jump/routers.shtml Provides information on Cisco's AC's

[16] http://www.cosinecom.com/products/index.html Provides information on Cosine's AC's

[17] http://www.juniper.net/solutions/edge/ Provides information on Juniper's AC's

[18] Hares, S. "RFC 1575: An Echo Function for CLNP (ISO 8473)" Network Working Group. Feburary 1994

APPENDICES
(to Table of Contents)

InterNetworking Class
Request for Comments Draft
Category: Draft

Chris Brezovec
Dalat Bui
Tyler Humprey
K. Fritz Lehr
North Carolina State University
December 2002

Access Concentrator Test Methodology

Status of this Memo

This memo provides a draft of a test methodology to interrogate the operational ranges of an Access Concentrator. It is still in the development phase and is not meant to be published. Distribution of this memo is unlimited.

Table of Contents

0   Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . .4

      0.1   Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . .4

      0.2   Term Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. . 4

      0.3   Methodology Definition Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . 5

               0.3.1   Subscriber Session Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 6

               0.3.2 Session Establishment Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 6

               0.3.4 Session Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . .. . 6

               0.3.5   Device Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . .. . 7

      0.4     Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 7

               0.4.1   Configuration Being Simulated (Logical) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . .7

               0.4.2   Configuration In The Lab (Physical) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.   Subscriber Session Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . 8

      1.1     Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

      1.2     Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

      1.3     Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

      1.4     Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

      1.5     Setup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . .. 8

               1.5.1   Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . .. . . 8

               1.5.2   Session Establishment Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 9

               1.5.3   Established Connection Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . 9

               1.5.4   Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . .. . . 9

               1.5.4 Physical Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . 9

      1.6     Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

               1.6.1 Session Capacity Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . 9

               1.6.2   Session Capacity Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . ... . 9

      1.7     Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

      1.8     Reporting Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. . . .. . . 10

2.   Session Establishment Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . .. . . .10

      2.1     Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . .. . . .10

      2.2     Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

      2.3     Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . .. . . 10

      2.4    Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . 10

      2.5     Setup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . .10

      2.6     Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

               2.6.1 Measuring Session Establishment Rate with Constant PADI's . . . . . . . . . . . . . . . . . . 11

               2.6.2 Measuring Session Establishment Rate with Bursty PADI's . . . . . . . . . . . . . . . .. .. . . . 11

      2.7     Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. . . . 12

      2.8     Reporting Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 12

3.   Session Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 12

      3.1     Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. ... . . ..12

      3.2     Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. . . . .13

      3.3     Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . .. 13

      3.4     Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 13

      3.5     Setup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . 13

      3.6     Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . 13

      3.7     Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 14

4.   Device Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . .14

      4.1     Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . .14

      4.2     Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . .15

      4.3     Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . .15

               4.3.1 Types of Traffic That Will Be Used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . .15

                        4.3.1.1 LCP Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . .15

                        4.3.1.2 NCP Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. ... . . . . 15

      4.4     Setup Parameters and Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 15

      4.5     Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 15

               4.5.1 Test case 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 16

                        4.5.1.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... .. . . . . .16

                        4.5.1.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .16

                        4.5.1.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . .. . . . . 16

                        4.5.1.4 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... .. . . . . . .16

                        4.5.1.5 Setup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 16

                        4.5.1.6 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

                        4.5.1.7 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... .. . . . . . .17

                        4.5.1.8 Reporting Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

               4.5.2 Test Case 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. . . . . ..18

                        4.5.2.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .18

                        4.5.2.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . .18

                        4.5.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 18

                        4.5.2.4 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . .18

                        4.5.2.5 Setup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . 19

                        4.5.2.6 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . .19

                        4.5.2.7 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . .19

                        4.5.2.8 Reporting Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 20

               4.5.3 Test case 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .20

                        4.5.3.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .20

                        4.5.3.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... .. . . . . .20

                        4.5.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 20

                        4.5.3.4 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . .20

                        4.5.3.4 Setup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . 21

                        4.5.3.5 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .21

                        4.5.3.6 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

                        4.5.3.7 Reporting Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

               4.5.4 Test case 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

                        4.5.4.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .22

                        4.5.4.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

                        4.5.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

                        4.5.4.4 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

                        4.5.4.5 Setup Parameters and Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

                        4.5.4.6 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

                        4.5.4.7 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

                        4.5.4.8 Reporting Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .25

               4.5.5 Test case 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

                        4.5.5.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

                        4.5.5.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

                        4.5.5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . ..25

                        4.5.5.5 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. . . . . .26

                        4.5.5.6 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . .26

                        4.5.5.7 Reporting Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

                  A.      Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..27

                  A.1    Types of Traffic (Specific) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28

                           A.1.1    LCP Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

                                       A.1.1.1     New Session Establishments . . . . . . . . . . . . . . . . . . . . . . . . . . .28

                                       A.1.1.2     Discovery Stage Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28

                                       A.1.1.3     Session Stage Packets . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . ..29

                                       A.1.1.4     Session Teardowns . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . ..30

                           A.1.2    NCP Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .30

                  A.2 Authentication in PPP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

                           A.2.1 Password Authentication Protocol (PAP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

                           A.3.2 Challenge Handshake Authentication Protocol (CHAP) . . . . . . . . . . . . . . . . . . 31

0. Overview

0.1 Introduction

This document provides methodologies for the performance benchmarking of Access Concentrators. It provides methodologies in four areas: throughput, session capacity, session establishment rate, and device stability. In addition to defining the tests, this document also describes specific formats for reporting the results of the tests.

0.2 Term Definition

Below are the definitions of terms used throughout this document.

DUT: Device Under Test. The Access Concentrator that is being tested.
PPP: Point-to-Point Protocol. The Point-to-Point Protocol is designed for simple links which transport packets between two peers. These links provide full-duplex simultaneous bi-directional operation, and are assumed to deliver packets in order. It is intended that PPP provides a common solution for easy connection of a wide variety of hosts, bridges and routers. (RFC 1661)
PPPoE: Point-to-Point Protocol over Ethernet
LCP: Link Control Protocol. In order to be sufficiently versatile to be portable to a wide variety of environments, PPP provides a Link Control Protocol (LCP). The LCP is used to automatically agree upon the encapsulation format options, handle varying limits on sizes of packets, detect a looped-back link and other common misconfiguration errors, and terminate the link. Other optional facilities provided are authentication of the identity of its peer on the link, and determination when a link is functioning properly and when it is failing. (RFC 1661) See Appendix A.1.
NCP: Network Control Protocols. Point-to-Point links tend to exacerbate many problems with the current family of network protocols. For instance, assignment and management of IP addresses, which is a problem even in LAN environments, is especially difficult over circuit-switched point-to-point links (such as dial-up modem servers). These problems are handled by a family of Network Control Protocols (NCPs), which each manage the specific needs required by their respective network-layer protocols. These NCPs are defined in companion documents. (RFC 1661) See Appendix A.2.
Datagram: The unit of transmission in the network layer (such as IP). A datagram MAY be encapsulated in one or more packets passed to the data link layer. (RFC 1661)
Frame: The unit of transmission at the data link layer. A frame MAY include a header and/or a trailer, along with some number of units of data. (RFC 1661)
Packet: The basic unit of encapsulation, which is passed across the interface between the network layer and the data link layer. A packet is usually mapped to a frame; the exceptions are when data link layer fragmentation is being performed, or when multiple packets are incorporated into a single frame. (RFC 1661)
Peer: The other end of the point-to-point link. (RFC 1661)
Session: When an end-to-end PPP connection is established between an end user and the DUT.
Echo-Request: Use definition found in RFC 1575 Sec 3.1
Echo-Reply: Use definition found in RFC 1575 Sec 3.2
PADI: The PPPoE Active Discovery Initiation packet. The Host sends the PADI packet with the DESTINATION_ADDR set to the broadcast address. The CODE field is set to 0x09 and the SESSION_ID MUST be set to 0x0000. The PADI packet MUST contain exactly one TAG of TAG_TYPE Service-Name, indicating the service the Host is requesting, and any number of other TAG types. An entire PADI packet (including the PPPoE header) MUST NOT exceed 1484 octets so as to leave sufficient room for a relay agent to add a Relay-Session-Id TAG. (RFC 2516)
PADO: The PPPoE Active Discovery Offer packet. When the Access Concentrator receives a PADI that it can serve, it replies by sending a PADO packet. The DESTINATION_ADDR is the unicast address of the Host that sent the PADI. The CODE field is set to 0x07 and the SESSION_ID MUST be set to 0x0000.The PADO packet MUST contain one AC-Name TAG containing the Access Concentrator's name, a Service-Name TAG identical to the one in the PADI, and any number of other Service-Name TAGs indicating other services that the Access Concentrator offers. If the Access Concentrator can not serve the PADI it MUST NOT respond with a PADO. (RFC 2516)
PADR: The PPPoE Active Discovery Request packet. Since the PADI was broadcast, the Host MAY receive more than one PADO. The Host looks through the PADO packets it receives and chooses one. The choice can be based on the AC-Name or the Services offered. The Host then sends one PADR packet to the Access Concentrator that it has chosen. The DESTINATION_ADDR field is set to the unicast Ethernet address of the Access Concentrator that sent the PADO. The CODE field is set to 0x19 and the SESSION_ID MUST be set to 0x0000. The PADR packet MUST contain exactly one TAG of TAG_TYPE Service-Name, indicating the service the Host is requesting, and any number of other TAG types. (RFC 2516)
PADS: The PPPoE Active Discovery Session-confirmation packet. When the Access Concentrator receives a PADR packet, it prepares to begin a PPP session. It generates a unique SESSION_ID for the PPPoE session and replies to the Host with a PADS packet. The DESTINATION_ADDR field is the unicast Ethernet address of the Host that sent the PADR. The CODE field is set to 0x65 and the SESSION_ID MUST be set to the unique value generated for this PPPoE session. The PADS packet contains exactly one TAG of TAG_TYPE Service-Name, indicating the service under which Access Concentrator has accepted the PPPoE session, and any number of other TAG types. If the Access Concentrator does not like the Service-Name in the PADR, then it MUST reply with a PADS containing a TAG of TAG_TYPE Service-Name-Error (and any number of other TAG types). In this case the SESSION_ID MUST be set to 0x0000. (RFC 2516)
PADT: The PPPoE Active Discovery Terminate (PADT) packet. This packet MAY be sent anytime after a session is established to indicate that a PPPoE session has been terminated. It MAY be sent by either the Host or the Access Concentrator. The DESTINATION_ADDR field is a unicast Ethernet address, the CODE field is set to 0xa7 and the SESSION_ID MUST be set to indicate which session is to be terminated. No TAGs are required. When a PADT is received, no further PPP traffic is allowed to be sent using that session. Even normal PPP termination packets MUST NOT be sent after sending or receiving a PADT. A PPP peer SHOULD use the PPP protocol itself to bring down a PPPoE session, but the PADT MAY be used when PPP can not be used. (RFC 2516)

0.3 Methodology Definition Format

The following sections define the four tests that can be used to measure the performance of an Access Concentrator.

Definition:

Specific definition of a term

Discussion:

A brief discussion about the term, its application, and any restrictions.

Measurement Units:

The units used to report measurements of this term, if applicable.

0.3.1 Subscriber Session Capacity

Definition:

The maximum number of PPPoE sessions a DUT can establish and maintain.

Discussion:

Session capacity is a function of memory and SHOULD be consistent from one machine to another machine.

Measurement Units:

Number of sessions

0.3.2 Session Establishment Rate

Definition:

The rate at which a DUT can establish sessions without dropping any connection request packet.

Discussion:

For these tests this is the maximum rate at which PPP session establishment requests can be sent to the DUT resulting in 100% successful sessions.

Measurement Units:

Sessions per second

0.3.4 Session Throughput

Definition:

The maximum rate at which none of the offered frames are dropped by the DUT.

Discussion:

The throughput will be tested with different numbers of sessions, packet lengths, and lengths of time.

Measurement Units:

Packets per second

0.3.5 Device Stability

Definition:

The device’s ability to handle traffic patterns. Some will cause the DUT to reach an unstable state meaning that this is beyond the threshold of the DUT’s stability.

Discussion:

There are many ways to test the stability of an Access Concentrator. The following RFC uses five test cases to test stability. The more seconds it takes the DUT to recover from an unstable state, the more unstable it is. The Access Concentrator SHOULD be up and running 99.999% of the time. That leaves about a few minutes of downtime over the year.

Measurement Units:

Seconds

0.4 Test Setup

Setting-up tests described below involves a DUT and a tester with both transmitting and receiving ports as shown in the figure below. Connection requests originate from the sending ports of the tester and are destined to the receiving ports of the DUT. Connection acknowledgements and replies are sent from the sending ports of the DUT back to the tester.

0.4.1 Configuration Being Simulated (Logical)

0.4.2 Configuration In The Lab (Physical)

1. Subscriber Session Capacity

1.1 Objective

To determine the maximum number of PPPoE sessions a DUT can establish and maintain.

1.2 Definition

Session capacity is the maximum number of sessions a DUT can establish.

1.3 Discussion

Several parameters MUST be controlled and clearly defined to develop a test methodology that accurately tests Session Capacity. This methodology MUST be consistent and repeatable. This does not mean that the Session Capacity determined by this methodology will be the same as that listed in the device vendor’s specifications. This test SHOULD determine the maximum number of sessions an DUT can establish and the memory used per session.

1.4 Assumptions

The manufacturer’s specification is available and provides enough information for initial values used in the test.

1.5 Setup Parameters

1.5.1 Authentication

The authentication method used MUST be consistent in test. If you use a static login for everyone, you will use less memory than a dynamic login, which is different for everyone and uses more memory. Also, note where the login is done, it can be done on a radius server or a DUT, which uses more memory. If authentication is done on a radius server it does not affect memory or session capacity. It is important to be consistent with this for comparisons. Also, before beginning each test, do a clean boot to remedy errors caused from the previous test. This will eliminate any instability that MAY affect results.

1.5.2 Session Establishment Rate

It is RECOMMENDED that sessions are established at low rates to eliminate any effect on the maximum number of sessions that can be established. The rate SHOULD be less than the maximum establishment rate. It is RECOMMENDED to start with a rate of 10 sessions per second.

1.5.3 Established Connection Testing

Use Echo-Requests to test all established connections with the DUT. This SHOULD be done after each test (See Appendix).

1.5.4 Traffic

The keep-alive option SHOULD be turned off. This prevents the DUT from sending echo-request packets, which generates additional traffic and complicates the test. The line traffic MUST be controlled, only traffic used to establish sessions and check session connection is allowed. Keep-alive traffic affects results so these parameters MUST be controlled for the results to be repeatable.

1.5.4 Physical Hardware

Before each test, measure the available memory. After the test, re-measure the memory. This information is used to determine how much memory is used for each session for comparisons with other machines. Determine this by dividing the memory used by the number of sessions to get memory per session.

1.6 Procedure

1.6.1 Session Capacity Variables

Initially, start the test with the number or sessions set to 50% of manufacturer’s specification of session capacity or well below what you expect session capacity to be. The increment value is 1000 and the rate is set very low, 10 sessions per second.

1.6.2 Session Capacity Algorithm

Begin the test by trying to establish and validate the initial number of sessions. If this is successful, then increment the test by trying to establish an additional 1000 sessions. If you validate that all the sessions are established, repeat this process until you can not validate that all the sessions are established

When you reach a number where you cannot validate that all the sessions are established, record the last number where you could validate all the sessions. Use this as the starting point and set the increment to 10 and repeat the above process until you again cannot validate all the sessions.

The final step will be to set the number of sessions to the last number where you could validate the established session. Use this as the starting point and set the increment to 1 and repeat the process in the first paragraph of this section until you again cannot validate all the sessions. The last number of sessions that could be established and validated in the test is you session capacity.

1.7 Measurements

The session capacity is measured by finding the maximum number of sessions established and validated. Measurements of the memory MUST be done before and after the test to determine the amount of memory used per session. It is important to note the authentication method for testing since it impacts the memory and the results of the test.

1.8 Reporting Format

The report format will be a number for the maximum session capacity and a number for the amount of memory used per session.

2. Session Establishment Rate

2.1 Objective

To determine the maximum rate that the DUT can establish sessions.

2.2 Definition

Session establishment rate is the rate at which a device can establish sessions. In this testing environment, the session establishment rate of the DUT is assumed to be equal to the rate of the PADI being sent without any loss. Therefore, to measure the maximum session establishment rate of the DUT, the rate of the PADI’s being send that results in no packet loss is measured.

2.3 Discussion

To establish a PPP connection between a user device and a DUT, the two devices have to go through four states: discovery stage, data link layer setup, authentication, and network layer setup. In the discovery stage, a user device starts with broadcasting an initiation packet (PADI) to look for a DUT. The DUT then replies using an offer packet (PADO). Upon receiving the PADO, the user device and the DUT communicate to provide the device a session ID and a MAC address. After this stage, the two peers proceed to the data link layer setup. Here, the two peers exchange LCP config-req, config-nak, config-rej, and config-ack packets to negotiate data link layer options (see RFC 1661 for more detailed information). Once the data link layer is setup, the peers have to go through an authentication process. In this process, the DUT verifies to see if the other peer is a valid user or not. Finally, when the authentication is successful, the two peers exchange IPCP packets to set up the network layer. Once this network layer is set up, the peers can start transferring data. (See the A.1 for detailed structures of the LCP and NCP packets).

2.4 Assumptions

Some assumptions SHOULD be made prior to doing this session establishment rate test. These assumptions are listed below:

Assume that the maximum session capacity of the DUT is determined.
Assume that there is no packet loss due to line faults.

2.5 Setup Parameters

The following parameters affect the outcome of the test and they MUST be defined.

N - Is the number of PADI packets to be sent to the DUT. This number SHOULD be initialized to the highest known number that the DUT is able to handle (not the highest specified by the vendor). For this test, this N is initialized to 5.
Time-Out - This is the time length the tester SHOULD wait before it assumes that the PADI is dropped by the DUT. This time-out SHOULD be set to one minute.
Sending Time - This is the total transmission time spent to send all the N number of PADI's. This number SHOULD be set to 1 second.
Request Rate - This is the rate at which the PADI's are sent to the DUT. A number of PADI's SHOULD be sent within a fixed period of time. The PADI's MAY be sent in a constant or in a bursty fashion.
Buffer Queue - The DUT SHOULD have a finite amount of buffer to store the incoming request packets. In general, the more memory is assigned to the buffer, the less memory is available for the session capacity memory. For this test, the buffer space is set to the default value.
Keep-Alive Packets – These packets are used to make sure that an established session is still up. In this test, we assume that once a session is established, the connection will stay up for the duration of the session. Therefore, no keep-alive packets will be exchanged.
Established Session Time – This is the time from when the first PADI packet leaves the test port to when the last IPCP connection acknowledgement packet is received.

2.6 Procedure

Two types of tests MUST be performed. One is to determine the maximum session establishment rate when the PADI's are sent at a constant rate. The other one is to determine the maximum session establishment rate when the PADI's are sent in a bursty fashion.

2.6.1 Measuring Session Establishment Rate with Constant PADI's

From the test port, initiate the sessions by sending out an N number of PADI packets at a constant rate for one second.
As soon as the last PADI packet is sent, start the Time-Out timer.
When the timer expires, record the number of successful established sessions by counting the number of IPCP Config-Ack packets received from the DUT.
If the number of established sessions is equal to N, increase N by 5 and restart the test.
If the number of established sessions is less than N, decrease N by 1 and restart the test. Keep decreasing N by 1 and restart the test until the number of established session is equal to N. The final N is the maximum number of connections the DUT can establish in one second.

2.6.2 Measuring Session Establishment Rate with Bursty PADI's

From the test port, initiate the sessions by sending out an N number of PADI packets for one second in a bursty fashion. There SHOULD be 5 packets in each burst. The space between each burst is 1 microsecond.
As soon as the last PADI packet is sent, start the Time-Out timer.
When the timer expires, record the number of successful established sessions by counting the number of IPCP Config-Ack packets received from the DUT.
If the number of established sessions is equal to N, increase N by 5 and restart the test.
If the number of established sessions is less than N, decrease N by 1 and restart the test.
Keep decreasing N by 1 and restart the test until the number of established session is equal to N. The final N is the maximum number of connections the DUT can establish in one second.

2.7 Measurements

Establishment Request Rate - This is the N number of PADI sent in 1 sec.

Percentage of Successful Sessions - This number is obtained by dividing the number of successful sessions by the number of PADI sent.

2.8 Reporting Format

The format SHOULD be in the form of graphs. There SHOULD be two separate graphs: One for constant back to back PADI rate and the other one for the bursty PADI rate. The x-axis is the Establishment Request Rate, and the y-axis is the corresponding percentage of successful established sessions. The point where the Establishment Request Rate is highest and the percentage of successful connection is still 100% is the maximum establishment session rate.

Below is a sample graph of how the result SHOULD look like.

3. Session Throughput

3.1 Objective

To determine the maximum rate at which none of the offered frames are dropped by the DUT.

3.2 Definition

In terms of data transmission, throughput is the amount of data successfully transmitted from access ports to network ports, under the assumption that each access port is the same speed. More formally defined, throughput is the maximum rate at which the number of frames transmitted to the DUT is equal to the number of frames received by the DUT. It is also important to note that the forwarding rate and packet loss test are distinctly different from the throughput test in that the throughput test is more precise because any packet loss has a large impact of the algorithm determining the various offered loads to use in the test. A test resulting in 100% throughput essentially means 0% packet loss rate, however the two are not complements of each other.

3.3 Discussion

For all of the unidirectional performance tests, traffic flows from transmitting tester ports to the DUT and back to receiving tester ports. The number of ports the traffic is sent to and from depends on the number of sessions already established (this assumption is made before the throughput test is run). If the number of frames received is less than the number of frames transmitted, then packet loss has occurred, in which case, the test is repeated using a reduced offered load. If the number of frames received is equal to the number of frames transmitted, then the test is repeated with an increased offered load. The test is repeated with varying offered loads until a maximum frame rate with no loss is determined. Throughput MAY be measured for Ethernet-to-Ethernet, Fast Ethernet-to-Fast Ethernet, or Gigabit Ethernet-to-Gigabit Ethernet.

3.4 Assumptions

When testing throughput, certain assumptions SHOULD be made prior to testing and these are listed as follows:

Assume frames can be sent in bursts of traffic or continuously.
The maximum rate SHOULD never exceed the rate of the receiving trunk port.
Assume sessions have already been established.
Assume tests will be done on full duplex, since half duplex is only used on 10Mgb lines and most of these DUT’s do no less than a gig on Ethernet.
Assume an actual physical pipe size is being used for the both access port and trunk port.
Assume Keep-Alives are turned off however it is recommended that Echo-Requests packets are occasionally sent to host to determine if the session state is still up after certain throughput tests have been run with large offered load.

3.5 Setup Parameters

Frame Size -The following IP Frame sizes SHOULD be used on PPPoE (Include the maximum and minimum frame sizes permitted by the Ethernet standard and a selection of sizes between the two extremes: 64,128, 256, 512, 1024, 1492 bytes.
Direction of Traffic - Can be downstream or upstream. Each direction will be handled by performing the reverse operations of each other.
Packet Count - This parameter will help determine the duration of the test. Packet count SHOULD be a multiple of the number of user sessions established to ensure that all user sessions have equal number of packets in transmission during testing.
Burst Count - Count of one is continuous and count of n signifies burst count of size n.
Number of Sessions - The number of established sessions, which need to be the same on each port. Throughput could possible be affected by this parameter and could actually decrease if a large number of sessions are established.

3.6 Procedure

Create traffic with N streams as uni-directional flows from one or more of the test device's source ports, through the DUT.

Transmit the initial load of packets (this value is half of what the max load is claimed to be) at a specific rate through the DUT from all sources to all destinations, making sure that each packet transmitted is distinctly flagged so that it will be counted at the other end. Identifying each packet ensures that no duplicate packets will be counted which will effect the results by making packet count too high. Once an iteration of transmission has been completed, and after some delay interval between transmitting packets and counting them passes, then the measurement of the number of received frames SHOULD be taken. It is recommended that Echo Request packets be sent between iterations in order to confirm that the sessions are still up.

If the count of offered frames is greater than the count of the received frames, then fewer frames are received than were transmitted. The rate of the offered stream is reduced lower than the failed rate by half the difference between previous load value and current load value.

The third trial and subsequent trials make use of the same algorithm to continue to increase or decrease the offered load.

When all of the transmitted frames are received by the receiving port, then no further trials are attempted, and maximum frame rate is recorded as throughput.

3.7 Results

In general, throughput tests stress the DUT’s forwarding engine under extreme conditions involving long bursts of traffic. Ideally, a DUT has the throughput to be fully non-blocking for all packet sizes and all stream counts; however, in practice there is little reason to avoid DUT’s with throughputs in the 90% range.

Each trial iteration results in a record of the total number of packets transmitted and received. For the successful and last trial, the packet rate and percentage of maximum rate for a given packet size is recorded. The results of the throughput test SHOULD be reported in graph format where the x coordinate SHOULD be the frame size, and the y coordinate SHOULD be the frame rate. There SHOULD be at least two lines, actual and theoretical on the graph. Text accompanying the graph SHOULD indicate the protocol, data stream format, and type of media used in the tests.

4. Device Stability

4.1 Objective

To measures the instability of the DUT with various traffic patterns

After this state is reached, the tests measure the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more severe the instability.

4.2 Definition

The instability of the DUT is measured based on how long to reach a stable state from an unstable state. Once an unstable state is reached, this is defined as a state where packets, sessions, or session requests are being dropped that should not be, the DUT is configured in such a way that it backs off to a known stable state. The severity of the instability is measured by the time, in seconds, that it takes for the DUT to recover without outside interference.

4.3 Discussion

Five test cases should be used to test the stability of a DUT. Four of the five test cases test different forms of instability. The fifth test case tests the DUT's stability with an internet/"real-world" traffic model.

4.3.1 Types of Traffic That Will Be Used

4.3.1.1 LCP Traffic

New Session Establishments will be used to create sessions and also when testing the results of invalid session requests. These session establishment requests can be valid, or invalid based on the parameters selected. See Appendix A for more details.

Session Teardowns will be used to teardown sessions and also when testing the results of invalid session teardown requests. These session teardown requests can be valid or invalid based on the parameter, bursty, steady state requests, or be due to keep-alive timeouts. See Appendix A for more details.

Re-Negotiation of Established Sessions will be used to test how the DUT handles re-negotiation under different circumstances. These session Configure-Requests can be valid or invalid based on the parameters, bursty, or steady state requests. For more information on this please review section 0.1.2 of this document.

4.3.1.2 NCP Traffic

To test how the sessions handle traffic flowing on them run varying amounts of valid and invalid NCP traffic on established sessions. Make sure you start at the minimum frame size and increment up to the maximum frame size. Make sure that you very the throughput on all established sessions upstream and downstream. For more information please review section 0.1.2 of this document.

4.4 Setup Parameters and Assumptions

This varies depending on which of the following test cases you are running. Refer to the individual test cases for what parameters are set and what the assumptions are when running the case.

4.5 Test Cases

Below are the test cases to be performed

4.5.1 Test case 1

4.5.1.1 Objective

To measure the unstable state of the DUT where sessions are no longer being created.

4.5.1.2 Definition

Once an unstable state is reached, the test measures the severity of the unstable state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.5.1.3 Discussion

When a DUT reaches a state where the number of valid requests and the number of current sessions out number the maximum number of sessions, the behavior of the DUT is unknown, and therefore unstable since some of these connections will have to be denied. This is going to be accomplished by creating sessions so fast that some the sessions no longer get created that are requested.

4.5.1.4 Assumptions

Some assumptions should be made prior to doing this session establishment rate test. These assumptions are listed below:

Assume that the maximum session capacity of the DUT is determined.
Assume that the maximum session establishment rate of the DUT is determined
Assume that there is no packet loss due to line faults.

4.5.1.5 Setup Parameters

The following parameters MUST be defined with the listed considerations:

Packet Size: The packets being sent are LCP packets, so the size would reflect such. Suggested sizes to reach are listed in RFC 2544 and Appendix B.
Number of Sessions not being created: This will fluctuate as time goes on. It should start at zero, once it gets to denoted limit before the back off is performed and the value should go back to zero.
Test Duration: The test should run until instability is reached and recovered from. We assume that instability will eventually be reached and recovered from.

Other packet options for creating sessions are in Appendix A.

4.5.1.5 Procedure

From the test port, begin the test by sending out PADI packets at 50% of the session establishment rate at a constant rate.
As the sessions are being established, this can be noted by the Session ID on the IPCP Config-Ack packets received from the DUT, send Echo-Requests to all of the established connections with the DUT.
Once 50% of the Session Subscriber Capacity is reached, start transmitting PADI packets at 110% of the maximum session establishment rate.
If the session capacity is reached, teardown 20% of the sessions and then continue sending PADI packets as before.
Once the PADI packets sent do not result in corresponding IPCP Config-Ack packets, start the timer.
Send (Session Subscriber Capacity – established session) PADI packets to the DUT at 95% Session Establishment Rate. If the maximum number of sessions are established, teardown 10% of the sessions, and then send that many PADI packets to the DUT at 95% Session Establishment Rate.
Once the number of PADI packets sent equals the number of Config-Acks received, stop the timer. This can be determined by verifying that no Config-Naks are received.
You SHOULD repeat this test 20 times.

4.5.1.6 Measurements

Total Number of Sessions Created – The total number of sessions that can be created and accept traffic on the DUT.
Maximum Rate To Create Sessions – The maximum rate that all PADI session requests result in sessions being setup on the DUT without dropping existing sessions.
Number of Drops Before Back Off – The number of sessions dropped and the number of PADI requests that did not result in a Config-Ack.
Time To Get Back To A Stable State – The number of seconds that it took the DUT to return to a predictable behavior where the number of valid sessions requested is the same as the number accepted, and the already established sessions can continue to send traffic.

4.5.1.7 Reporting Format

Plot the session creation rate vs time. You MAY also run this test and plot the number of sessions created vs. time (getting rid of max limit).

The graphs SHOULD look something like this:

NOTE: This is an approximation

The ridge in the graph shows where stability was lost. We backed off to the max right before that, so the rate should go back up to this max, and be a straight line the rest of the time. The difference in the x-axis is the recovery time.

4.5.2 Test Case 2

4.5.2.1 Objective

To measure the unstable state of the DUT where sessions are being torn down early.

4.5.2.2 Definition

When an unstable state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more severe the instability.

4.5.2.3 Discussion

When a DUT reaches a state where the sessions that are up and working start to be torn down when they should not be, the behavior of the DUT is unknown. When the behavior of the DUT is unpredictable it is said to be unstable. The sessions that were dropped are still in use and now need recreated so that traffic can continue to be transmitted. In order to simulate this, excess traffic will be sent on some of the sessions. This will cause some of the session keep-alives to get lost resulting in active sessions being torn down early.

4.5.2.4 Assumptions

Some assumptions should be made prior to doing this session establishment rate test. These assumptions are listed below:

Assume that the maximum session capacity of the DUT is determined.
Assume that the maximum session establishment rate of the DUT is determined
Assume that there is no packet loss due to line faults.

4.5.2.5 Setup Parameters

The following parameters MUST be defined with the listed considerations:

Packet Size: The packets being sent are NCP packets, so the size would reflect such. Suggested sizes to reach are listed in RFC 2544 and Appendix B.
Maximum Number of Sessions: The maximum number of sessions that the DUT can handle is required for this test. If it is not known run the Session Capacity test above, and wait a cool off period so that all of the memory is freed before running the test.
Line Rate: The speed if packets were sent back to back on the line ignoring errors. This is a function of the media being used.
Number of Sessions Dropped: This will fluctuate as time goes on. It should start at zero, once it gets to denoted limit before the back off is performed and the value should go back to zero.
Test Duration: The test should run until instability is reached and recovered from. We assume that instability will eventually be reached and recovered from.
Other packet options for creating sessions are in A.1

4.5.2.6 Procedure

From the test port, begin the test by creating 10% of the maximum number of sessions on the DUT. Do this by sending out PADI packets for each connection to be created at 50% of the session establishment rate at a constant rate.

Begin to send a small amount of NCP traffic (25% of the throughput) on the existing sessions. Keep-alives should be turned on and should be transmitted for each session.

Increase the amount of NCP traffic until sessions start to get dropped. Since we are not sending out any session termination packets, this is unexpected, and therefore considered a sign of instability.

Start the timer.

Send out new PADI’s for each session that was dropped. Back off to the amount of NCP traffic being transmitted on the lines to the point right before the instability occurred. Continue sending out PADI packets until all of the sessions are back up running at this known stable rate.

Stop timer when the sessions are not dropping.

You SHOULD run this test 20 times.

4.5.2.7 Measurements

Traffic rate – The rate at which NCP traffic is being sent on all of the sessions.
Maximum Number of sessions – The maximum number of sessions that the DUT can create and maintain.
Number of Sessions Dropped – The number of PPP sessions that were dropped during the unstable period of time.
Burstiness – The irregularity in the rate that the NCP traffic was sent at.
Time To Get Back To A Stable State – The number of seconds that it took the DUT to return to a predictable behavior where any valid number of sessions can send traffic without effecting and eventually dropping other sessions.

4.5.2.8 Reporting Format

Plot the traffic rate vs time. You MAY also run this test and plot burstiness vs. time.

The graphs SHOULD look something like this:

NOTE: This is an approximation

The ridge in the graph shows where stability was lost. We backed off to the max right before that, so the rate should go back up to this max, and be a straight line the rest of the time. The difference in the x-axis is the recovery time.

4.5.3 Test case 3

4.5.3.1 Objective

To measure the unstable state of the DUT where session throughputs have been reduced below an acceptable range.

4.5.3.2 Definition

When an unstable state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.5.3.3 Discussion

When a DUT reaches a state where the throughput is under 10% of the requested throughput, the behavior of the DUT is unknown, and therefore unstable since some of these connections will have to be denied. This assumes that the requests are not unreasonable, and nobody is trying to send beyond the maximum throughput.

4.5.3.4 Assumptions

Some assumptions should be made prior to doing this session establishment rate test. These assumptions are listed below:

Assume that the maximum session capacity of the DUT is determined.
Assume that the maximum session establishment rate of the DUT is determined
Assume that the maximum device and session throughput on the DUT is known.
Assume that there is no packet loss due to line faults.

4.5.3.4 Setup Parameters

The following parameters MUST be defined with the listed considerations:

Packet Size: The packets being sent are NCP packets, so the size would reflect such. Suggested sizes to reach are listed in RFC 2544 and Appendix B.
Maximum Number of Sessions: The maximum number of sessions that the DUT can handle is required for this test. If it is not known run the Session Capacity test above, and wait a cool off period so that all of the memory is freed before running the test.
Line Rate: The speed if packets were sent back to back on the line ignoring errors. This is a function of the media being used.
Maximum Throughput: The max speed packets can be sent at before drops start to occur.
Number of Sessions Dropped: This will fluctuate as time goes on. It should start at zero, once it gets to denoted limit before the back off is performed and the value should go back to zero.
Test Duration: The test should run until instability is reached and recovered from. We assume that instability will eventually be reached and recovered from.

Other packet options for creating sessions are in Appendix A.

4.5.3.5 Procedure

Send all session establishment traffic at 50% of the maximum session establishment rate.
Create 50% of maximum number of sessions the DUT can handle by sending out PADI requests to start the sessions initiation.
Randomly start sending:
a.       New valid session requests to the DUT, about 5% of the maximum number of sessions the DUT can handle
b.       Invalid session requests to the DUT, about 5% of the maximum number of sessions the DUT can handle.
c.       Valid session teardown requests for the sessions that are being created randomly.
Start to send NCP traffic so that the request throughput is equal to 5% of the max throughput.
Increase the requested throughput but 5% until, the DUT cannot handle all of the data on the line, and the throughput drops down to where throughput seen is less then 50% of the requested throughput.
If the maximum throughput is reached before reaching an unstable state increase the requested valid and invalid random sessions creation/teardowns, and start sending NCP traffic so that the request throughput is equal to 5% of the max throughput.
Start the timer
Stop sending random traffic, and set the requested throughput to 90% of the maximum throughput of the device.
Stop timer when the throughput is within acceptable ranges again
You SHOULD run this test 20 times.

4.5.3.6 Measurements

Traffic rate – The rate at which NCP traffic is being sent on the sessions and on the give physical line.
Burstiness – The irregularity in rate that NCP and LCP traffic is sent.
Max throughput – The maximum amount of data that the DUT could handle in a second.|
Current throughput – The throughput that was being requested when the DUT became unstable.
Time To Get Back To A Stable State – The number of seconds that it took the DUT to return to a predictable behavior where the actual throughput was at least 50% of the request throughput.

4.5.3.7 Reporting Format

Plot the throughput vs. time.

The graphs SHOULD look something like this:

NOTE: This is an approximation

The ridge in the graph shows where stability was lost. We backed off to the max right before that, so the rate should go back up to this max, and be a straight line the rest of the time. The difference in the x-axis is the recovery time.

4.5.4 Test case 4

4.5.4.1 Objective

To measure the unstable state of the DUT where jitter associated with each of the existing sessions becomes outside of tolerated limits.

4.5.4.2 Definition

After this state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.5.4.3 Discussion

When a DUT reaches a state where the jitter of the sessions is varying by more then X%, the behavior of the DUT is unknown, and therefore unstable since some of these connections will have to be denied.

4.5.4.4 Assumptions

Some assumptions should be made prior to doing this session establishment rate test. These assumptions are listed below:

Assume that the maximum session capacity of the DUT is determined.
Assume that the maximum session establishment rate of the DUT is determined
Assume that an unreasonable jitter is one where the round trip time does not vary by more then plus or minus 10%.
Assume that the maximum device and session throughput on the DUT is known.|
Assume that there is no packet loss due to line faults.

4.5.4.5 Setup Parameters and Assumptions

The following parameters MUST be defined with the listed considerations:

Packet Size: The packets being sent are NCP packets, so the size would reflect such. Suggested sizes to reach are listed in RFC 2544 and Appendix B.
Maximum Number of Sessions: The maximum number of sessions that the DUT can handle is required for this test. If it is not known run the Session Capacity test above, and wait a cool off period so that all of the memory is freed before running the test.
Line Rate: The speed if packets were sent back to back on the line ignoring errors. This is a function of the media being used.
Maximum Throughput: The max speed packets can be sent at before drops start to occur.
Number of Sessions Dropped: This will fluctuate as time goes on. It should start at zero, once it gets to denoted limit before the back off is performed and the value should go back to zero.
Test Duration: The test should run until instability is reached and recovered from. We assume that instability will eventually be reached and recovered from.

Other packet options for creating sessions are in Appendix A.

4.5.4.6 Procedure

Send all session establishment traffic at 50% of the maximum session establishment rate.
Create 50% of maximum number of sessions the DUT can handle by sending out PADI requests to start the sessions initiation.
Randomly start sending:
(a)     New valid session requests to the DUT, about 5% of the maximum number of sessions the DUT can handle.
(b)     Invalid session requests to the DUT, about 5% of the maximum number of sessions the DUT can handle.
(c)     Valid session teardown requests for the sessions that are being created randomly.
Start to send NCP traffic so that the request throughput is equal to 5% of the max throughput. Measuring the jitter of each packet as it moves through the DUT.
Increase the requested throughput but 5% until, the DUT cannot handle all of the data on the line, and the jitter begins to vary by 10% in a group of 1,000 NCP packets.
If the maximum throughput is reached before reaching an unstable state increase the requested valid and invalid random sessions creation/teardowns, and start sending NCP traffic so that the request throughput is equal to 5% of the max throughput.
Start the timer
Stop sending random traffic, and set the requested throughput to 90% of the maximum throughput of the device.
Stop timer when the jitter varies less the 10% per group of 1,000 NCP packets.
You SHOULD run this test 20 times.

4.5.4.7 Measurements

Traffic rate – The rate at which NCP traffic is being sent on the sessions and on the give physical line.
Burstiness – The irregularity in rate that NCP and LCP traffic is sent.
Max throughput – The maximum amount of data that the DUT could handle in a second.
Current throughput – The throughput that was being requested when the DUT became unstable.
Amount of jitter – The time that it took each packet to get from the sending port on the test device to the DUT and
back out to the receiving port on the test device.
Time To Get Back To A Stable State – The number of seconds that it took the DUT to return to a predictable behavior where the jitter is 10% within a 1,000 packet section of the session traffic.

4.5.4.8 Reporting Format

Plot the jitter vs time.

The graphs SHOULD look something like this:

NOTE: This is an approximation

The ridge in the graph shows where stability was lost. We backed off to the max right before that, so the rate should go back up to this max, and be a straight line the rest of the time. The difference in the x-axis is the recovery time.

4.5.5 Test case 5

4.5.5.1 Objective

To measure the stability of the DUT under "real world" traffic circumstances. The DUT should never reach an unstable state, but if it does, the severity should be very low.

4.5.5.2 Definition

“Real World” traffic circumstances are when sessions are constantly being torn down and being initialized. There are a few peek times in the day as can be witnessed on the graph further down.

4.5.5.3 Discussion

Many DUTs can pass individual tests in the lab, but in a "real world" traffic setting, the DUT may reach an unstable state. Since the goal of the DUT, is to be installed, and never be touched again, unless some new configuration is wanted, the DUT needs to be stable. If an unstable state is reached, the DUT MUST has to become stable in a very short amount of time. Anytime the DUT is unstable in the field, it costs the buyer money.

The goal is to mimic Internet traffic. Internet traffic, from the perspective of sessions, and the DUT, will look like this:

The x-axis is time of day starting at midnight and ending at midnight.

4.5.5.5 Procedure

The general procedure is to allow 10% of the links to time out randomly because of no, purposeful, keep-alives. This is simulating users logging off, but the session not getting torn down until the keep-alive response is missed for a while.

We will traverse time, creating sessions and tearing them down based on the given chart. The traffic running on these sessions will be a variety of small queries like e-mail to big downloads. The size of the traffic will be randomly generated.

Run the test for 24 hrs, to see at what points the DUT becomes unstable and/or unable to recover.

Record the traffic patterns being sent at the time of instability so that more research can be done later on this.

You MAY wish to repeat this test 7 times in a row without clearing the DUTs memory and connections. This will give an idea of how it performs during the week.

4.5.5.6 Variables

Number of Connections – Make sure that statistics are kept on the number of connections that are active at any one time. This should be the same as the graph given above.

Number of Connection retries – The number of connections that did not get initiated on the first try. Record the number of sessions that needed retries, and the number of retries.

Burstiness of source – The variability in the rate at which traffic is sent to the DUT.

Throughput on sessions – The amount of traffic that the DUT handled in a given second on the given links.

Jitter through the sessions – The amount of variability in the throughput over time.

Time that stability was lost – The time of day that some form of DUT instability was seen. Make sure to also record why type of instability it was:

Throughput dropped below a reasonable level

The jitter was too great

Sessions could not be established

Sessions were being torn down early

Time stability was regained – The time of day that the DUT regained stability.

4.5.5.7 Reporting Format

Plot Variables vs. time for the entire 24 hr period. You MAY wish to run this test case 7 times back to back to simulate an entire weeks worth of traffic.

A. Appendix

A.1 Types of Traffic (Specific)

A.1.1 LCP Traffic

A.1.1.1 New Session Establishments

To configure a new PPP session, one peer (host) will send out a Configure-Request Packet. Before this is done, the host needs to find out some information about the server. The stage in which this is done is called the discovery stage. This stage starts out with the host sending out a PADI, waiting for a PADO response. Once the PADO packet is received, it sends out a PADR packet waiting for a PADS packet to respond. These PADR and PADS packets are to set up the MAC Address (for Ethernet) and the Session ID number.

A.1.1.2 Discovery Stage Packets

Below are the Discovery Stage packet formats and options:

A.1.1.3 Session Stage Packets

The following are PPP session stage packets and options.

Parts of the Packet:

Code - Which LCP Packet is being sent:

Identifier - Must be changed when the data inside of request changes, or a response with that number is received. It is used to match requests with replies.

Options - List of Configurations option:

Structure -

A.1.1.4 Session Teardowns

There are three ways to teardown a session:

1. If you the host does not respond to the keep alive message

2. If a PADT is sent (Discovery Stage)

3. If a Terminate Request and Terminate Ack are sent/received

Everything is the same as in Session Establishment, except there are no options, and the codes are as follows:

Re-Negotiation of Established Sessions -This is the same as the Session Establishment PPP session stage. Send the following LCP packets with the appropriate codes and options to re-negotiate different parameters.

A.1.2 NCP Traffic

Normal Network Layer Traffic:

A.2 Authentication in PPP

PPP can support authentication operations when the establishing state is successful. It can be requested by either peer. In this test, the DUT is the one that SHOULD request an authentication before each session. Below are the discussions of two most common authentication protocols.

A.2.1 Password Authentication Protocol (PAP)

PAP is the simplest form of authentication that is supported by PPP. It is implemented with the traditional username and password method. Authentication takes place when one device requests an authentication from its peer. The other peer then replies with both the name and password in a single transaction. The requesting peer will then validate this information and then replies with a positive or negative acknowledgement. This acknowledgement can include readable texts such as “Permission Granted” or “Access Denied.”

When the PPP frame is carrying a PAP packet, the protocol field has the value 0xc023. The PAP packet code field identifies the PAP message. The authentication-request packet, which has the code 0x01, includes fields allocated for a username and password. The authentication-ack and authentication-nak packets are used to notify an authentication success or a failure.

A.3.2 Challenge Handshake Authentication Protocol (CHAP)

CHAP is another form of PPP authentication that requires a challenge and a response. This authentication starts when one peer (challenger) challenges its other peer (client peer) with its CHAP name and a random string. The client peer needs to transform this random string using an algorithm and a CHAP secret key, and then return the result with its own name. The challenger evaluates the reply by doing the same computation with its own copy of the secret key. If the results are the same, it sends a success acknowledgement to its client peer. Otherwise, it sends a failure acknowledgment instead.

When the peers negotiate a CHAP authentication, and LCP configure-request packet carries the authentication protocol 0xc233 option. This option has an extra octet for specifying the CHAP algorithm. An example of a CHAP algorithm is Message Digest 5 (MD5), which has the code 0x05.

Once both peers have agreed to use a CHAP authentication, they exchange CHAP packets which include four different messages. Challenge (code 0x01) and response (code 0x02) packets both carry an identifying CHAP name, representing the packet sender and a string necessary for the authentication process. The challenger sends either a success (code 0x03) or a failure (code 0x04) packet at the end of the process. Like PAP, the acknowledgement packets may include explanatory texts.