Library: Test Plans

Quality of Service (QoS) Test Plan

1. Overview

1. Layer 2-3 Stateless QoS Functional Test
2. Layer 4-7 Stateful QoS Functional Test
3. Triple Play QoS Functional Test
4. Conclusion

QoS Overview and Deployment

The implementation of Quality of Service mechanisms is a critical element to solving the problems of handling the increasing volume and disparate types of traffic seen on today's public and private networks. Whenever a mission critical application - whether voice, video, or data - is being delivered over a network, Quality of Service traffic shaping and policing can assist network managers in maintaining the best network performance and good-put for applications during periods of network congestion.

QoS offers the following core principles:

• Classification and Marking
• Policing
• Queuing and Dropping

Both Class of Service (CoS) and ToS IP Precedence (or DSCP) are Quality of Service methods used to manage the network during congestion.

Class of Service allows the network administrator to identify the order of priority that the incoming traffic be processed for transmission when the switch or network is congested. As an example, a weighted round robin (WRR) method prevents head-of-line blocking. Network administrator can setup the queues to match various business requirements and priority levels. The IEEE 802.1p standard recommends the Class of Service traffic categories described in Table 1.

Priority	Type of Traffic	Equivalent Application Traffic
0 (lowest)	Best effort	Ordinary LAN priority traffic
2	Background	File transfers, games, AIM
2	(spare)	Not used
3	Excellent effort	For critical users applications
4	Controlled load	For important applications
5	Video, < 100ms latency and jitter	Video application or mix with Voice
6	Voice, < 10ms latency and jitter	Voice application
7 (highest)	Network control	Critical traffic to maintain network

Table 1. Class of Service traffic parameters

IP Precedence allows the network administrator to prioritize IP traffic. Using the ToS IP Precedence or DSCP fields, the network administrator can design the edge switch or router to effectively classify, mark, and process packets according to the application requirements. Most critical application traffic takes precedence over any routine traffic processing. The IP Precedence consists of 3 bits, while the DSCP consists of 6 bits (64 possible values).

Ixia's IP testing solutions can help meet the challenging requirements for QoS testing and assist in the benchmarking and pre-deployment analysis of network devices and systems. Solutions are provided with Ixia's Layer 2/3 IxScriptMate and IxExplorer applications, as well as the Layer 4 based IxChariot stateful network traffic emulation tool. Examples of the usage of these applications to execute QoS test cases are described in this document.

This test plan provides a framework and structure for custom test plan development to meet individual test requirements.

1. Layer 2-3 Stateless QoS Functional Test

Objectives

Measure the baseline performance of the DUT with and without QoS when stateless traffic is injected into the network. Stateless traffic is Layer 2 -3 data and does not emulate true user application traffic. This test verifies that the latency and the packets loss on the egress traffic change to much lower values when QoS is enabled on the receiving DUT. Without enabling QoS both the latency and packets loss are much higher. The 1st step is to take measurements and collect statistics when QoS is disabled on the DUT. The 2nd step is to take measurement and collect statistics when QoS with IP Precedence classifying and marking is enabled on the DUT.

Setup

The baseline setup for this test requires four test ports. Three ports are used to generate Layer 3 traffic connected to three DUT ports. These connections are considered the ingress ports to the DUT or the network. Each port carries a separate stream with a specific IP Precedence marked value. The fourth test port is connected to a fourth DUT port to evaluate the outgoing network traffic (egress) based on the QoS service characteristics and settings. See Figure 1.

Ixia's IxScriptMate Many-to-One test can help setup and execute this test case.
Figure 1. Many-to-One QoS test setup

Input Parameters

Two sets of parameters are required prior to running the Layer 2/3 QoS functional test. One set of parameters are for the test tool and the other for the DUT. Table 2 lists the test parameters.

Parameters	Description
Frame size	Packet frame size can bet set for fixed or random
Duration	Test Duration to run ranges from hours down to seconds.
Traffic Rate	Traffic rate per priority level
DUT- QoS	Administrative DUT QoS setting (enabled or disabled)
DUT-Line speed	The link / interface speeds of the DUT ports
DUT-QoS type	DUT QoS type settings: COS, ToS IP Precedence, or DSCP
DUT-QoS Policies	DUT QoS Policies applied to the ingress traffic
DUT-Queue type	Queuing mechanism such as Weighted Random Early Detection (WRED) and Weighted Round Robin (WRR) Queuing

Table 2. QoS Input Parameters Table

Methodology

TEST 1- QoS is disabled on the DUT

1. With QoS is disabled on the DUT, configure the network according to Figure 1.
2. Setup the simulated traffic rate per type. Refer to Figure 2. Refer to Table 2 for input parameters required to set up the traffic and the test.
3. Start the traffic and run for the test duration.
4. The traffic is received by the DUT and is not prioritized or classified. See Figure 3. Note the latency measurements and values and the packet loss percentage in Figure 4 and 5.

TEST 2- QoS is enabled on the DUT

1. Enable QoS on the DUT and rerun the same test.
2. The traffic that is received (ingress) by the DUT is classified, prioritized and processed accordingly. The resultant traffic (egress port) is measured for Packet Loss, and Latency.
3. Observe the new latency measurements and the packet loss (refer to Figures 6, 7 and 8)

Figure 2. Test Setup

Results

The results show some packet loss but no particular order in latency is shown when QoS is disabled. See Figure 3. The lower priority traffic (priority 0) is still shown the highest packet loss.


Figure 3. log with QoS disabled on the DUT

Figure 4.Receive and Loss rate for all three streams

Figure 5.Latency for all three streams

Figure 6. IxScriptMate log with QoS enabled on the DUT

Figure 7.Receive and Loss rate for all three streams

Figure 8.Latency for all three streams

In addition, IxScriptMate also provides the user additional reporting (PDF format) with various types of charts, including csv file available for analysis. Below is a one page sample report (see Figure 9) that was generated based on this test case execution with various frame sizes and a few trials.
Figure 9. IxScriptMate report sample

2. Layer 4-7 Stateful QoS Functional Test

Objective

Measure the baseline performance of the DUT with and without QoS when stateful traffic is injected into the network. Stateful traffic is of Layer 4-7 type data and it represents user application traffic with Layer 4 traffic emulation. This test verifies that during network congestion condition and when QoS is enabled on the DUT, the stateful traffic is handled much more efficiently on the most delay and traffic loss sensitive traffic. The 1st step is to take measurement and collect packets statistics and application response time when QoS is disabled on the DUT. The 2nd step is to do the same and collect application response time when QoS with IP Precedence classifying and marking is enabled on the DUT.

Setup

The baseline setup for this test requires four test ports. Three ports are configured on the ingress side and are used to send emulated stateful application traffic. The fourth port is used to monitor and evaluate the outgoing egress traffic. See Figure 10.

Ixia's stateful network traffic emulation application IxChariot can help setup and execute this test case.


Figure 10. Stateful traffic network setup

Input Parameters

Parameters	Description
Traffic Rate	stateful traffic rate injected into the network per stream
DUT-QoS	Administrative DUT QoS setting (enabled or disabled)
DUT-Line speed	The link / interface speed of the DUT
DUT-QoS type	DUT QoS type settings: COS, ToS IP Precedence, or DSCP
DUT-QoS Policies	DUT QoS Policies applied to the ingress traffic
DUT-Queue type	Queuing mechanism such as Weighted Random Early Detection (WRED) and Weighted Round Robin (WRR) Queuing

Table 3. QoS Input Parameters Table

Methodology

TEST 1- QoS is disabled on the DUT

1. DUT is setup for no QoS processing.
2. Setup the traffic test. Refer to Table 3 for the input parameters required to setup the test.
3. No special queuing or other QoS related parameters are required on the DUT.
4. With QoS is disabled on the DUT, run the traffic test.
5. Figure 11 shows that the application response time per traffic type is not prioritized in any particular order. Therefore the network response time is almost the same for all data pairs or streams.

TEST 2- QoS is enabled on the DUT

1. Enable QoS on the DUT, and rerun the test.
2. Note the new application response time is now different and scaled according to the priority of traffic.
3. Figure 12 shows that the traffic is now prioritized based on configured priority level, with lowest priority traffic seeing the highest network response time.

Results

The results for this test case show that when QoS is enabled and the network is congested, the response time is progressively less for higher priority traffic (Figure 12). In the QoS disabled case (Figure 11), the response times are random between the different traffic flows.


Figure 11. Stateful TCP/IP traffic behavior and response time with QoS disabled


Figure 12. Stateful TCP/IP traffic behavior and response time with QoS enabled

3. Triple Play QoS Functional Test

Objective

This test case validates that VoIP and Video application traffic is effectively handled when QoS is enabled on a Triple Play network delivering multiple services. This test is accomplished by having simulated clients send various Data, Voice and Video traffic types into the network with the SUT/DUT applying QoS classifications and policies on the ingress traffic. The resultant egress traffic from the DUT should show effective performance for delay and packet loss sensitive applications such as VoIP and Video.

Setup

The baseline setup for this test requires four test ports. Three ports are configured on the ingress side and are used to send emulated Triple Play application traffic. Each emulated client transmits a separate application stream with associated QoS Priority value. The fourth port monitors the resultant traffic (i.e. egress) that has been prioritized and classified by the DUT. See Figure 13.

Ixia's stateful network traffic emulation application IxChariot can help setup and execute this test case.


Figure 13. Triple Play Data, Voice and Video traffic simulation setup

Input Parameters

Parameters	Description
Traffic Rate	stateful traffic rate injected into the network per stream
Frame size	Packet frame size
DUT-QoS	Administrative DUT QoS setting (enabled or disabled)
DUT-Line speed	The link / interface speed of the DUT
DUT-QoS type	DUT QoS type settings: COS, ToS IP Precedence, or DSCP
DUT-QoS Policies	DUT QoS Policies applied to the ingress traffic
DUT-Queue type	Queuing mechanism such as Weighted Random Early Detection (WRED) and Weighted Round Robin (WRR) Queuing

Table 4. QoS Input Parameters Table

Methodology

TEST 1- QoS is disabled on the DUT

1. With QoS is disabled on the DUT, run the traffic simulation test.
2. The receiver traffic (egress port) is measured for MOS (Voice quality), and MDI (Media Loss Rate (MLR) for Video quality). See Figure 14 and 15.

TEST 2- QoS is enabled on the DUT

1. Enable QoS on the DUT and rerun this same test.
2. The traffic that is received by the DUT is classified, prioritized and processed accordingly.
3. Note the new MOS and MLR scores. See Figure 16 and 17.

Results

The most important statistics and measurements to note in this test case are those seen in real time as the DUT QoS administrative state is changed from disabled to enabled or vice versa.

Figure 18 shows that the VoIP MOS estimate drops from a high of 4.4 down to a low of 1.0, while the Video MLR changes from 0 to an average of 400 media packets/second (see Figure 19) when QoS is changed from enabled to disabled.

Figure 14. IxChariot VoIP MOS estimate and DUT performance- QoS is enabled

Figure 15. IxChariot Video Media Loss Rate and DUT performance- QoS is enabled

Figure 16. IxChariot VoIP MOS estimate and DUT performance- QoS is disabled

Figure 17. IxChariot Video Media Loss Rate and DUT performance- QoS is disabled

Figure 18. Real Time VoIP MOS Estimate QoS enabled to disabled

Figure 19. Real time Video Media Loss Rate QoS enabled to disabled

4. Conclusion

The first two test scenarios (stateless and stateful traffic) demonstrate the benefits of implementing Quality of Service features in the network. Network efficiency improves and preferred treatment is given to higher priority traffic. Running Triple Play traffic scenarios (Data, Voice and Video) through a QoS capable network demonstrates the most effective performance and response time for delay and packet loss sensitive applications such as Voice and Video. Without QoS, a video and voice quality degradation is seen as reflected by the change in MOS and MDI quality metrics.

[ back | top of page | back to test plans ]