Library: Test Plans

IPTV - Channel Change Performance Testing

• Background
• Performance Metrics
• Test 1 - Optimal Channel Change Performance
• Test 2 - Single Subscriber Experience
• Test 3 - Triple Play Traffic

In the context of Triple Play services delivery, IPTV represents the most significant challenge and often the most complex to test and verify. This channel changing performance test plan assists you in comprehensively qualifying the performance of one or more Devices Under Test (DUT) or Systems Under Test (SUT) in an Internet Protocol Television (IPTV) deployment.

This test plan encompasses several key IPTV performance metrics that are critical in qualifying the DUT/SUT and ensuring a successful deployment. Each of the test contained here provides comprehensive information on stress testing the DUT/SUT to ensure that is it able to perform as specified, and able to handle the typical and extreme load conditions as required. For more information, see the "Performance Metrics" section later in this document.

The recommended testing methodology presented here is also meant to be used as a guideline for creating more case-specific testing scenarios to further characterize the performance limits of the IPTV infrastructure.

Background

In the telecommunications marketplace today, traditional cable and telephone operators are aggressively staging and deploying converged networks that can deliver data, voice, and video services to their rapidly expanding customer base. Such networks are referred to as Triple Play networks.

The effective delivery of the video component of Triple Play services, called IPTV, has a fundamental requirement to operate within a multicast-capable network, delivering broadcast service television over the IP network by ensuring the efficient use of network resources. Using this technology, IPTV services are delivered directly to the home and are intended to replace the traditional video delivery network.

The figure below is a simplified view of a multicast network used to deliver a single channel (using a single multicast address) to a number of group members (or viewers of a particular channel). The flow of data from the Group Source to the Group Members follows the multicast routing tree. A multicast routing protocol such as PIM-SM/DM is used to create an efficient routing path for the delivery of multicast packets between routers. The host to edge-router communication is generally facilitated using IGMPv2/v3.



What is channel changing or channel zapping?
Channel changing or "channel zapping" in the IPTV domain is the equivalent of surfing channels on a television set in the traditional sense. Channel changing, then, always entails leaving one TV channel and joining/watching the next.

Why is it important?
In the traditional cable television network, channels were always "present" on the wire. For this reason, channel changing performance was generally not an issue. Switching from one channel to the next was usually instant, typically was not a source of user dissatisfaction.

For satellite TV providers, on the other hand, the inherent distance that the video signals had travel always introduced a "delay" in switching between channels. The usual approach to reduce the perceived delay was to provide some kind of on-screen feedback when switching channels so that the user knew the channel change request had been received.

Now, with video being delivered over an IP network, one that was not necessarily designed with video transport as a key goal, several optimizations and novel methods are being deployed throughout a providers' network to improve the perceived "instant" channel change behavior that is expected by traditional television consumers.

IGMP is the industry standard protocol used for host to router communication in a multicast network. Therefore, channel changing (CC) performance measured at an edge router or an aggregation device is intricately tied to inherent implementation and fine tuning of the IGMP protocol. The CC performance is also a function of the emulated end device receiving the multicast stream for final output, such as a set-top box. For this reason, channel changing performance is vital in characterizing performance for devices such as DSLAMS or CMTS and is equally important for an end-to-end IPTV test.

For more information on the specific performance metrics attainable by using channel changing profiles to test the IPTV network, refer to the "Performance Metrics" section later in this document.

What are the channel changing (CC) requirements?
The channel changing performance in an IP network is primarily the result of end-to-end processing of associated multicast protocols, packet switching, and the end-device's ability to decode or "show" the video with relative consistency.

A deployed system must be able to perform predictably in a sustained mode. A sustained mode of operation refers to a realistic load condition, not a best-performance condition with light use.

In addition to sustained performance, a deployed system must be resilient to fault conditions where peak loads may easily exceed the sustained performance limits. For a system to be resilient, it must be able to perform acceptably well and be able to recover from an overloaded condition.

Channel surfing presents varying load conditions for a multicast network. For example, if there is a sudden power outage in a neighborhood on a typical Sunday "game day," a mass flood of traffic will be injected into the IP network requesting network configuration information followed by several channel changes to tune back to a desired channel by hundreds, if not thousands, of viewers at almost the same time. Testing such realistic load conditions is critical in ensuring optimal QoS on the network and help tune device settings.

Requirements of good channel performance are characterized using the metrics in the "Performance Metrics" section later in this document.

Performance Metrics

To determine channel changing performance, several performance metrics are used. The following terms are defined and used to provide objective performance. Note that the terms defined here comprise generally accepted industry terms and protocol-specific terms that help characterize the performance.

IGMP Join Latency - The time between a request to join a multicast group and the receipt of the first byte of data for a multicast group.

IGMP Leave Latency - The time between a request to leave a multicast group and the receipt of the last byte of data for the multicast group

Channel Overlap - The duration of time when data is received for a new joined multicast group and a previously left multicast group. This time is usually zero units.

Channel Switch Delay (STB dependent) - An internal IGMP processing delay between a Leave and Join request. This value is ideally insignificant; however, it can be otherwise.

Channel Change/Zap Delay - The inter-channel change delay, which is the time between a channel Leave request sent and the receipt of the first byte of data from the new multicast channel. It is the IGMP Join Latency + Channel Switch Delay (STB dependent). This value is ideally very close to the IGMP Join Latency; however, the STB can introduce a significant delay. The following timing diagrams outline the delays for 2 channels.

Figure 2 - IGMP Join and Leave latency timing diagram

The specific metrics outlined above must be available on a per-channel/per-subscriber basis. The various metrics make up the parts of a sum that help provide an at-a-glance health check for every subscriber and the channels being watched.
It is not sufficient, though, to use such metrics alone to characterize the performance of an IPTV network/device. To isolate adverse network conditions causing unacceptable channel change performance, network centric measurements must be available to provide an at-a-glance health check of the network on a per-channel/per-subscriber.

Media Delivery Index (MDI) - A scoring mechanism that combines packet delay variation (jitter) and media packet loss to determine the quality of the network to transport good quality video. It is measured in milliseconds.

MDI:DF (Delay Factor) - Defined as cumulative IP jitter. It represents the time it would take to drain an output buffer and ensure good video playback.

MDI:MLR (Media Loss Rate) - Defined as the packet loss rate due to dropped packets, bad/corrupted packets, or out-of-sequence packets.
The Media Delivery Index (MDI) is important because it characterizes the performance of the network and its ability to handle good video streams. The index provides two measures separately so that each IPTV device can be tested with various channel change patterns to help insolate device-specific issues. By identifying problems on an IPTV network device, it becomes easy to troubleshoot and optimize the configuration to provide optimal performance end-to-end.
This approach can be extended to characterize the end-to-end channel change performance of an IPTV deployment.

Test 1 - Optimal Channel Change Performance

Objective
This test setup focuses on determining the optimal channel change performance with IPTV traffic running across one or more DUTs.

Since IPTV subscriber has a different usage pattern, it is necessary to determine the end user's experience based on realistic channel change patterns. It is also important to determine how the subscriber's channel change performance changes as the complexity of the subscriber's behavior (channel change profile) increases and as the number of subscribers increase.

The variability of the CC patterns will have a direct impact on the performance of devices supporting such traffic. The importance of determining the optimal performance of an IPTV DUT is to ensure that it is within operating limits of providing excellent channel change performance experienced by subscribers. For example, is there channel overlap when users are rapidly "surfing" channels? How does poor jitter experienced by one video stream affect the others on the same link? How is the transport of video streams affected by a growing number of subscribers? These are just some of the questions that can be answered by emulating realistic user channel change behavior to determine various acceptable levels of performance.

This test emulates typical load conditions of 1000s of subscribers with multiple channel changing profiles and 100s of video streams. Several iterations of the test will help ensure that the device/network has sufficient raw bandwidth to support the load, reveal at localized congestions, and assist in fine-tuning devices for maximum performance.

The maximum number of users supported by the DUT/network does not necessarily translate to optimal channel change performance. For this reason, there is a tradeoff between the best channel change performance with realistic subscriber loads and the maximum performance possible by the device/network. Both metrics have merit and both must be determined.

This test is applicable to an IPTV device and can be extended to characterize optimal end-to-end performance.

Setup
The setup requires at least 2 or more Ixia test ports to generate stateful IPTV traffic. A video server port is used to stream 100s of standard and high definition video streams. The topology presented here is representative of a typical Triple Play deployment network. There are several configurations possible and this test setup is used as a guideline. The video subscribers are connected to an IGMPv3 enabled core router and the video server is connected to a carrier core router. The core multicast network has a 4Gbps backbone and it runs BGP for route determination and PIM-DM as the multicast protocol.



Input Parameters

Parameter	Description
Client Configuration	50 - 1000+ users per test port
Client Network	One or more VLAN with untagged and/or 802.1q tagged packets per VLAN
Client Channel Change Profiles	Sequential channel surfing - for 10-20 sec Random channel surfing - for 10sec-20min Other random profiles as per tester's requirements
Client Command-list	JOIN channels as per Channel change Profile selected per run
Client IGMP parameters	The IGMP options below may be configured as desired. Consider the specific tradeoffs for each options before enabling/disabling them. Unsolicited Response Mode General Query/Group Specific Response Mode Immediate Response Suppress Reports
Client/Server Test Ports	At least one. More ports can be used to scale the test
Video Server	At least one.
Video Server Configuration	100+ video streams. Use MPEG2-TS with synthetic payload at 3.75Mbps (for SD). Use higher bitrate video streams to simulate HD streams (9.8Mbps+) Multicast group address - configurable as per network design. Must have sufficient count to ensure all video streams have a multicast group address
Test Objective	Iterative method to determine the maximum simulated users.

Methodology

1. Before starting the test, determine the baseline performance characteristics of the test equipment. This will ensure that the upper limits of the test equipment are known and that the performance per-port or per-system is well defined.
2. Once the baseline is established using the test ports, configure the test setup network or the IPTV DUT.
   
   To support multicast traffic, at least one switch will be required to support IGMPv2 or v3. In addition, a multicast protocol must be running to efficiently deliver multicast traffic.
3. Set up the test, and configure the parameters for the protocols as outlined in Input Parameters for the Client Traffic.
   
   Choose the version of IGMP supported by the switch. Also, ensure that the IGMP parameters are using default values as specified in RFC 2236 (IGMPv2) or RFC 3376 (IGMPv3).
   
   The command-list is where the channel changing profile can be configured. The desirable options here are to run the test iteratively and modify the channel changing profiles to simulate different load conditions on the multicast fabric. Some channel changing profiles can include:
   • A set of subscribers surfing through channels 1-50 for 10-20 seconds each.
   • A set of subscribers surfing through channels 51-100 for 1-30 seconds each.
   • A set of subscribers emulating a typical habit of watching channel A for 30 minutes with frequent channel surfing within that time for commercial breaks.
   The channel-changing profiles must be iteratively set up to test the multicast network and monitor an average subscriber's channel change performance. Ensure that there are sufficient ports to run the test.
4. Configure the video server traffic to serve 100s of video streams. Refer to the Input Parameters to configure the stream contents appropriately.
5. Configure the Client and Server networks to ensure end-to-end connectivity. Confirm that multicast protocol is running and reachable end-to-end.
6. Set up the test objective to determine the maximum number of sustained users.
7. Run several iterations of the test to determine the maximum performance of the DUT/network (in handling # of subscribers and # of simultaneous channels).
   
   However, consider that the maximum number of subscribers supported by the DUT/network does not necessarily translate to optimal channel change performance. There is a tradeoff between the best channel change performance with realistic subscriber loads and the maximum performance possible by the network. Both metrics have merit and both must be determined.
9. Adjust the IGMP parameters to optimize channel change latency.
   
   Vendor implementations of IGMP may vary and specific optimized configurations must be determined from configuration and user guides.

Results
This test was used to characterize various subscriber load conditions and determine the channel change performance experienced by each subscriber. In addition, network flow quality of video streams were examined per stream to determine how the device/system performed when it was incrementally loaded with more subscribers, more channels, and more complex subscriber channel change patterns.

The optimal channel change performance was a tradeoff between the maximum performance possible by the DUT/network without packet loss and the acceptable channel change latencies under realistic load conditions. Both metrics have merit. Knowing the peak performance of the DUT/network ensures that extreme load conditions are serviceable during failure or startup times. However, at peak performance a subscriber's channel changing performance may be less than optimal.

Optimal channel changing performance was measured at an equilibrium point that was below the maximum device performance but still simulated realistic subscriber load conditions.

A brief result of three runs is presented below:

Stat Name	Stream Bit Rate (bps)	MDI-DF (us)	MDI-MLR	MPEG2 TS Loss	Jitter (ns)	Inter Pkt Arrival Time (ns)	Packet Latency (ns)	Join Latency (ms)	Leave Latency (ms)
RUN 1/ User 1 Channel 1	3750007	2835	0	0	280	2805680	38160	69	8479
RUN 2/ User 1 Channel 1	3750007	2835	0	0	1780	2807180	38320	108	8460
RUN 3/User Channel 1	3750007	2835	0	0	300	2216420	38540	100	5045

The first run simulated 300 users watching 100 channels for 10-20 seconds each. The second run increased the number of users to 1200. The jitter increased considerably with the increase in subscriber count. The third run reduced the subscriber count to 800 and modified the channel change behavior for different sets of subscribers.

If the subscribers were increased to more than 1200, there were MPEG2 TS losses seen. The optimal configuration for the network was 800 subscribers - all having different channel changing profiles.

To further characterize the channel change performance, real-time statistics of the Channel Change/Zap Delay were actively monitored. A snippet of these real-time statistics for the third run is presented below.



The average jitter distribution for several channel requests is presented below for the third run, and 99.87% of the packets had jitter below 50 us, which is very good.

Test 2 - Single Subscriber Experience

Objective This test is composed of a series of runs with 1000s of subscribers with various channel change patterns. The test will assess the experience of a single subscriber who is watching a few channels while several other subscribers place different load requirements on the multicast network. In this way, the test setup will assess the join/leave latency and the channel change/zap delay, including MDI scores.

Setup
The setup will require at least 2 or more Ixia test ports to generate stateful IPTV traffic. A video server port is used to stream 100s of standard and high definition video streams. The topology presented here is representative of a typical Triple Play deployment network. There are several configurations possible and this test setup is simply used as a guideline. The video subscribers are connected to an IGMPv3 enabled core router and the video server is connected to a carrier core router. The core multicast network has a 4Gbps backbone and it runs BGP for route determination and PIM-DM as the multicast protocol.



Input Parameters

Parameter	Description
Client Configuration	50 - 1000+ users per test port
Client Network	One or more VLAN with untagged and/or 802.1q tagged packets per VLAN
Client Channel Change Profiles	1 subscriber with fixed channel change pattern Other random profiles as per tester's requirements
Client Command-list	JOIN channels as per Channel change Profile selected per run
Client IGMP parameters	The IGMP options below may be configured as desired. Consider the specific tradeoffs for each options before enabling/disabling them. Unsolicited Response Mode General Query/Group Specific Response Mode Immediate Response Suppress Reports
Client/Server Test Ports	At least one. More ports can be used to scale the test
Video Server	At least one.
Video Server Configuration	100+ video streams. Use MPEG2-TS with synthetic payload at 3.75Mbps (for SD). Use higher bitrate video streams to simulate HD streams (9.8Mbps+) Multicast group address - configurable as per network design. Must have sufficient count to ensure all video streams have a multicast group address
Test Objective	Iterative method to determine the optimal single subscriber performance (using maximum simulated users)

Methodology

1. Configure the test setup network or the IPTV DUT.
   
   To support multicast traffic, at least one switch will be required to support IGMPv2 or v3. In addition, a multicast protocol must be running to efficiently deliver multicast traffic.
2. Set up the test, and configure the parameters for the protocols as outlined in Input Parameters for the Client Traffic.
   
   Choose the version of IGMP supported by the switch. Also ensure that the IGMP parameters are using default values as specified in RFC 2236 (IGMPv2) or RFC 3376 (IGMPv3).
   
   The command-list is where the channel changing profile can be configured. The desirable options here are to run the test iteratively and modify the channel changing profiles to simulate different load conditions on the multicast fabric. Some channel changing profiles can include:
   • One subscriber (pilot) watching a few channels for long durations.
   • A set of subscribers emulating many channel change patterns to stress the multicast network differently. The channel-changing profiles must be iteratively set up to test the multicast network to monitor a single subscriber's channel change performance for the iterations. Depending on how the clients profiles, ensure that there are sufficient ports to run the test.
3. Configure the video server traffic to serve 100s of video streams. Refer to the Input Parameters to configure the stream contents appropriately.
4. Configure the Client and Server networks to ensure end-to-end connectivity. Confirm that multicast protocol is running and reachable end-to-end.
5. Set up the test objective to determine the maximum number of sustained users.
6. Run several iterations of the test to determine the channel change performance for a single pilot subscriber.
   
   Change the multicast load conditions by iteratively changing the other subscribers' channel change patterns and monitor the pilot user's Join/Leave and Change Change/Zap delays.

Results
The objective of this test was to assess an single subscriber's channel change performance under various loads by changing the channel change profiles of the subscribers. The real-time nature of monitoring a single subscriber's Join/Leave latencies is essential. An instantaneous view of the real-time information of the single subscriber is presented below.

Stat Name	Stream Bit Rate (bps)	MDI-DF (us)	MDI-MLR	MPEG2 TS Loss	Jitter (ns)	Inter Pkt Arrival Time (ns)	Packet Latency (ns)	Join Latency (ms)	Leave Latency (ms)
RUN 1/ User 1	3750007	2229	0	0	198	2216220	38360	100	4503
RUN 2/ User 1	3750007	2233	0	0	422	2217140	38020	170	6901

The first run included 500 users with varying channel changing patterns. The second run increased the subscriber count to 1000. From the table above, it can be seen that increasing the subscribers and hence the load on the multicast network, increased the subscriber's perceived Join/Leave latencies by 70% and 65% respectively.

The channel change/zap delay observed for the single subscriber was similarly affected.

The DUT was not tested for raw performance. Therefore, the behavior of a single subscriber being affected by other subscribers is primarily a result of the network having no QoS setup to police any traffic. Generally, the QoS on the network must be set up so that the single subscriber sessions are not affected by other subscribers on the same link.

By monitoring a single subscriber, it is possible to troubleshoot devices that may not be observing the QoS settings or traffic policy properly. Such monitoring, however, may not be possible if the only metric observed is the optimal channel change for all aggregate subscribers.

Test 3 - Triple Play Traffic

Objective
This test extends the IPTV channel change performance test by provisioning data and voice traffic on the same link as the video streams to assess the performance of this additional traffic.

In a deployed network, the convergence of data, voice, and video traffic creates a realistic scenario where the video streams are sharing resources with data and voice traffic. The traffic types are very different from each other, and each requires a different level of service from the network.

To ensure that each of the devices on the converged network is working properly as part of the IPTV solution, it is necessary to test several Triple Play scenarios where the channel change behavior of several subscribers creates realistic load conditions. This is in addition to the data and possibly voice traffic that must transit the same link, either from the same subscriber or other subscribers sharing an uplink from an aggregate router. By emulating various load conditions on the Triple Play network, optimizations can be made to ensure that the QoS on the devices in the network is working as expected..

Setup
The setup will require at least 3 or more Ixia test ports to generate stateful Triple Play traffic. A video server port will be used to stream 100s of standard and high definition video streams. A second server port will accept data and voice traffic. At least one client test port will generate stateful video, voice, and data traffic based on the conditions in the Input Parameters table. The topology presented here is representative of a typical Triple Play deployment network. There are several configurations possible and this test setup should be used as a guideline. The video subscribers are connected to an IGMPv3 enabled core router and the video server is connected to a carrier core router. The core multicast network has a 4Gbps backbone, and it runs BGP for route determination and PIM-DM as the multicast protocol.



Input Parameters

Parameter	Description
Client Configuration	50 - 1000+ users per test port
Client Network	One or more VLAN with untagged and/or 802.1q tagged packets per VLAN
Client Channel Change Profiles	Sequential channel surfing - for 10-20 sec Other random profiles as per tester's requirements
Client Command-list	JOIN channels as per Channel change Profile selected per run FTP traffic - varying bandwidth to simulate typical and adverse conditions HTTP traffic - varying page retrievals SIP traffic - to simulate voice-over-IP calls
Client Parameters	The IGMP options below may be configured as desired. Consider the specific tradeoffs for each options before enabling/disabling them. Unsolicited Response Mode General Query/Group Specific Response Mode Immediate Response Suppress Reports FTP client - use Passive mode SIP - use UDP for signaling with RTP media (codec selectable)
Traffic Impairment (optional)	Introduce traffic impairment such as latency, jitter, packet drop profiles and duplicate packets (to further stress the edge devices in processing packets with errors or inherent delays/jitter than may cause queuing problems on the interface buffers)
Client/Server Test Ports	At least one. More ports can be used to scale the test
Video Server	At least two. A dedicated video server port and a dedicated data and voice traffic port
Video Server Configuration	100+ video streams. Use MPEG2-TS with synthetic payload at 3.75Mbps (for SD). Use higher bitrate video streams to simulate HD streams (9.8Mbps+) Multicast group address - configurable as per network design. Must have sufficient count to ensure all video streams have a multicast group address
Test Objective	Iterative method to determine the maximum simulated users

Methodology

1. Configure the test setup network or the IPTV DUT.
   
   For multicast traffic, at least one switch is required to support IGMPv2 or v3. In addition, a multicast protocol must be running to efficiently deliver multicast traffic.
2. Set up the test, and configure the parameters for the protocols as outlined in Input Parameters table for the Client Traffic.
   
   Choose the version of IGMP supported by the switch. Also, ensure that the IGMP parameters are using default values as specified in RFC 2236 (IGMPv2) or RFC 3376 (IGMPv3).
   
   The command-list is where the channel changing profile should be configured. The desirable options here are to run the test iteratively, and then modify the channel changing profiles to simulate different load conditions on the multicast fabric. Include data traffic such as FTP and/or HTTP with various page retrievals. Also, enable SIP voice-over-IP traffic (1 call per subscriber).
3. Configure the video server traffic to serve 100s of video streams. Refer to the Input Parameters table to configure the stream contents appropriately. Also configure the SIP callee and FTP server side traffic profile.
4. Configure the Client and Server networks to ensure end-to-end connectivity. Confirm that a multicast protocol is running and reachable end-to-end.
5. Set up the test objective to determine the optimal subscriber channel change performance. The test Objective will vary for each of the activities defined (video, voice and data), depending on the activity and test iteration. The objectives will reflect the load conditions required to effectively determine how background data and voice traffic affect an average subscriber's channel change performance.
6. To test the capability of the edge routers in handling malformed packets and being able to manage its interface buffer queues properly, introduce impairment such as latency, jitter, duplicate packets or random drop packets.

Results
The objective of this test was to determine the subscriber's channel change performance on a converged network that is running video and voice traffic.

Several test runs were conducted. Two of the runs are examined here.

The results below show successful SIP calls during one of the test run 2 with 200 subscribers.



HTTP transactions are presented below for the same test run.

The HTTP transaction latency for two runs is given below. The transaction latency increased on the second run as the FTP throughput was increased from 1.6Mbps to 24Mbps. The FTP traffic directly impacted the HTTP traffic (and the video traffic).

A similar trend was seen on the voice calls. The per-call jitter increased an average of 18% between the first and second run.

The impact of background data traffic was also apparent on the video streams.

The channel change/zap delay increased between runs as the data traffic on the link increased. In deployed systems, however, such behavior can be mitigated by designing and implementing strict QoS policies at the edge of the network. In addition, a per-subscriber QoS model must be used to ensure that different traffic originating from the same subscriber is treated differently by the edge routers. This will allow predictable behavior to ensure that data traffic does not impede other more sensitive traffic such as video streams.

[ back | top of page | back to test plans ]