Support: Downloads & Updates

IxChariot Support: Testing Tips

This section includes some performance testing tips and techniques we think you'll find helpful. As you become more experienced with IxChariot, you'll uncover your own ways to use this software more productively. Contact us with what you learn; we'll share your lessons with other users via the Web and in upcoming releases.

Using IxChariot for Network Performance Testing

Here are suggestions for creating effective, repeatable performance tests. In general, choose the following options for performance testing:

Using IxChariot for Stress Testing

Ixia's suggestions for stress testing IxChariot intentionally conflict with our suggestions for performance testing. In performance testing, the objective is to increase performance of the network under actual working conditions. Stress testing, by comparison, generates a lot of extra network traffic-really stressing your hardware and software. The following suggestions help you make better use of IxChariot while stress testing:

Avoiding Too Many Timing Records

When you run a test for a fixed duration, IxChariot ignores the number_of_timing_records value specified in the scripts. It runs as many transactions during that time as it can.

When you run a test for a duration much greater than typically required, you greatly increase the number of timing records the endpoints generate. IxChariot becomes cumbersome when the number of returned timing records is above 10,000.

For example, you may run a test with one endpoint pair which generates on your network 100 timing records in 20 seconds. If you run the same test with the fixed duration set to one hour, IxChariot generates approximately 18,000 timing records. Additional pairs multiply the number.

When running a test for a fixed duration, Ixia strongly recommends increasing the scripts' transactions_per_record values so the test's total number of timing records is less than 10,000.

To reduce the number of timing records:
  1. In the Test window, click on the toolbar Add pair or Edit pairs icon. When the dialog box appears, you can open or edit a script, and change the endpoint pair's addresses and protocols.
  2. In the Add or Edit an Endpoint Pair dialog box, click on the Open a script file button which opens a list of scripts. Once you have chosen a script, click on the Edit this script button.
  3. An Edit a Script dialog box appears. Double-click on transactions_per_record, at the bottom half of the dialog box.
  4. Increase by ten times the Current Value for the transactions_per_record. For example, if the number of transactions_per_record is 50, change the number to 500. Save your changes by pressing OK on the open dialog boxes.
  5. Open Set run options from the Run menu. Click the Run for a fixed duration button. In the Duration field, select one (1) minute.
  6. Run the test and view the results. Using the Raw Data Totals tab, look at the Number of Records. This is the total number of timing records generated in one minute. You would prefer about 50 to 100 records per pair for good statistical significance. If the results contain more than a total of 10,000 timing records, go back and further increase the transactions_per_record. Otherwise, continue to the next step.
  7. Increase the scripts' transactions_per_record to match the duration of your test.

Here is the formula to determine the number of transactions per record:
transactions_per_record = (minutes to run) x (transactions_per_record in the one minute test)

Example:

In the steps above, you entered 500 for the script's transactions_per_record and ran the test for one minute. The results should have about 300 timing records per pair (100 per 20 seconds = 300 per minute). Three hundred timing records per pair is a fine number if you are not running hundreds of concurrent endpoint pairs.

Now you would like to run the test for the whole weekend (48 hours). Consider the math:
transactions_per_record =
(60 minutes per hour) x (48 hours) x (500 transactions per record)

transactions_per_record =
(2880 minutes) x (500 transactions per record)

transactions_per_record = 1,440,000

Open the script and change the transactions_per_record value to 1,440,000. Then, change the Duration to 48 hours in the Set Run Options dialog.

Your test is ready to run over the weekend.

Testing Unstable Environments

Fundamentally, IxChariot was designed to test stable network connections. If a connection drops, IxChariot will not attempt to reestablish it, and it will end the test. The design rationale for this was that the meaning and interpretation of results can be badly skewed if IxChariot is spending time doing work (i.e., reestablishing a connection) that is not part of the script. However, there are network environments that see outages of several minutes as normal. The Internet and wireless networks are two possibilities.

For short tests, this is not likely to be a problem. Long tests, however, require some different ways of thinking about the testing process. In either case, keep in mind that you may want to set the Run options not to 'Stop run on initialization failure.

Using the Datagram Protocols (UDP and IPX)

A surprisingly simple solution to overcoming lost connections is to not lose them to begin with. Perhaps you would like to run a test for several hours over TCP on a network on which individual endpoints may become disconnected for a few minutes at a time. If this happens, the TCP stack will decide the connection has been lost and tell IxChariot, ending the test. It is possible to go into the TCP stack parameters and configure them to retry the connection, or to take longer to timeout, but it is a cumbersome task, and must be done on each machine that will act as an endpoint in your test. If you switch to UDP, there is a more elegant method. UDP uses the same IP layer as TCP. It's not exactly the same thing as TCP, but it's close, and it moves the connection oriented controls into the application layer (i.e., IxChariot) where they are easily adjusted from one central point.

Likewise, the above arguments all apply to IPX in the NetWare environment.

Look at a IxChariot test screen. Click on the Run menu item, and then on the Set run options.... When the Run options window appears, go to the Datagram tab. There are three items that can be configured here. We are only interested in the 2nd and 3rd, the Retransmission Timeout and the Number of retransmits before aborting. The maximum values these fields can be set to are 99,999 milliseconds (100 seconds) and 999, respectively. If one were to set the retransmission number to the maximum of 999, and the timeout value to 300 ms, once a test has begun, an individual node could be disconnected from the network, and as long as it was reconnected within 5 minutes, the test would continue to run.

So the question becomes, how long a network outage do we need to be able to absorb? If the number of retransmits is set to 999, then take the desired maximum outage time, convert it to ms, and divide by 1000. Thus 1 min needs 60 ms, 5 min requires 300 ms, 10 min, 600 ms, etc. The maximum possible time a test can be set to continue after a lost connection is 27 hrs, 45 min.

What kind of numbers will this produce? It might be wise to export the resulting timing records to a spreadsheet, and perhaps set up a macro to massage the data however we would wish. IxChariot will report the aggregate throughput, etc, and that value will average in the 'down time' from any temporarily lost connections. The detailed timing records will provide the average throughput for much smaller intervals, and one could compare the rates significantly above zero to see when a connection is working as opposed to when it isn't. When a connection had been lost, and for how long, can readily be determined from the detailed timing records. The timing records will display very close to the same time interval while the connection is up, and a large jump in time whenever there was a down time. The elapsed time will inform us when it happened, and the measured time will tell us for how long. The accuracy of these values will be affected by the timeout setting, and thus we would want to keep the timeout setting to the lowest value that copes with the networks expected outages. IxChariot's line graphs will display a diagonal line dropping to zero during outage periods, making them easy to spot.

The Datagram tab in the IxChariot test results will reveal how many datagrams were lost, retransmitted, etc. A little massage of this data will extract the retransmits due to long outages, allowing us determine if there are many short connection drops. Once again, a spreadsheet export may be helpful here.

Batch test execution

It is also possible to run tests in series, rather than running one large one. IxChariot comes with command line executables to facilitate running tests in batch files or with a scripting language. Note that an important consideration in the test setup is the Run option, Stop run on initialization failure. In testing unstable environments, one would likely want this option turned off.

Runtst.exe will run a previously set up test and save the results to either the original file, or to a new one. Fmttst.exe can be used to output the test results to spreadsheet to allow multiple tests to be easily compared in a single file.

These utilities could be used to run tests back to back, or perhaps the same test over and over, so that a test that may have ended because of a temporary problem can be restarted immediately.

There is an excellent example of a perl script that expands IxChariot's capabilities in this way on this web site.

IXChariot Endpoint Machine Resource Guidelines



Determining the computer requirements for a given IxChariot endpoint can be challenging. There are many variables involved, such as processor speed, operating system, protocol stack, memory, disk space, and the underlying network. To determine the requirements, you must first define how you plan to use IxChariot. Do you want to generate data as fast as possible to determine network performance, stress a local resource such as an adapter card or protocol stack, or just generate some network traffic? The type of information you need depends upon your usage.

Generating Maximum Throughput

The main factors in getting the most throughput from a computer are CPU speed and memory. You need a CPU that is fast enough to match your network capacity, and with enough memory to hold the code and data used for the test. For best throughput, we recommend using a 32-bit operating system. The memory you need is based on your operating system. Ensure that you have enough memory so that no swapping takes place while running a test.

The table below shows some guidelines in determining the best CPU for different network speeds.

Throughput Recommended Computer
less than 100 Mbs PCI-based machine with a 32-bit operating system
100 Mbs Pentium 166 or greater with multiple pairs (24 recommended) 1
100 - 155 Mbs Pentium II or Pentium Pro 200 (or greater)
over 155 Mbs Multiple Pentium II or Pentium Pro 200 endpoints
1 With a TcpWindowSize setting of 64586.


We saw an 18% improvement when we modified this value in the windows registry.

Refer to the Windows NT Resource Kit on the Microsoft Developer Network for more information. Calculating Memory Requirements IxChariot endpoints are designed to run in any machine with enough memory to sufficiently run the operating system. If you plan to use multiple pairs on a single machine, you may want to calculate the number of pairs that will run without causing the operating system to swap either code or data.

The table below can be used to plan this number of pairs. The Base RAM column indicates the amount of memory that is allocated by the endpoint before running any pairs. If the endpoint is not being used, this amount may go towards zero if the operating system supports swapping. The protocol columns indicate the amount of memory required for a pair of that protocol.


Operating System Base RAM (in KB) TCP KB/pair UDP KB/pair SPX KB/pair IPX KB/pair APPC KB/pair
Windows
NT 4.0		 2,076 35 - 60 160 - 180 35 - 60 160 - 180 55 - 85
Windows
95		 1,100 40 - 65 100 - 145 40 - 65 55 - 75 N/A
OS/2 1,096 50 - 65 150 - 170 315 - 340 150 - 170 65 - 90
NetWare 1,100 80 - 110 320 - 340 70 - 100 260 - 280 N/A
Windows
3.1		 550 72 - 600 72 - 600 N/A N/A N/A
UNIX 1,260 680 - 1,100 688 - 1,250 N/A N/A N/A
MVS 666 25 - 48 24 - 52 N/A N/A 22 - 44




Operating System Base RAM (in KB) TCP KB/pair UDP KB/pair SPX KB/pair IPX KB/pair APPC KB/pair
Windows NT 4.0 2,076 35 - 60 160 - 180 35 - 60 160 - 180 55 - 85
Windows 95 1,100 40 - 65 100 - 145 40 - 65 55 - 75 N/A
OS/2 1,096 50 - 65 150 - 170 315 - 340 150 - 170 65 - 90
NetWare 1,100 80 - 110 320 - 340 70 - 100 260 - 280 N/A
Windows 3.1 550 72 - 600 72 - 600 N/A N/A N/A
UNIX 1,260 680 - 1,100 688 - 1,250 N/A N/A N/A
MVS 666 25 - 48 24 - 52 N/A N/A 22 - 44


These RAM usage numbers represent sending with "send_datatype = ZEROS". Other send_datatypes require memory buffers roughly equivalent to the disk size of the .CMP file being used. Add 2KBytes when using send_datatype = NOCOMPRESS.

Capacities

The table below shows some example pair capacities we have tested on various computers. These pairs ran on a 10Mb Ethernet LAN. The number in the pairs column represents the number of pairs this machine supported as Endpoint 2 for a single test. We used the default values for all tests. An exception: for datagram testing, we lengthened the time-out values as well as the initial delay in the tests.


Operating System Computer RAM TCP pairs UDP pairs SPX pairs IPX pairs APPC pairs
Windows NT 4.0 Dell P166 32 M 500 100 300 100 200 1
Windows 95 2 Dell P133 16 M 18 100 40 175 N/A
OS/2 4.0 3 Dell P166 32 M 500 200 20 20 500
NetWare 4.12 Dell P100 64 M 500 200 100 100 N/A
Windows 3.x Compaq 8 M 1 1 N/A N/A N/A
AIX 4.1 IBM Power PC 604 133Mhz 64 M 200 180 N/A N/A N/A
1 This test ran on the Microsoft SNA Server
2 with Novell Client32
3 with Novell Client for OS/2


NOTE: This table does not represent the full capacities of these operating systems and stacks. They merely represent some of the larger tests that we have run in our test lab.

CPU Utilization

What effect does running IxChariot endpoint have on the other applications you may be running on your desktop? This depends upon the IxChariot tasking structure for the particular endpoint platform.

All IxChariot tasks run at normal priority, meaning they compete for CPU cycles on an equal basis with most applications. A base endpoint runs between 2 - 14 tasks in idle state (depending on what protocols are installed), and creates an additional task per endpoint pair. Although it depends on machine capacity and applications, you can normally expect to run up to 20 pairs before you will notice degradation of such applications as spreadsheets or word processors.

APPC Performance with MVS/VTAM

Our MVS endpoint has been tested in a wide variety of environments-and we've seen a wide range of throughput and response time values. Here's what we've learned about what to expect with APPC and how to tune for improved performance.

APPC Performance Expectations

This section discusses the performance you can expect, depending on the CPU usage and speed, the type of channel-connected device, and the speed of the partner endpoint hardware and software. (In the numbers cited below, a lowercase "b" stands for bits, while an uppercase "B" stands for bytes.)

CPU Usage and Speed

We don't have precise numbers here, because there are so many variables. The CPU speed and the percentage of its consumption can make a big difference in the performance values you see.

We strive for as little CPU overhead as possible in all of our endpoints. An idle MVS endpoint consumes 0.0% CPU usage. However, CPU usage can reach 100% when running an intense IxChariot test. We've seen that the majority of the CPU usage is in the protocol stack. It's obviously difficult to improve your performance once you're using 100% of the CPU's resources. So think through what you want to accomplish with a given computer and make sure it's tuned appropriately.

MVS Channel Attachments

Performance of Partner Endpoints

We offer APPC support today on our OS/2, Windows 95/98/Me, Windows NT/2000, and MVS endpoints. When one endpoint is MVS, the hardware and APPC software the partner endpoint uses can greatly affect the throughput you see.

We recommend careful evaluation of these products. In configurations representative of your environment, performance can vary dramatically. All of these products benefit from fast CPUs and lots of RAM.

Some Guidelines for Tuning APPC Performance for MVS/VTAM

Depending on what you're trying to measure, you will want to consider to following tuning guidelines.

New Changes in VTAM version 4.4

Much of IBM's work in VTAM version 4.4 is a performance rewrite. IBM extensively reworked the APIs and storage management for performance. SeeIBM's websitefor information on performance improvements in VTAM version 4.4.

IBM's APING program is an internal application in VTAM version 4.4. That means that it takes less overhead than before-avoiding LU 6.2 API crossings and security, for example. Whereas APING and our software gave comparable performance numbers in previous VTAM releases, expect improved performance numbers with the new APING. These are a little unrealistic, though, since they can't be attained by real VTAM applications.

The MVS endpoint provides the same performance results seen by real applications.

Storage handling and pre-allocation (with VTAM 4.4)

To take advantage of the improved storage handling in VTAM 4.4, we suggest an initial Communications Storage Manager (CSM) setting of 4 MBytes.

Emulating Multiple Users

A single IxChariot pair connection can represent the traffic generated by many users of a given network application. The values in the IxChariot application scripts default to, in effect 'drive this traffic as fast as is possible.' This is in fact far more than any single end user can generate.

A concrete example will help. For a simple case, look at the IxChariot script FILESNDS.SCR. This script represents a client machine connecting to a file server, transferring a file to the server, and receiving an application level acknowledgement that the transaction is complete. The defaults involve a 100kByte file size, and the file transfer repeats 100 times, each new transfer happening immediately after the conclusion of the last. On 10Mbps Ethernet, this takes about 10 seconds. The most active power user on a given network does not do 100 file transfers every 10 seconds! It is impossible for a person to hit the keys fast enough to accomplish this. So the trick with IxChariot is actually to slow things down.

There is a variable in all of the IxChariot application scripts, the "transaction_delay." The transaction delay variable sets the time between repetitions of the scripted transaction. The default is to immediately repeat the scripted transaction until the script completes. This is fine for testing that transaction's behavior on a given network, but it is considerably more traffic than a normal user generates. The question is, 'How much traffic does a normal user generate?" For now, we'll confine ourselves to a single application. For example purposes, we will stick with our file transfer script, FILESNDS.SCR. We will have to engage in some educated guesswork, or use a protocol analyzer to track discrete uses of an application during the day, or perhaps follow an average end user around for a day. So, as an example, we might determine that our typical end user transfers a file to the file server about every 20 minutes. So for the average user, there is an average delay between transactions of 20 minutes. The variable 'transaction_delay' available in the script accepts values in milliseconds. 20 minutes is 1,200 seconds and 1,200,000 milliseconds. If we enter a delay time between transactions of 1200000 (do not use commas when entering values in IxChariot scripts), we will slow IxChariot down to where it is generating the typical file transfer traffic of a single user.

What about representing multiple users? If we have created the traffic level of one typical user, then 10 times as much traffic would be the traffic generated by 10 users. Reduce the transaction delay by a factor of 10, to 120,000 milliseconds (every 2 minutes), and we have 10 user's traffic. Reduce the transaction delay by a factor of 100, to 12,000 milliseconds (every 12 seconds), and we are representing 100 people.

How far can we go with this process? That depends on your situation. The mathematical limit of the reasoning process we have gone though breaks down when the delay time is not significantly larger than the response time of a given script. To find out the limit in your network's circumstances, run a quick test, having made any changes you need to for other modeling purposes, but with 'transaction_delay' set to zero. Look at the resulting response time. This value is the lower limit to consider in our modeling process. If the transaction delay is less than the response time, then the next transaction will need to start before the last one ends, and that is not going to happen. In the example we have been using, FILSENDS.SCR always has a relatively high response time, and on 10 megabit Ethernet, going through one hub only, the response time for that script is about 105 ms on average, maximum response time about 140 ms. Under such circumstances, 500 ms is probably about as low as we would want to go. So our limit as to the number of users we can represent is 1,200,000ms/500ms = 2400 end users (doing file sends). Notice that we can actually represent more users on a faster data link technology. ATM or Fast Ethernet would give us a lower response time, and we could probably represent more users by a factor of about 100 (240,000 users). Also, if the script we were using was a shorter transaction, but still had the same typical amount of time between transactions, then we would also have a lower response time, and could represent more users. If however, we used larger file sizes, one Meg perhaps, rather than the default sizes of 100,000 kB, then we would have to reduce the number of users we can represent.

Now that's the mathematical limit of this model. Is its practical limit that high? Well, with IxChariot, the limit really depends on what one is trying to accomplish. Some deep thought concerning just what is under test and whether the test being applied is reasonable is always warranted. There are also additional factors to keep in mind. Calls to the system clocks work differently on different operating systems, so OS/2, Windows 3.1 and NetWare should not be used for delay times below 1 second. When using our benchmarking class of scripts, always use the 'short connection' scripts rather than the 'long connection' ones (FILESNDS, not FILESENDL). This is because one wants to include the overhead of set up and teardown of connections. The specific application emulation scripts generally already account for this distinction. This traffic representation is also very 'bursty,' with long periods of no traffic being generated, and then brief periods of full activity. This is how traffic in the 'real world' works, and it is specifically to emulate such traffic that IxChariot was designed.

There is room to refine our model further. The option exists in IxChariot's application scripts to randomize the sleep times, of which 'transaction_delay' is one. Rather than leaving our delay's at a set time, we can randomize them over a range of values. So rather than a 20 minute delay, we could give it a range of 15-25 minutes, or perhaps 0-40 minutes. We would change the radio button in the 'Edit a sleep variable - transaction_delay' window to uniform distribution, and set each boundary limit. For example, 900000 for 15 minutes, 1500000 for 25 minutes, etc.

The most important factor we haven't yet accounted for is the possibility of two systems attempting to use the network at the same time. The model we have developed so far uses a single connection at a time. It can emulate thousands of users over a period of time, but it cannot emulate more than one user at EXACTLY the same time. There is never any contention for bandwidth. The solution to this is to divide the number of users among several pairs, preferably among several machines. If several machines are available for testing, with even numbers on each end of a network cloud, one can divide the number of users one wishes to emulate among several endpoint connection pairs, and spread those among different pairs of machines. Additionally, if multiple network cards (or multiport cards) are available, the connections to a given machine can be divided among them. If we wish to take into account contention for network bandwidth, the question to consider is, 'how many connections must potentially happen at the exact same time?' The answer to that is also the minimum number of pairs that must be used to represent that application's traffic. A caveat to this is, if the testing need is to maintain the number of simultaneous connections a piece of equipment is capable of, the number of connection pairs should match that value, and there should no delay between transactions.

So the two theoretical factors to account for are 'How many users to represent?' and 'How many can be connected at precisely the same time?' However, the more practical factor to start with is, 'How much equipment is available to work with?' Determine how many machines, and how many network ports, are available, and then decide how many pairs to use, and how much traffic each pair should drive. Keep in mind that 1000 users are better represented by several pairs spread over several machines than by one pair from one machine.

For further discussion, see the Testing Tip, "Representing an Entire Network's Traffic with IxChariot."

[ back | top of page ]
 
Homepage | Sitemap | Contact Us | Privacy Policy | Terms Of Use | Ixia Suppliers