Constantin Cretu
Ixia Senior System Test Engineer

Automation 101—Large-Scale Application Testing Needs? DIY Build a Machine Gun

August 16, 2017 by Constantin Cretu

We introduced readers to the basic concepts of BreakingPoint automation in our TCL and REST blogs. Now can move forward with interesting scenarios that unleash the true power of automation.

Next-generation firewalls (NGFWs) changed the way firewalling was done by moving from inspecting Layer-4 ports to inspecting the actual applications. Applications became more complex and sometimes malicious, and almost all of them were using the HTTP(S) ports. The NGFWs allowed IT teams to inspect, police, and block specific applications that were using the same ports, thereby providing the much-needed granular control. Testing the types of applications and the network’s ability to handle them still remains a challenge because of the sheer number of popular apps and their continuous growth. Manually testing individual applications against the device under test (DUT) and finding out each device’s performance and inspections/policing/blocking abilities can be a super exhaustive task.

Our objective in this blog is to select one application flow at a time from the large cache of BreakingPoint application flows and send it through the DUT. The test will emulate client and server of this application and an automated validation will check if the application was blocked/allowed by the DUT and if allowed, the performance scale the application could achieve. The script will dynamically cycle through all of BreakingPoint’s 3,700+ application flows. A log file will be created that will print the summary of the results for each test. Since the tests will be saved on the BreakingPoint system, they can be rerun manually in case additional investigations are needed for a particular application.

Let’s get started

For simplicity, we will start with the little “Hello World” script that created in our previous blog: BreakingPoint Automation Series: Tcl 101. The first twenty lines at the beginning are the same as in the “Hello World” example, only the rest of 120 are different and will enable us to create this magic J.

We’ll explain below the modifications for: Network Topology, Traffic Component Parameters, Component Results Summary, and Creating a New Application Profile.

Customizing the Network Neighborhood

Since we will be interacting with a DUT, we need to define a network that the DUT can route. In this example, we will add 10 endpoints on each side of the firewall.

When creating a new network object instance, it is always easier if you start from a predefined neighborhood as that saves us from making too many modifications. (You’ll find on the BreakingPoint system, predefined networks for various topologies that can be used as a starting point). We will use the existing network neighborhood from the “Hello World” test as a template and modify the ipv4 static host ranges associated with each interface.

As a first step, let’s get the name of the network from the current test Instance (templateNetworkName) and re-use the template to create a new one.

set templateNetworkName [$testInstance cget -neighborhood]
set newNet [$bps createNetwork -template $templateNetworkName]

Use the <networkObj> getAll <net element-type> method to navigate to and update each object defining the IP ranges used in this new Network created above. We can filter the retuned values also by their parameters.

In the below case, I will get a list with all IP ranges defined for “Interface 1,” which is our first reserved port.

set if1_dict [$newNet getAll ip_static_hosts -container {Interface 1} ]

The returned value is a dictionary of ID names and Tcl object instances for each range matching the query.

Example: if1_dict is: {Static Hosts i1_default} ::bps::NetworkClient::networkElement0

In this case, will have only one range of IPs for each interface, so we do not necessarily have to use the foreach loop. We used a loop and an incremental value (“x” in this case) to show how we could easily expand this network to several ranges.

Below we assign 10 hosts in each range with ips for the first one, ips for the second, and so on. All have mask 16 and as gateway.

set if1_dict [$newNet getAll ip_static_hosts -container {Interface 1} ]
foreach { if1_iprange_key_name range_obj }  $if1_dict {
puts "Configuring network ipv4 static host range \"$if1_iprange_key_name\" ..."
        	incr x
        	set err [ $range_obj  configure -count 10 -ip_address 40.1.${x}.1 -netmask 16  -gateway_ip_address -netmask 16]
        	if {$err != ""} 
{puts "Could not configure : $if1_iprange_key_name because $err"}

Similar commands are used for the Interface 2 to set 10 IPs with base IP of, gateway of, and netmask of 16. After all the modifications, we need to save it with a new name and assign it in the test instead of the previous one.

#save the new created
$newNet save -name "nn_firewall" -force
#configure the new created network to be used in the test
$testInstance configure -network "nn_firewall"

This is how the network “nn_firewall” will look if opened in the BreakingPoint UI.

Breaking Point UI

Modifying test parameters.

As explained in the introduction, we will have one application in a single traffic component of the appsim type. We can get to its instance using the “$testInstance getComponents” method, which will return a dictionary of available components in the testInstance.  The dictionary will contain pairs of component name and the Tcl object instance for each. An extra component can be found in the dictionary named “aggstats” that contains general settings and results for all components.

For the names and current values of component parameters, simply execute: <componentInstance> configure.

To get a single value, use the “cget” method:

$appsim_component_obj cget -rateDist.min

Through the below snippet, we used the “configure” command to set the performance parameters of the AppSim component.

For consistency, all the created tests will have the objective set to:

  • Data Rate: 10000 mbps bidirectional per interface traffic
  • Superflow target: 10000000 Simultaneous at a max 1000000 per sec with 1 streams per each flow
  • Load Profile ramp up: 30 s sustain: 75 s, ramp down: 15 s

The idea is to try to achieve the max performance possible with the test while remaining within the constraints/limitation of the test tool.

#Navigating to the object for component named appsim1
set component_dict [$testInstance getComponents] 
set appsim_component_obj [dict get $component_dict appsim1]
#Modify appsim component parameters
$appsim_component_obj configure -rateDist.min          10000
$appsim_component_obj configure -sessions.maxPerSecond 1000000
$appsim_component_obj configure -rampDist.up          30
$appsim_component_obj configure -rampDist.steady      75


Logging results of each tests

There are multiple ways to access stats in BreakingPoint via automation, some of which we will cover in a future blog. In this example, we leverage the summary results statistics available on each component object to gauge the performance of each application.

For this we need to navigate our way to the components and get the result instance for each. In this example, we have only one component called “appsim1”, however the Tcl API has two, the appsim1 component and another called aggstats, which is aggregating stats across multiple components.

#Navigating to the object for component named appsim1
set component_dict [$testInstance getComponents] 
set appsim_component_obj [dict get $component_dict appsim1]
set agregated_components_obj [dict get $component_dict aggstats]

#obtaining the result object for appsim and overall aggregated stats
set appsim_result   [$appsim_component_obj result]
set aggstats_result [$agregated_components_obj result]

Each result instance contains stat names and values specific to the components. To get the stat names available, the method “values” can be used. Ex: “$aggstats_result values”

Get the data using the “get” method. Ex. “$aggstats_result get superFlowsConcurrent”

puts "********* RESULTS ********"
#Printing a result summary by getting the results from appsim.
puts "Traffic combination $trafic_profile TX TPUT:  [$appsim_result get appTxFrameDataRate] mbps"
puts "Traffic combination $trafic_profile RX TPUT:  [$appsim_result get appRxFrameDataRate] mbps"
puts "Overall Super Flow Rate:  [$aggstats_result get superFlowRate]/s"
puts "Overall Concurrent Super Flows:  [$aggstats_result get superFlowsConcurrent]"
puts "Unsuccessful application attempts:  [$appsim_result get appUnsuccessful]"


Cycling through all default BreakingPoint applications, one app at a time

The biggest value-add of this automation is that it can cycle through BreakingPoint’s more than 3,700 application flows, running a two-minute test for each application. You can also provide a subset of application flows, reduce the test durations, and leave the script running for the weekend—and voila, you will have your categorized test runs and performance analysis once you are back. Isn’t that cool?

As a first step to get all their names in Tcl use the <bpsConnection>  listSuperflows“ method to get all the names of the superflows.

We will select the superflows in the order of display and add them to an application profile that will be used in the AppSim component.

#creating a new custom appprofile for the selected flow
set app_profile [$bps createAppProfile]
set profileName "automation101_$sfcleanName"

Then we add the superflow by name ( $sf in this example), save it, and add it to the test.

$app_profile addSuperflow $sf 10
$app_profile save -name $profileName -force
$appsim_component_obj configure -profile $profileName

In the “bps_loopAplications_10gb_10mil_superflows_firewall_test.tcl” example script, this procedure is done in a loop for the list of superflow names obtained with the “ listSuperflows “ method. Below are snippets of the important configurations.

#Get all canned (created by BPS team not by user) flows availlable 
set superFlowList [$bps listSuperflows -limit 4000 -class canned]
puts "There are [llength $superFlowList] pre-defined aplications flows availlable"


Sample Scripts:

  1. bps_http_10gb_10mil_superflows_firewall_test.tcl - > (Link)
  2. bps_loopAplications_10gb_10mil_superflows_firewall_test.tcl -> (Link)

Script #1 executes the “BreakingPoint Bandwidth_HTTP” canned application mix containing only a GET and 200 OK HTTP messages. This serves as a pre-curser example to verify everything is working fine. Script #2 creates and executes tests similar to the one above for each superflow definition available in BreakingPoint. Once a test is completed, it logs the execution results summary in the “report.out.csv” file.

Once the script executes, the output should resemble this:

Script Output

Link to output sample.

Application Throughput Chart

A snippet of the report showing performances of individual application flows once the test run is completed.

Application Block/Run Analysis


The above script is a great example of the important testing that can be done through automation that is otherwise unachievable using manual testing. This test can significantly increase regression coverage for any security device that does deep packet inspections (DPI). The tester can get creative with automations and can actually create different variants of the same test, like increasing the number of IPs, changing TCP parameters to see effects, and inserting encryption on top of the flows.

Stay tuned for upcoming editions of this blog series where we will discuss tips to modify various parameters of a single test and analyze how each of them affects performance.

Author’s Note: This blog is co-written with my colleague Amritam Putatunda and is the 3rd of our automation series. Read the first blog here.