Enterprise Security Test Part 3: Quantifying the Dollars of Data Driven Security Investments

January 30, 2015 by Ixia Blog Team

What if you could show quantifiable saving of 20%, 30%, or even 50% of the budget allocated for your next security investment spend— while unequivocally knowing that you made the right decision? Wouldn’t that make you a hero in your organization, with fodder for a great review and a huge bonus?

With security resilience testing, you’ll have the quantifiable data you need to know that you are purchasing the most optimized and economically feasible security devices for your one-of-a-kind network. You’ll be able to negotiate better prices with security device vendors, and right-size your investments. Let’s take a look at why you can’t afford not to test.

Maximize Security Investment with a Data-Driven Onsite PoC

Acquiring new security technologies is an important and highly visible activity, where real-world testing can impact the bottom line. I recently read an Infonetics report that reinforced their publicly available 2013 forecast for enterprise data center security spending. It stated that the average spending on data center security products was expected be $17M. This number may be high or low depending on your network scale and use and the functions you are working to secure.

Now consider the selection process for procuring those security products. Typically IT organizations will do research on products that are available in the market and send out a request for information (RFI), with the goal of narrowing down the selection to 2-3 vendors. Then some level of more-detailed research and evaluation of the solutions begins.

However, it’sIxia’s experience that vendor data sheet performance numbers are not a good estimation of how they will perform on your particular network, with your specific network traffic. To get meaningful performance numbers that will lead to both the purchase of the best gear for your implementation, but significant cost savings as well, enterprises must conduct onsite head-to-head bakeoffs when evaluating new investments.

The diagram below shows you a real PoC that Ixia helped to conduct that demonstrates the deviation from stated performance of a set of industry-leading next-generation firewalls (NGFW) when real-world simulated workloads are applied. You can’t garner this kind of information from data sheets.

Head-to-head throughput performance comparison when handling real-world workloads that go beyond best-case TCP workloads

From the diagram, you can see that the synthetic TCP workload doesn’t provide much information other than validating the best-case data sheet numbers provided by the vendors. However once the real-world workloads were applied, with the target features enabled on the security product, you can see how the technology and its compute-intensive algorithms behaves in your real network. Interesting note, it took just three days to get to this level of quantifiable data.

I love showing this kind of diagram because people get it right away. There are technologies available for you to do the same validation on your investment decisions for a new NGFW or other security device.

Vendors develop their technologies to solve specific problems. Then they take these products to market as general solutions with the hope of reaching a wide customer base. The reality is that performance and security effectiveness will never be the same in any two networks. Choosing the right technology that matches best with your network needs, and then right-sizing that investment, will add quantifiable dollars to the bottom line.

Let’s take NGFWs as an example of price variance that can have significant impact to investment costs. First we will choose four NGFWs that are commonly competitive solutions that enterprise are evaluating today. We choose the four based on their inclusion in the NSS Labs NGFW Security Value Map.

As the devices all have different functionality, performance, and capacity, this is by no means a scientific apples-to-apples comparison, but it does show a wide variance of prices for solutions that vendors advertise to solve similar problems.

Randomly chosen models from NSS report and price available on CDW advertised price
(, Jan 2015)
% to highest advertised price
Cisco ASA 5585-X Security Plus Firewall Edition SSP-20 bundle $50,061.99  
McAfee Next Generation Firewall 1402-C1 $34,707.99 69%
Fortinet FortiGate 1500D $43,658.99 87%
Palo PA-3020 $15,573.99 31%


By using simplified math based on the average data center security spend mentioned above, it works out that there is a huge cost variance if all the products satisfied an IT department’s need. However, it’s likely that not all products will satisfy your needs.

Generalizing that all security product categories have similar price variances, and doing a simple calculations, knowing which solution can satisfy your needs at the lowest cost could result in significant cost savings.

Average Data Center security spend (Infonetics 2015 estimate) = $17M

Highest cost solution = $17M

Lowest cost solution = $17M * 31% = $5M

Range = $5M ---- $17M

Surely that kind of savings is worth conducting your own PoC. From a PoC, you can extract a confidence that a particular cost-effective solution will meet your one-of-a-kind network’s needs.

Maybe it’s time to ask for a bonus by splitting the saving with your company?

ABCs of Negotiating

Whether for a personal or work purchase, everyone wants a discount. Many companies have entire purchasing departments that are graded on deviations to standard price, otherwise called discounts. Negotiating is an art form, but its roots lie in information. PoCs reveal quantifiable data on performance, security effectiveness, and actual feature viability. Getting the right data to your purchasing department removes the need to be heavy-handed or come from a weak position of unbacked-up demands.

Let’s take a look at the real cost savings that can result from a strengthened negotiating position. With the security spend stakes approaching $17M annually, a 15%, 10%, 5%, or even 3% discount significantly impacts the bottom line.

Discount % Discounted $ from $17M budget
15% $2.6M
10% $1.7M
5% $850K
3% $510K


Helping your company reduce capital outlay is good for all the stake-holders, and for your career.

Right Size the Investment

Without knowing exactly how a security solution will perform in your network, the only other option is to guess and work off of a de-rating factor. De-rating by its nature is conservative and forces you to buy up, rather than down. Depending on the technology and the historic experience with a vendor, you may choose 30% or even 70% de-rating from the data sheet.

Let’s take one of the above vendor solutions as an example of sizing. Without confidence in top-end performance scaling, you would need to scale-out by adding another product or scale-up by upgrading to the next higher solution. Either way, the cost impact is significant. Performance and cost do not scale linearly, so scaling-up may be more expensive than scaling-out.

A solution from above table CDW advertised price
(, Jan 2015)
Variance to the lowest advertised price
Fortinet FortiGate 1500D $43,658.99 100%
2x Fortinet FortiGate 1500D $87,317.98 200%
Fortinet FortiGate 3600C $157,844.99 362%


Testing your technologies and network with real-world workloads under attack will give you the data to most-efficiently right-size investments. De-rating and guessing isn’t a winning strategy.

As all networks are unique, I will leave the adding up of the total savings to you. The net is that you can’t afford not to.

In the final blog of this 4-part series, I’ll cover the circle of life.Well, the life cycle of your security resilience testing at least.

Additional Resources

Firewall Test

BreakingPoint – Application and Security Test Solution

Application and Threat Intelligence – Intelligence Subscription Service