Enterprise Security Test Part 2: Why Test, and Debunking Enterprise Testing Myths

January 22, 2015 by Ixia Blog Team

We all learned in our technical schooling that testing is an integral and essential part of good design practice. Testing proves or disproves our assumptions and validates the objectives of the design. More importantly, testing uncovers unknowns that we did not consider as inputs to the complex systems that we are trying to create. These axioms are especially true for security, as failures to protect an organization may cause financial, legal, and job-retention ramifications. Rigorous high-fidelity security testing conducted in a safe environment prior to technology and architecture roll-out into the real-world is a must, before the stakes become too high.

However in the real world, many IT and security professionals choose to make their live production networks the test bed. They use their support line ticketing as the gauge to the success or failure of the implementation. For example, rolling out a next-generation firewall (NGFW) with vender default configuration or a new patch to existing technology, then being forced to do a costly rollback when the IT help desk ticket logs go beyond what can be ignored. Or worse yet, the company’s name is splashed as yet another security headline in the media.

We can all see that this logic is deeply flawed, so why do professionals, who have all the intention of building and running secure and resilient networks, work this way? As I meet with network, security, and DevOps managers at onsite meetings, conferences, and trade shows, I hear then repeat a number of excuses:

  • “Vendors test, so I don’t need to.”
  • “We contract a PEN tester every year to stay compliant, so we don’t need to test.”
  • “We don’t have time to test.”
  • “Management doesn’t understand why we have to delay rollout to test.”
  • “We don’t have budget to test.”
  • “We tested the network when we initially rolled it out years ago.”

Let’s take a deeper look at these fallacies.

“Venders test, so I don’t need to.”

It is definitely true that vendors conduct exhaustive testing of their technologies. Ixia’s core business comes from the network equipment manufacturers (NEMs) who purchase millions of dollars of test equipment and services to create innovative and world-class technologies. However we need to consider the objectives of vendor testing and how that relates to your real-world, one-of-a-kind network and application implementations.

First, vendors can’t reproduce every potential customer environment, nor do they advertise that they do. Their testing objective is to satisfy the advertised data sheet functions in the context of a reproducible fixed use-case. Their data sheet parameters are largely driven by marketing with the objective of creating the biggest eye-catching parameters. For example, when it comes to performance, vendors use industry standards like RFC2544 (UDP/TCP) or RFC3511 (HTTP) to validate performance. Both of these standards are more than 10 years old and use synthetic transport streams with artificial data in the payload. Our modern networks are content-aware and driven by applications. The real performance of a content-aware next-generation technology will behave radically different when passing a string of “AAAA” to a pair of IP address and ports versus application traffic from thousands of users setting up and tearing down multiple sessions. Understanding application and user behavior that uses deep packet inspection (DPI) is compute-intensive and cache-inefficient versus synthetic traffic that is easily hardware-accelerated and cache-efficient.

Second, when it comes to security effectiveness, the parameters are captured while no real valid workloads are active. Is this a valid use case for your networks? Detecting threats is like finding a needle in a haystack. With no haystack, it’s easy to find the needle. For example, do the attacks come on Saturday at 2AM when there is little activity on your network, or it is more likely that you will experience a DDoS attack or exfiltration at the most precarious time when there are thousand of critical transactions that need to be defended? Vendors can’t give you these performance numbers on a data sheet.

What about interworking? How can any vendor give you assurance that their technology will seamlessly interoperate in your complex environment?

Only you can answer these critical questions for your one-of-a-kind network.

“We contract a PEN tester every year to stay compliant, so we don’t need to test.”

PEN testing and vulnerabilities assessments are a critical step and used as evidence of compliance in making effort to secure a network. But compliance does not equal secure. PEN testing has many benefits, but does not cover elements of security resilience. What about knowing how a security technology or the network will behave under real user workload while under attack? How about your network’s and security/IT personnel’s ability to defend against DDoS during peak customer hours? How does PEN testing help you know the right technologies to bring into your network and to right-size the investment?

Only testing security technologies or the whole network with real-world workloads and attacks at scale can let you know that the network will bounce back during and after an attack and stay resilient.

“We don’t have time to test.”

Time is a tricky one. Security professional are under tremendous pressure and understaffed in most organization. Time is fixed and has to be managed. Effective use of time is best explained in the simple time-proven adages: “Measure twice, cut once”

Quality time investment up front following best practices to careful plan, design, implement, and test will dramatically cut the huge time sink of daily firefighting. Intuitively we all know this is true, yet in many cases choose to ignore this reality and focus on urgent firefighting over vital steps that will lower future firefighting.

“Management doesn’t understand why we have to delay rollout to test.”

Dealing with CxO demands can be challenging. They too are under a lot of pressure and in many organizations come from non-technical backgrounds or have been away from technology for a long time and do not understand the complexity of modern content-aware networks. But they too must answer the important question from stakeholders, “How do you know it will work?” Consider these possible answers:

  1. “Because we only buy the best technology.”
  2. “Because it worked yesterday, it should work tomorrow.”
  3. “Because we tested it.”

Number 3 is the only acceptable answer.

However this simple axiom may not be enough for CxOs, as they are primarily focused on business. And in business, it is not about doing the right and smart thing, but rather doing things that make money, save money, or if someone told them to do it (compliance). Testing equals dollars, but we will leave that to the next blog.

“We don’t have budget to test.”

It is true, building robust testing platforms has been difficult and even expensive; both in terms of implementing test technologies and the man power to conduct the tests. In the past, one would have to build massive racks of servers to model user behavior to create realistic traffic loads. Introducing security attacks into the test bed was nearly impossible…DDoS attacks at scale, forget about it.

The options were to do testing at small scale for functionality or resort to low-level brute-force packet generation tools to flood ports.

However the eco-system for testing has advanced quite a bit. Today, technologies available from vendors like Ixia are able to model enterprise-wide scale realistic traffic that can effectively model your own unique network as well as the attacker behavior, all from low-cost appliances (Ixia’s PerfectStorm ONE) or from software that can be loaded on servers or in the cloud (Ixia’s IxNetwork VE, IxChariot 8, IxLoad VE).

The benefits of “knowing” versus “guessing” that it works right, pays dividends in your job security and ultimately results in secure and resilient networks that pay off in the near term and long run.

“We tested the network when we initially rolled it out years ago.”

Networks and the applications change constantly. That is the nature of today’s modern networks. If you take a look back two years ago to the size of the network, average bandwidth consumed by users, and types of services, I am sure there is not much that is the same today. In the modern world of an ever-changing threat landscape, perpetual patch rollouts, and virtualization, a two-year statement on change may be as narrow as one hour ago.

In the next blog, I’ll tackle the money side of why it pays to test.

  • Enterprise Network Test Part 1: Does Buying the “Industry Best” Mean Anything Anymore?
  • Enterprise Network Test Part 3: Quantifying the Dollar Gaines of Enterprise Security Testing
  • Enterprise Network Test Part 4: Keep Cranking the Money Machine Throughout the Lifecycle of Security Investments

Additional Resources

Firewall Test

BreakingPoint – Application and Security Test Solution

Application and Threat Intelligence – Intelligence Subscription Service