Dynamic Filtering: The ABCs of Network Visibility
For a network monitoring solution to operate efficiently, monitoring data is sent to a central collection point, called a network packet broker (NPB). The captured data needs to be optimized, i.e., edited, before transmission to monitoring tools to process that data. Specifically, the data coming in from the tap is a complete copy of all data, so some of it will need to be filtered before being sent on to the appropriate monitoring tool. Filtering means that only the “right” information is sent to the tools, and can be segmented out so that only certain pieces of information go to specific tools. Data filtering is a core feature for NPBs.
Surprisingly though, there is significant variation in how the filtering process can be accomplished. Some NPBs implement filtering through a command line interface (CLI). Other solutions have internal filtering engines programmed by a drag and drop graphical user interface (GUI). Other solutions using a graphical front end to a CLI engine. How this data filtering is accomplished will affect the speed and quality of the resulting monitoring data. For instance, according to ZK Research, 20% or more of filters created through a CLI process have errors.
What Is Dynamic Filtering?
When considering a network packet broker, it is important to understand its filtering capabilities. Filtering is usually performed in three stages. The first stage is performed at the port where the network is attached (network port). The second stage is a highly capable, port-independent filter that is located between the network port and the port to which the monitoring tool is attached (tool port). The third stage of filtering is performed at the tool port itself. Three-stage filtering is important because filtering at the network port completely eliminates the excluded traffic from being available to all tool ports. Once this traffic is removed, it is no longer available for analysis.
An alternative to this approach is to only filter data at the tool port, but this causes two problems. First, the tool port can be overrun by the volume of traffic coming from the network ports. Second, the interaction between the network filters and the tool filter is complex and not obvious, unless you are well versed in set theory. The port-independent (dynamic) filter is the ideal place to perform the bulk of the filtering, as it is possible to understand exactly what is happening by looking at this single filter definition.
Traditional data (network and tool port) filtering within the NPB runs sequentially. This means that during the process, each tool receives the data it needs, and then, the next tool downstream receives the rest of the data with any overlapping packets from previous tools missing (since that traffic was already claimed by tools earlier in the sequence). Only the first tool in the sequence is guaranteed to receive all of the network traffic that it should be seeing. Subsequent tools may receive clipped data, or no data at all.
Dynamic filters are similar to that of ingress and egress filters except that the dynamic filter is located—and processed—in the middle, between the ingress and egress port filters. If dynamic filtering is used, the ingress filter can be left wide open so that the dynamic filters can segment and then aggregate packets from multiple ports and then send those packets on to the appropriate tool. This solution addresses problems that occur when some packets meet the filter criteria of multiple tools and must be sorted out properly for each tool to do its job. This is referred to as overlapping packets and must be addressed to ensure that your tools can “see” the right packets for successful monitoring.
CLI-based filtering schemes put the onus of correcting this problem on the operator, requiring IT engineers to correctly identify any overlaps, quantify those overlaps, and then, write out detailed filter rules and exceptions to account for the data that needs to go to multiple destinations (tools). The dynamic filter eliminates the time and cost associated with CLI-based filters.
Typical Use Cases
In regards to network monitoring, there are several specific use cases and instances where dynamic filtering is used. Here are some specific situations.
- Fast creation of data filters – Business objectives require quick problem resolution. This includes monitoring data analysis. These data filters need to be created quickly and correctly to prevent improper monitoring tool analysis/conclusions, false positives, and fast troubleshooting. The same filter can be created in a quarter of the time using a dynamic filter engine versus a CLI process.
- Floating data filter creation – Specific NPB filters can be pre-staged and connected to standby troubleshooting tools (e.g., analyzers, Wireshark). This requires an NPB that can handle the creation of these intermediate data filters and hold them in standby mode. These types of filters can dramatically cut data collection times as the troubleshooting filter simply needs to be connected to an incoming network port to the NPB. This can be done remotely using a drag-and-drop interface on the NPB. Once the connection is made, the tool can start capturing critical data in less than 1 minute to reduce troubleshooting time and costs.
- Deployment of adaptive monitoring (automation) – Adaptive monitoring is the ability of the NPB to respond to network commands and make configuration changes. This automation capability improves monitoring response times by being able to respond to network incidents with actions in near real-time. Commands can be received using a REST interface from network management systems (NMS), orchestration systems, SIEMs, etc. In order for this to work, the NPB data filter creation process must be automated, not manual.
The following are some things to keep in mind about visibility solutions:
Ease of use – You want the configuration of the data filters to be quick and simple. Time is often a factor, especially when troubleshooting. Make sure the NPB has a GUI interface and the ability to create both pre-staged (floating) filters and a filter library.
Eliminate complexity and boost productivity – You can eliminate complexity and boost productivity by up to four times or more by using a dynamic filter instead of a manual CLI process.
Eliminate accuracy issues associated with CLI-based filters – Eliminate the 20% or more filter inaccuracy problem associated with the CLI-based filters. The dynamic filter guarantees filter and data accuracy.
More Information on Dynamic Filtering and Network Visibility
Further information about dynamic filtering for visibility solutions can be found here. More information about Ixia network performance, network security and network visibility solutions (along with how they can help generate the insight needed for your business) is available on the Ixia website.