Kris Raney
Distinguished Engineer
Blog

CloudLens (post 3 in a series)

August 27, 2018 by Kris Raney

In my previous posts in this series, I laid out my plan to enable Threat Hunting in a scalable way for a cloud environment by integrating Bro IDS with CloudLens, hosted on Kubernetes, with Elasticsearch and Kibana as the user interface. I also described why Bro will serve as the intrusion detection system in my project.

Next I'd like to explain what CloudLens is and how it fits into the picture. Full disclosure, CloudLens was kind of my brainchild, so it's a foregone conclusion that I would use it. Nevertheless I made it for a reason, and those reasons still apply here.

The problem

First of all, it's worthwhile to discuss why CloudLens is even necessary. If you've been using Bro in your datacenter, you probably had a machine running Bro just hooked up to a network tap or a SPAN port on your network switch. Or maybe you had a packet broker hooked up to various taps, and you had your Bro system plugged into that.

The thing about a cloud environment is, you don't own it, you're a tenant. So you can't just drop a tap into the network, and by and large cloud providers don't provide any kind of interface or API that would allow you to grab your packets straight off the network in a way similar to the datacenter. That leaves only a couple of options.

Avoiding inline packet capture

The most obvious option to get a look at network traffic is to play tricks with routing in your VPC. You can set up a private subnet and have it pass its traffic through a VM that does network address translation, or NAT. This funnels all of your traffic through that VM, and you can either run Bro on the VM directly, or send the traffic from there to somewhere else. You could roll your own, or there are commercial solutions available that work this way.

Packet capture using a NAT gatewayThis approach has some big drawbacks, though.

  • This forces you into a specific network architecture. You want your network architecture to be built to accomodate the app you are running, not the monitoring you want to do for that app.
  • You can only see north-south traffic flowing in and out of your VPC through the NAT gateway. You won't be able to see communications between VMs.
  • This VM becomes a bottleneck, potentially limiting your performance. Cloud providers in general natively offer fantastic bandwidth - until you choke it down through a single VM.
  • This VM also becomes a single point of failure in your network. If it's down, so is all of your communication.
  • This becomes yet another piece of infrastructure that needs long term operations management and automation.

I really had no interest to try and force this kind of network layout on anyone, and I definitely didn't want to manage it myself long term. So, I wanted a less intrusive approach to capturing cloud traffic.

CloudLens

CloudLens uses a docker-based agent for packet capture. This agent gets installed on each VM that you want to monitor. It's relatively simple to roll it out to existing VMs using AWS SSM or by adding into your existing config management. One thing a bit different than other solutions, the CloudLens agent _also_ is installed on the receiving end - for this project, the Bro workers.

The CloudLens configuration is managed through CloudLens backend service. All of the installed agents phone home to this service, and report back metadata about the host where they live. You can search through this list based on that information, and save searches as groups. All of the configuration for CloudLens is based on these groups.

Searching in the CloudLens UI

So you would search for the instances you want to monitor, and save that search as a group. Then search again for your Bro workers, and save that. Draw a line between them, and ta-da! You have packet capture. It's as simple as that.

Sending packets from a source group to a tool group

The great thing is, if you add more VMs because you scale out, or for any other reason, as long as they match your saved search they'll automatically be added to your monitoring. Similarly, if you decide you need more Bro workers to analyze the traffic, as they appear as long as they match your search, the traffic gets distributed to them. It's all based on this search, so it's autonomous. You don't have to write special orchestration.

Also, because the capturing agent sits right in the VM you're monitoring, it's not limited to seeing only north-south traffic. Your Bro workers will see all of your traffic, even east-west traffic within the VPC. You can see traffic going from your VMs to the native services of your cloud provider, for example.

Behind the scenes

Behind the scenes, what happens is that the capturing agent picks up network packets as the flow in and out of the VM. It packages them up in a tunnel, and sends them to one of the agents in the receiving group. The source-side agents automatically divide themselves into equal sized groups, each going to a different receiving agent, so the traffic is effectively balanced over however many receivers exist in the group. The source agents also coordinate so that  only one agent captures traffic that flows between two monitored VMs, and you don't see duplicates.

CloudLens peer-to-peer tunnelingThe CloudLens agents on both ends of this tunnel work together to get through firewalls or NAT. So CloudLens is very flexible about the network architecture. You could have sources and receivers in the same VPC if you like, to keep bandwidth costs low. But you don't have to, you can capture sources in one VPC and deliver them to another one where all of your analysis lives. Sources and receivers don't even have to be in the same cloud service provider. For that matter, they don't have to be on VMs, they could be on hardware in your data center. You simply leave your network structured in whatever way best serves your primary application. The monitoring solution just fits seamlessly into
whatever exists.

On the receiving end, the CloudLens agent creates a virtual network interface, named `cloudlens0` by default. The Bro worker simply has to listen to that interface. The agent takes care of receiving the incoming packets, verifying they're from a trusted source, and aggregating them onto that virtual interface. From Bro's point of view it's just like that interface is connected to a network tap, just like it would be in the data center. CloudLens takes care of all of the details.

What's Next

Bro has been selected for intrusion detection, and CloudLens will provide the plumbing so that it can see my cloud infrastructure's network traffic. In the next post, I'll give a brief overview of Kubernetes and explain why it's my preferred hosting environment for this project. I'll follow that up with discussions about Elasticsearch before getting into the details of how this all fits together and how you can replicate it in your environment.

Resources