Threat Hunting at Scale (part one of a series)
“Threat hunting” is what you do when you know there’s a compromise in your system, and you need to identify exactly what it was, find the scope of the damage, and put rules in place to prevent and / or detect it next time. A favorite element of a threat-hunting solution is the Bro open-source IDS.
Companies are moving to the cloud, but very few companies are cloud-only. Virtually every company today is dealing with a hybrid environment where some resources continue on in the data center, while others are hosted in one or more different cloud environments. It can be challenging to make any tool work across such a diverse environment, and Bro is no exception.
Information about running Bro in the data center is readily available, but I had difficulty finding any information about running it for cloud-based resources. So, I thought I’d sort that out and write up my findings.
In order to adapt Bro for the cloud, there are a few key problems to solve. We need to:
- feed Bro with packets from cloud infrastructure
- host Bro in a way that adapts well to dynamic changes in cloud scale
- where ongoing operations are automatable
- taking advantage of existing tools to make queries easier
My goal is to use a “light touch” in adapting Bro to these requirements. That is, try to work with Bro as it is, without forking and making modifications. In some cases this may not result in an optimal solution, but I favor an incremental approach. Get something simple working first, learn where it can be improved in the process, and then tackle the weak points one by one.
To address the first requirement, I intend to integrate Bro with CloudLens. CloudLens provides a cloud-native approach to capturing packets for analysis that adapts and scales easily.
For the next two requirements, I intend to transplant Bro to be hosted on Kubernetes. Kubernetes is a cluster management solution based on containers, started by Google drawing on their experience from their Borg cluster management, and now owned by the Cloud Native Computing Foundation, or CNCF. It natively includes concepts and elements intended to help with scale and with long term operations.
A lot of the threat hunting descriptions and papers talk about using grep and cut to dig around in log files. I’ve done that kind of thing before, but it really shouldn’t be necessary to use such brute force in today’s world. So along with setting up Bro, I’m going to set up an ELK stack — Elasticsearch, Logstash, and Kibana — to aggregate the logs into a searchable database.
Before I get into the details of the solution, in the next posts in this series, I’ll start with brief introductions to the key architectural elements - Bro, CloudLens, Kubernetes, and Elasticsearch, with a little background on why I've chosen them. Then, I'll continue with posts on how the solution fits together and how to replicate it in your environment.