Threat Hunting in Kibana (post 11 of a series)
In my previous posts in this series, I laid out my plan to enable Threat Hunting in a scalable way for a cloud environment by integrating Bro IDS with CloudLens, hosted on Kubernetes, with Elasticsearch and Kibana as the user interface. I also gave brief overviews of the key components, how to configure CloudLens to deliver network packets to Bro, and how Bro will be configured. Then I gave instructions on how to set up a Kubernetes cluster that will host Bro. Then, finally, I gave the steps to get Bro running on the cluster.
In this post I will give a brief introduction to how threat hunting works when using Kibana as the interface.
Before you begin
The first time you use a new instance of Kibana, there are a couple of setup steps you'll need to do. Since Elasticsearch can store multiple different kinds of data at once, Kibana wants you to narrow it down to a particular set of data using a search pattern. Data in Elasticsearch is put into index, so what it's looking for is pattern matching by index name.
Kibana will insist that the pattern you use matches on at least one index. That means you can't really set one until Bro logs at least one thing that gets picked up and stuffed into Elasticsearch, creating an index. The Bro Helm chart names its indices starting with
bro- so you'll want to set your pattern matching to
bro-*. Just type that in, and hit
Most of the time with Kibana, you'll be looking at time series data, and Bro data is no exception. The next thing you have to do is select which field in the data is the timestamp on the time series. Select
@timestamp and continue.
Finally, Kibana will go look at the data that's present, gather up all of the field names, and remember their data type. Those names become things you can click on and use in queries. If you have done this very early, before there's much Bro data, some things that show up later may be missing from those lists. You can come back and resolve this later when more data is present by going to "Management", selecting "Index Patterns", choosing "bro-*" as the index pattern to edit, and just hit the refresh button to reload the field names from the data.
Once you have this set up you should be able to go to "Discover" and see a little bit of data on display.
From there, one last thing you'll want to do is choose some fields to show as the columns in the data. For a starting place I suggest clicking
id.orig_h(the originating host IP address)
id.orig_p(the originating host port number)
id.resp_h(the receiving host IP address)
id.resp_p(the receiving host port number)
type(the type of Bro event)
Showing those columns will highlight what generally the event is, and then you can open that row to look at all the detailed data. Of course you're free to add more columns or change this in any way you like, it's merely a suggestion.
Everything you see in Kibana is constrained to fall within a configurable time window. There are a couple of ways to edit what time period is being shown.
In the top right of the Kibana window, there is an indicator showing what the time window is. If you click this, you can edit the time window directly. The simplest way to edit is to select a "Quick" range, probably for the last hour or so. Doing that is a good way to reset and start a threat hunt.
The other way to restrict the time range is to interact with a visualization that shows a time range. On the discover page, there's a timeline showing the count of events over time throughout the range you've selected. If you notice a blip in there that looks interesting, you can click and drag over it in the chart, and Kibana will zoom in on your selected range. The same is true for charts in Visualizations or Dashboards.
When you're just exploring and experimenting, it's a good idea to keep your time range fairly short. That makes queries return data a lot faster, since there's a lot less data to search through. Once you're more confident in what you're searching for, you can always expand the time range to see what similar things have happened over a broader time range.
A simple hunt
To get a feel for how it all works, let's go through a simple threat hunt scenario. This is not going to be a complex, ground-breaking technique - in fact in the interest of simplicity I won't be using techniques with a high expectation of finding anything. I just want something simple that takes advantage of the tools and points out useful features.
A threat hunt should generally start from a hypothesis - something you think you might find if you go looking for it. For my hypothesis, I'm going to build on another tool - a honeypot. (Setting up a honeypot is not covered in this series.)
We have a honeypot set up on the internet based on T-Pot. All sorts of attackers and bots are interacting with this honeypot all the time. Since there's no legitimate reason for anybody to interact with it, I can assume that all of this is malicious traffic.
My hypothesis is simple - if people are doing bad stuff to my honeypot, there's a reasonable chance they are attempting that same bad stuff on my actual network.
Here's a dashboard of what's going on in my honeypot.
If this dashboard looks a little familiar, it should. T-pot uses Kibana to provide its dashboard. See how cool it looks? You can build dashboards for yourself along these lines from your Bro data.
OK, let's take a look at this guy. This is an IP address with a bad reputation that's repeatedly trying to log into my honeypot over ssh using lame username / password combinations.
Let's go to my Bro Kibana and search for his IP address. Does Bro have anything to say about this guy? (It should, because my honeypot is included in the traffic that I'm capturing and sending to Bro.)
I do get 232 results back from Bro. It can be interesting to see what Bro says about the interaction with the honeypot, but ultimately I want to exclude that and look for interactions with other things.
Kibana makes this easy. If you mouse over a column, in this case id.resp_h, the responding host, you will see a couple of magnifying glass icons appear, one with a plus and one with a minus.
Clicking the 'plus' icon will add a filter to your search, showing only results that match that value in that field. In this case, what I want is the 'minus'. That will filter the results _excluding_ anything that matches that value in that field. So I just mouse over my honeypot's address, and click the minus icon.
Now Kibana will re-run the query but exclude my honeypot. As luck would have it, I don't have any other results for this malicious actor.
That filter that I added shows up under the search as a red oval. Since my general hypothesis revolves around this honeypot, I'm surely going to be filtering and unfiltering on this value repeatedly as I try different things. If I mouse over that oval, some controls appear that allow me to delete it or to invert it from plus to minus. There's also a check box that allows me to temporarily disable it. I'll use that to disable it, leaving the filter right there so I can easily toggle it back on.
Widening the net
Since the first guy I picked at random didn't turn up much, let's look for others.
I start by inverting the filter on the honeypot address, so now I'm looking at results for everyone who was talking to the honeypot. You do this by mousing over the filter just under the search bar, finding the magnifying glass icon, and clicking it. In this case that will flip this from being a "minus" (exclude) to a "plus" (matching only.)
In the "Selected Fields" area of Kibana, I click on the "id.orig_h" field. This shows me a query of the top 5 addresses with the most records in my search results - the top 5 guys interacting with my honeypot the most.
From there, I can click the same magnifier 'plus' icon and filter for one of these IPs. Once that filter is added, I can invert the honeypot filter and see who else that address has been talking to.
... With no real result. Let's face it, this particular avenue is kind of a bust.
Now instead of looking for particular IP addresses, how about we look at behaviors and try to find additional instances of those behaviors?
Filtering again to select for events related to the honeypot, let me choose one particular event. Let's take a look at everything Bro has to tell me about this event.
In order to do this, I want to find the column for "uid" and add it to my display. The UID field in Bro is a unique identifier Bro sets for a particular session. Every Bro event related to that session will carry that uid.
Scrolling down through my Bro events, I can choose something that looks interesting. If I then add that UID to my filter, I will see all of the Bro events from that one session. Also from the timeline, I can easily see when the event actually happened.
Good news and bad news
So those are some basic interactions with Kibana that can help you search through your Bro data to look for threats. The bad news is, I didn't find anything nefarious going on in my production network (which is not so exciting for a post on Threat Hunting.)
The good news is, I didn't find anything nefarious going on in my production network (especially considering the simplicity of these searches.) It's important to realize that proving a hypothesis false is still a successful experiment, and has value. Keeping a notebook of the negative results is nearly as important as remembering the positive results.
And in this case, my goal really was to demonstrate how this kind of setup can be useful to you as you do your own threat hunting. I encourage you to get this far and try a similar setup. The ability to interactively "poke around" in your data in a nice, graphical environment can be very enlightening. Not to mention, Kibana's ability to create graphs as visualizations and combine them into dashboards (which I haven't really covered, but is discussed in detail here) can help see a broad overview of what's going on.