How to Integrate CIF feeds into Bro Intel files

So you set up a SSH honeypot and are gathering data, but what do you do with it?

If you have Bro installed, you can integrate your feeds for monitoring in less than 15 minutes. I’ll show you how to pull a feed of your data from CSIRT Gadgets back down in a format that Bro can ingest and log (and alert if you choose) any time Bro sees any of the data in these feeds. I’m only setting up IP addresses in this tutorial, but you can output feeds of URLs, FQDNs, and even file hashes for Bro to use.

If you’ve never used the Bro Network Security Monitor tool, check out Security Onion it’s great for a quick set up in a small environment, but if you’re looking to inspect multiple gigs of traffic you’ll likely want to configure your own environment. This tutorial uses Bro inside of Security Onion, but the concepts are the same if you installed a custom installation.

1. Configure CSIRTG.io account/token
2. Install PIP, and csirtgsdk
3. Configure csirtg client
4. Create cron job to pull feeds hourly
5. Enable intelligence framework in Bro
6. Install/deploy updated Bro Config

Step 1: Configure CSIRTG account

The goal here is to pull your feed back down in Bro format, in order to do that you do that you need to install and configure the csirtgsdk client.

Since you already created your CSIRT Gadgets account when you set up your SSH honeypot, you  just need to log in and take note of an existing token or create a new one.

Step 2: Install Python/PIP and the csirtgsdk

Ubuntu 16.04 is straightforward, but if you’re running the Security Onion ISO it’s based on Ubuntu 14.04 and you need to manually install a newer version of python/PIP.

For Ubuntu 14.04:
Go in to a ‘working’ directory, and pull the Python 2.7.14 source:

$ wget https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz

Unzip the tarball, and then go in to the directory and configure/build to install Python 2.7.14

$ tar -zxvf Python-2.7.14.tgz
$ cd Python-2.7.14 && ./configure && sudo make altinstall

The altinstall flag tells the installer not to overwrite the oringal python binary, so now you’ll have python 2.7.6 in /usr/bin and python 2.7.14 in /usr/local/bin

Go back to your ‘working’ directory, and pull down this script to pull the latest PIP installer

$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo /usr/local/bin/python2.7 get-pip.py

Now that you have a working python2.7.14/PIP environment you can just install install csirtgsdk

$ sudo pip install csirtgsdk

For Ubuntu 16.04:
You just need to update your system, and then install PIP and then csirtgsdk.

$ sudo apt-get update && sudo apt-get upgrade
$ sudo apt-get install python-pip
$ sudo pip install csirtgsdk

Step 3: Configure the csirtgsdk

Create a .csirtg.yml file under the users home directory. I run csirtg as root, so the file is just /root/.csirtg.yml.

In the yaml file you only need to specify the user and token

# cat /root/.csirtg.yml
user: sfinlon
token: 967f618a576155a996d4423459b3d611

With your yaml file set up you should be able to run # csirtg --feeds to show all of the feeds your user has.

With your csirtgsdk set up, you can run # csirtg --feed ssh-logger --format bro --limit 5 and it will output the first 5 entries in your feed in format for a BRO intel file.

Step 4: Create a cron job

Create a feeds directory at /opt/feeds, and pipe the feed results to ssh.intel. You need to pull the feeds on a regular basis, because new data is always being added. In order to do this, set up a cron job to run every hour (I use a random minute so that the CSIRTG servers don’t get slammed by everyone at the same time. Edit the user cron with crontab -e and go to the bottom of the file and put something similar to this, but change the 18 number to a random number between 1-60.

Bro versions before 2.5.1 have issues with where reading a corrupted entry in the intel file will cause it to stop the intel alerting until Bro is restarted. Workaround is to write the feed to a temp file and then copy it to the one Bro is reading:

# crontab -e
18 * * * * /usr/local/bin/csirtg --feed ssh-scanning --format bro > /opt/feeds/ssh.intel.new && mv /opt/feeds/ssh.intel.new /opt/feeds/ssh.intel 

Step 5: Enable the frameworks in Bro

Now that the feeds are being pulled and updated, you need to configure Bro to read the intel file and set up logging/alerting.

To do this, you need to enable the Intelligence framework, and the Collective-Intel policy script in Bro.

Since we are pulling the feeds hourly, I set the intel feed entries to expire in Bro after 60 minutes. This way if an item is removed from the feed, it doesn’t stay forever.

You just need to tell Bro where the intel feed files are, and to read the files in.

To do this edit /opt/bro/share/bro/site/local.bro and put in the following lines:

@load policy/integration/collective-intel
@load frameworks/intel/seen
@load frameworks/intel/do_notice
@load frameworks/intel/do_expire
redef Intel::item_expiration = 60min;
redef Intel::read_files += {
        "/opt/feeds/ssh.intel",
 };

Step 6: Deploy the new configs

After you edit the local.bro file, you’ll need to install/deploy the updated config. Bro reads the .intel files in real time so any time they are changed/updated you don’t need to do anything else for Bro to update the feeds as they come in. Just running broctl deploy will check the config, update broctl installation and config, and restart the services.

# broctl deploy

Now any time that Bro encounters an IP, URL, or Hash that is in one of the feeds, it will generate an entry in the intel.log file, which on Security Onion is here: /nsm/bro/logs/current/intel.log