Splunk and Free Open-Source Threat Intelligence Feeds

This is a write-up for integrating some readily available free and open-source threat intelligence feeds and block lists into Splunk.

Threat Feeds

The following threat feeds/block lists can be used for many other use cases/applications which I have included below for convenience:

Emerging Threats
AlienVault - Appears to require a login now to visit this page but the script still works fine
SSLBL
ZeuS Tracker
Palevo Tracker
SpyEye Tracker
Malc0de

Creating A Script

After my initial idea to integrate some threat feeds into Splunk, I began to do some research and luckily ran into someone that was thinking along the same lines I was. His name is Keith and his blog can be found here. I used Keith's experience as a starting point to create a small bash script which grabs the threat feeds/block lists and parses them in order to prepare some Splunk-ready input files.

I am by no means an expert nor do I possess much programming experience which was the greatest motivation to continue using bash. This video by Lee Baird at Hack3rcon 3 on Adrian's website Irongeek was very helpful. Lee also has some BackTrack and Kali Linux scripts which automate various tasks with bash scripts here with a related SkyDogCon 2 video here if you are interested.

I have included my script below and I welcome any input or suggestions for improving it. Feel free to modify it for your particular use case or application as needed. You can also download it from here. Please keep in mind if you copy and paste from the blog entry it will not have the correct line endings/wrapping. I should also mention that that I utilized prips which prints IP addresses on a given range in the script. Since some of the feeds provide CIDR notation addresses it was necessary to convert them for my particular use case. So if you plan on using the script make sure you install prips with the following command:

sudo apt-get install prips

I like to place the scripts I use for Splunk in the directory: /opt/splunk/bin/scripts

Updated script to include SSLBL from abuse.ch 07/17/14
Updated script to include Binary Defense Systems Artillery Threat Intelligence Feed and Banlist Feed  Binary Defense Systems
02/10/15

#!/bin/bash 
#Script that downloads the Emerging Threats - Shadowserver C&C List, #Spamhaus 
#DROP Nets, Dshield Top Attackers, Known RBN Nets #and IPs, Compromised IP List, 
#RBN Malvertisers IP List;  AlienVault - IP Reputation Database; ZeuS Tracker - 
#IP Block List; SpyEye Tracker - IP Block List; Palevo Tracker - IP Block List; 
#SSLBL - SSL Blacklist; Malc0de Blacklist; Binary Defense Systems Artillery 
#Threat Intelligence Feed and Banlist Feedand then strips any junk/formatting 
#that can't be used and creates Splunk-ready inputs.    
#   
#Feel free to use and modify as needed   
#   
#Author: Adrian Daucourt based on work from Keith
#(http://#sysadminnygoodness.blogspot.com)   
#
#==============================================================================
#Fix error when calling script from Splunk
#==============================================================================

unset LD_LIBRARY_PATH

#==============================================================================
#Emerging Threats - Shadowserver C&C List, Spamhaus DROP Nets, Dshield Top
#Attackers
#==============================================================================

wget http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt -O 
/tmp/emerging-Block-IPs.txt --no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/emerging_threats_shadowserver_ips.txt

cat /tmp/emerging-Block-IPs.txt | sed -e '1,/# \Shadowserver C&C List/d' -e '/#/,$d' | 
sed -n '/^[0-9]/p' | sed 's/$/ Shadowserver IP/' >> 
/home/ubuntu/downloads/emerging_threats_shadowserver_ips.txt

echo "# Generated: `date`" > /home/ubuntu/downloads/emerging_threats_spamhaus_drop_ips.txt

cat /tmp/emerging-Block-IPs.txt | sed -e '1,/#Spamhaus DROP Nets/d' -e '/#/,$d' | xargs 
-n 1 prips | sed -n '/^[0-9]/p' | sed 's/$/ Spamhaus IP/' >> 
/home/ubuntu/downloads/emerging_threats_spamhaus_drop_ips.txt

echo "# Generated: `date`" > /home/ubuntu/downloads/emerging_threats_dshield_ips.txt

cat /tmp/emerging-Block-IPs.txt | sed -e '1,/#Dshield Top Attackers/d' -e '/#/,$d' | 
xargs -n 1 prips | sed -n '/^[0-9]/p' | sed 's/$/ Dshield IP/' >> 
/home/ubuntu/downloads/emerging_threats_dshield_ips.txt

rm /tmp/emerging-Block-IPs.txt

#==============================================================================
#Emerging Threats - Compromised IP List
#==============================================================================

wget http://rules.emergingthreats.net/blockrules/compromised-ips.txt -O 
/tmp/compromised-ips.txt --no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/emerging_threats_compromised_ips.txt

cat /tmp/compromised-ips.txt | sed -n '/^[0-9]/p' | sed 's/$/ Compromised IP/' >> 
/home/ubuntu/downloads/emerging_threats_compromised_ips.txt

rm /tmp/compromised-ips.txt

#==============================================================================
#Binary Defense Systems Artillery Threat Intelligence Feed and Banlist Feed
#==============================================================================

wget http://www.binarydefense.com/banlist.txt -O /tmp/binary_defense_ips.txt 
--no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/binary_defense_ban_list.txt

cat /tmp/binary_defense_ips.txt | sed -n '/^[0-9]/p' | sed 's/$/ Binary Defense IP/' >> 
/home/ubuntu/downloads/binary_defense_ban_list.txt

rm /tmp/binary_defense_ips.txt

#==============================================================================
#AlienVault - IP Reputation Database
#==============================================================================

wget https://reputation.alienvault.com/reputation.snort.gz -P /tmp --no-check-certificate 
-N

gzip -d /tmp/reputation.snort.gz

echo "# Generated: `date`" > /home/ubuntu/downloads/av_ip_rep_list.txt

cat /tmp/reputation.snort | sed -n '/^[0-9]/p' | sed "s/# //">> 
/home/ubuntu/downloads/av_ip_rep_list.txt

rm /tmp/reputation.snort

#==============================================================================
#SSLBL - SSL Blacklist
#==============================================================================

wget https://sslbl.abuse.ch/blacklist/sslipblacklist.csv -O /tmp/sslipblacklist.csv 
--no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/sslipblacklist.txt

cat /tmp/sslipblacklist.csv | sed -n '/^[0-9]/p' | cut -d',' -f1,3 | sed "s/,/ /" | sed 
's/$/ SSLBL IP/' >> /home/ubuntu/downloads/sslipblacklist.txt

rm /tmp/sslipblacklist.csv

#==============================================================================
#ZeuS Tracker - IP Block List
#==============================================================================

wget https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist -O 
/tmp/zeustracker.txt --no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/zeus_ip_block_list.txt

cat /tmp/zeustracker.txt | sed -n '/^[0-9]/p' | sed 's/$/ Zeus IP/' >> 
/home/ubuntu/downloads/zeus_ip_block_list.txt

rm /tmp/zeustracker.txt

#==============================================================================
#SpyEye Tracker - IP Block List
#==============================================================================

wget https://spyeyetracker.abuse.ch/blocklist.php?download=ipblocklist -O 
/tmp/spyeyetracker.txt --no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/spyeye_ip_block_list.txt

cat /tmp/spyeyetracker.txt | sed -n '/^[0-9]/p' | sed 's/$/ Spyeye IP/' >> 
/home/ubuntu/downloads/spyeye_ip_block_list.txt

rm /tmp/spyeyetracker.txt

#==============================================================================
#Palevo Tracker - IP Block List
#==============================================================================

wget https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist -O 
/tmp/palevotracker.txt --no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/palevo_ip_block_list.txt

cat /tmp/palevotracker.txt | sed -n '/^[0-9]/p' | sed 's/$/ Palevo IP/' >> 
/home/ubuntu/downloads/palevo_ip_block_list.txt

rm /tmp/palevotracker.txt

#==============================================================================
#Malc0de - Malc0de Blacklist
#==============================================================================

wget http://malc0de.com/bl/IP_Blacklist.txt -O /tmp/IP_Blacklist.txt 
--no-check-certificate -N

echo "# Generated: `date`" > /home/ubuntu/downloads/malc0de_black_list.txt

cat /tmp/IP_Blacklist.txt | sed -n '/^[0-9]/p' | sed 's/$/ Malc0de IP/' >> 
/home/ubuntu/downloads/malc0de_black_list.txt

rm /tmp/IP_Blacklist.txt

Creating Indexes

In Splunk we started by creating an index that I named "threat". If you are not familiar with creating Splunk indexes more information can be found here

http://www.deepimpact.io/blog/splunkandfreeopen-sourcethreatintelligencefeeds/index.png?attredirects=0

Scripted Inputs

I then proceeded to create a Splunk scripted input to run the script every four hours or 14400 seconds and set the destination index for this source to "threat", and I also set the sourcetype to "threat" as well. Keep in mind some of these threat feeds or block lists are updated every time you query the list, some every few hours, and some are updated daily or possibly even longer, so don't be abusive in your queries. More information on Splunk scripted inputs can be found here and here.


File Monitoring

You would then be required to tell Splunk to continuously collect data from the files which will be created using our script and index the data as it comes in. Information for File and Directory inputs can be found here. Remember you would have to do this for all the files created by the script:

/home/ubuntu/downloads/av_ip_rep_list.txt
/home/ubuntu/downloads/binary_defense_ban_list.txt    
/home/ubuntu/downloads/emerging_threats_compromised_ips.txt 
/home/ubuntu/downloads/emerging_threats_dshield_ips.txt  
/home/ubuntu/downloads/emerging_threats_rbn_ips.txt    
/home/ubuntu/downloads/emerging_threats_rbn_malvertisers_ips.txt    
/home/ubuntu/downloads/emerging_threats_shadowserver_ips.txt    
/home/ubuntu/downloads/emerging_threats_spamhaus_drop_ips.txt
/home/ubuntu/downloads/palevo_ip_block_list.txt    
/home/ubuntu/downloads/spyeye_ip_block_list.txt    
/home/ubuntu/downloads/zeus_ip_block_list.txt
/home/ubuntu/downloads/malc0de_black_list.txt



After you complete the configuration above you should begin to see data in the Splunk Summary Dashboard when the script runs and begins to create the input files.


Searching and Visualizations

I also use Splunk to gather Intrusion Prevention and Detection System (IPS/IDS) logs. This is where the really interesting part starts. I can now use this new threat feed information and begin to correlate attacks on my IPS/IDS with known threat sources. I can use some readily available free apps for Splunk like the Google Maps for Splunk app and start plotting geolocation information from attacking IPs on a map using the sample search below:

your search here | join type=inner max=0 src [search index=threat] | stats count as _geo_count by src | geoip src | search _geo=* | stats sum(_geo_count) as _geo_count by _geo

You can then get some really cool map visualizations like this:



Answering Questions

You can start answering all kinds of questions about your infrastructure using this simple method:
  • What countries/cities are accessing my web server or a specific directory on that server the most?
  • What country/city has the most dropped/blocked traffic on my firewall or IPS?
  • What country/city is bruteforcing SSH on my Wordpress site?
  • What country/city is attempting SQL injection (SQLi) on my web app?
You can create some dashboard panels like the one below that answers questions like:
  • What are the top 20 known threat source IPs and their threat type and how many occurences were there in the last 30 days?



You are only limited by your creativity! If you have any questions, improvements, or ideas please feel free to comment.

I would like to thank @EmergingThreats, @abuse_ch, @alienvault, @Shadowserver, @malc0de, @binary_defense and all the volunteers who provide us with these great feeds and of course @splunk for making it all possible.


Happy Splunking!