Last updated at Tue, 05 Nov 2019 14:02:00 GMT
This blog post is a follow-up to my previous blog on using the InsightIDR API to create and maintain threat feeds into InsightIDR.
What is InsightIDR?
InsightIDR is Rapid7’s SIEM solution that helps you centralize and unify your security data, detect behavior behind breaches, and respond to attacks targeting your modern network.
Leveraging third-party threat feeds in InsightIDR
In the previous blog, I explained how to scrape the website https://abuse.ch for possible bad actors that are part of the Feodo Tracker project. In this post, we’ll expand our options and scrape a few more sites to demonstrate how this process works for more complex indicators.
As I mentioned previously, you don’t need any programming experience to use the API. This blog explains in some detail how to write your own scripts to use the Rapid7 REST API, but you can also skip this part and scroll down to the working scripts.
For demonstration purposes, I am using Microsoft’s PowerShell, but please use the scripting language of your choice.
Getting started with the InsightIDR Threat API
You will need your platform API key for this exercise. You will also need to create some private threats and have their associated threat keys handy. If you need to get the steps on how to gather these keys, please see our previous blog.
Let’s start by looking at the next abuse.ch project, Ransomware Tracker. It contains a blocklist of ransomware C&C domains, URLs, and IPs, which you can learn more about here. I want to create a threat for it, too. These blocklists look exactly like the list of bad IPs that we pulled down for Feodo Tracker, except that this time, there are three lists that we need to pull in. This site has a separate list for domains, URLs, and IPs. InsightIDR does not require a separate feed for each data type, so let’s just scrape all three pages into one list of indicators and feed that in.
In my PowerShell script, I want to pull from three sites. I could be fancy with it, but let’s do this the easy way and just add in two more $IOCURL values and read from them:
#Change the value below to the threat list that you wish to import.
$IOCURL1 = "https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt"
$IOCURL2 = "https://ransomwaretracker.abuse.ch/downloads/RW_URLBL.txt"
$IOCURL3 = "https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt"
We also need to append the results from the second and third websites to the file:
#Download the indicators from the specified URL.
Write-Host "Downloading indicators from website"
$IOCblocklist = New-Object Net.WebClient
$IOCblocklist.DownloadString($IOCURL1) > tempindicators.txt
$IOCblocklist.DownloadString($IOCURL2) | Out-File tempindicators.txt -Append
$IOCblocklist.DownloadString($IOCURL3) | Out-File tempindicators.txt -Append
That’s really all there is to it. This will scrape all three sites and put the results into one file. The final script then uploads into the indicated threat feed in InsightIDR. You can get the final version of this script below.
We are on a roll here, so let’s move on to a more complex type of IOC list: URL Haus. You can read about the formation and success of this project in this article, and the list of indicators for it can be downloaded from this site. As you can see, there are more fields in the list. The header for the download explains which fields are included:
# id,dateadded,url,url_status,threat,tags,urlhaus_link
The only field we actually want in our upload is the field called “url”, which is the third field in the list. Let’s go back to our script and change a few things in it to pull down the URL Haus indicators.
First, we need to change $IOCURL1 to point to the URL Haus blocklist:
#Change the value below to the threat list that you wish to import.
$IOCURL1 = "https://urlhaus.abuse.ch/downloads/csv/"
Let’s delete the lines that download from the extra URLs, so we end up with this block of script reading just one website:
#Download the indicators from the specified URLs.
$IOCblocklist = New-Object Net.WebClient
Write-Host "Downloading indicators from website: " $IOCURL1
$IOCblocklist.DownloadString($IOCURL1) > tempindicators.txt
Next, let’s find the line that reformats the downloaded indicator, and change it so that it selects the “url” field, which is the third field in the header (i.e. Select Field3):
Write-Host "Reformat the downloaded list of indicators into a comma-delimited text file"
$IOCblocklist = Import-CSV tempindicators.txt -Header "Field1", "Field2", "Field3", "Field4", "Field5", "Field6" `
| Select Field3 `
| ConvertTo-CSV -NoTypeInformation `
| % {$_ -replace ` '\G(?<start>^|,)(("(?<output>[^,"]*?)"(?=,|$))|(?<output>".*?(?<!")("")*?"(?=,|$)))' ` ,'${start}${output}'} `
| %{$_ -replace '$',','}`
| Out-File $IOCFilePath -fo -en ascii ;
gain, the final version of the script is below. There are quite a few indicators, so this script does take a few minutes to complete.
As you can see, you can use this same method to scrape many other sites that contains IOC and upload them into InsightIDR using the Community Threat feature.
Do you have some comments or interesting use cases for the Threat API? Have some websites you find useful and want to share? Please feel free to comment below to add to the discussion!
Final script for ransomware tracker
############################################################################
# Copyright (c) Rapid7, LLC 2016 All Right Reserved, http://www.rapid7.com/
# All rights reserved. This material contains unpublished, copyrighted
# work including confidential and proprietary information of Rapid7.
############################################################################
#
# abusech_ransomwaretrackers_indicators.ps1
#
# Script version: 2.1.0
# PowerShell Version: 4.0.1.1
# Source: consultant-public
#
# THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY
# KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
# PARTICULAR PURPOSE.
#
# Tags: INSIGHTIDR
#
# Description:
# This script will download indicators from the location specified, place
# them into a CSV file, and then upload them to the private threat feed
# specified. This script is intended to be used with the InsightIDR
# Threat Community threats and uses the InsightiDR REST API v1.
#
#***** VARIABLES TO BE UPDATED *****
#Change the value below to the threat list that you wish to import.
$IOCURL1 = "https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt"
$IOCURL2 = "https://ransomwaretracker.abuse.ch/downloads/RW_URLBL.txt"
$IOCURL3 = "https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt"
#Change this value to the Threat Key for the threat that is being modified.
#Get the threat key by opening your community threat and selecting Threat Key.
$ThreatKey = ""
$headers = @{}
#Enter in your platform API key. This can be generated from the Rapid7 Platform home.
#Log into https://insight.rapid7.com and use the API Management section to generate a key.
$headers["X-Api-Key"] = ""
#***** END OF VARIABLES TO BE UPDATED *****
#These files are used when downloading the indicators and converting them to CSV format.
#They are left insitu on purpose so that you can verify that the script works. If this bothers you,
#use the sections below to delete these temp files after the indicators are uploaded.
#The first file contains a list of indicators scraped from the $IOCURL website. It is not cleaned up.
$IOCOutputFileName = "indicators.txt"
#The CSV file is clean and ready to be uploaded.
$CSVOutputFileName = "indicators.csv"
# Get the location of the script for the output files. Output files
# will be located where script is being run.
$path = Get-Location
$IOCFilePath = "$path\" + "$IOCOutputFileName"
$CSVFilePath = "$path\" + "$CSVOutputFileName"
#This location is where the threats will be uploaded.
$Url = "https://us.api.insight.rapid7.com/idr/v1/customthreats/key/" + $ThreatKey + "/indicators/replace?format=csv"
#Configure the download to use TLS 1.2
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = 'Tls12'
#delete text download file if it exists already
if (Test-Path tempindicators.txt) {
Write-Host "Deleting existing indicator file: tempindicators.txt"
Remove-Item tempindicators.txt
}
#delete text download file if it exists already
if (Test-Path $IOCFilePath) {
Write-Host "Deleting existing indicator file: $IOCFilePath"
Remove-Item $IOCFilePath
}
#delete csv file of downloaded indicators if it exists already
if (Test-Path $CSVFilePath) {
Write-Host "Deleting existing CSV file: $CSVFilePath"
Remove-Item $CSVFilePath
}
#Download the indicators from the specified URLs.
$IOCblocklist = New-Object Net.WebClient
Write-Host "Downloading indicators from website: " $IOCURL1
$IOCblocklist.DownloadString($IOCURL1) > tempindicators.txt
Write-Host "Downloading indicators from website: " $IOCURL2
$IOCblocklist.DownloadString($IOCURL2) | Out-File tempindicators.txt -Append
Write-Host "Downloading indicators from website: " $IOCURL3
$IOCblocklist.DownloadString($IOCURL3) | Out-File tempindicators.txt -Append
#Clean up the temp file of downloaded indicators.
#This script pulls out an indicator from the first field in the list of output. You may need to select a different field.
#Change the Select Field1 line to match whatever field has the indicators in it.
#The rest of this block cleans up the download and adds commas to end of each line (so it will be a CSV file).
Write-Host "Reformat the downloaded list of indicators into a comma-delimited text file"
$IOCblocklist = Import-CSV tempindicators.txt -Header "Field1", "Field2", "Field3", "Field4", "Field5", "Field6" `
| Select Field1 `
| ConvertTo-CSV -NoTypeInformation `
| % {$_ -replace ` '\G(?<start>^|,)(("(?<output>[^,"]*?)"(?=,|$))|(?<output>".*?(?<!")("")*?"(?=,|$)))' ` ,'${start}${output}'} `
| %{$_ -replace '$',','}`
| Out-File $IOCFilePath -fo -en ascii ;
#You can uncomment the following line to delete blank lines from the output, if there are any.
#(Get-Content $IOCFilePath) | ? {$_.trim() -ne "" } | set-content $CSVFilePath
Write-Host "Clean up the file by removing the header"
#Skip reading the first line of the file, which is a header.
#Delete all of the lines that start with a #, which are also part of the header.
Get-Content $IOCFilePath | Select-Object -Skip 1 | Where { $_ -notmatch '^\#' } | Set-Content $CSVFilePath
#checks for indicator csv file. If it does not exist, end script.
if (-not (Test-Path $CSVFilePath)) {
Write-Host "Empty Indicators List, Ending Script without uploading any content"
Break
}
#Command to emulate curl with powershell.
Write-Host "Starting command to connect to API"
$ContentType = 'text/csv'
$Response = Invoke-WebRequest -Uri $url -Headers $headers -InFile $CSVFilePath -Method Post -ContentType $ContentType -UseBasicParsing
Write-Host "Script has finished running. Check your results."
Final script for URL Haus
############################################################################
# Copyright (c) Rapid7, LLC 2016 All Right Reserved, http://www.rapid7.com/
# All rights reserved. This material contains unpublished, copyrighted
# work including confidential and proprietary information of Rapid7.
############################################################################
#
# abusech_urlhaus_indicators.ps1
#
# Script version: 2.1.0
# PowerShell Version: 4.0.1.1
# Source: consultant-public
#
# THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY
# KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
# PARTICULAR PURPOSE.
#
# Tags: INSIGHTIDR
#
# Description:
# This script will download indicators from the location specified, place
# them into a CSV file, and then upload them to the private threat feed
# specified. This script is intended to be used with the InsightIDR
# Threat Community threats and uses the InsightiDR REST API v1.
#
#***** VARIABLES TO BE UPDATED *****
#Change the value below to the threat list that you wish to import.
$IOCURL1 = "https://urlhaus.abuse.ch/downloads/csv/"
#Change this value to the Threat Key for the threat that is being modified.
#Get the threat key by opening your community threat and selecting Threat Key.
$ThreatKey = ""
$headers = @{}
#Enter in your platform API key. This can be generated from the Rapid7 Platform home.
#Log into https://insight.rapid7.com and use the API Management section to generate a key.
$headers["X-Api-Key"] = ""
#***** END OF VARIABLES TO BE UPDATED *****
#These files are used when downloading the indicators and converting them to CSV format.
#They are left insitu on purpose so that you can verify that the script works. If this bothers you,
#use the sections below to delete these temp files after the indicators are uploaded.
#The first file contains a list of indicators scraped from the $IOCURL website. It is not cleaned up.
$IOCOutputFileName = "indicators.txt"
#The CSV file is clean and ready to be uploaded.
$CSVOutputFileName = "indicators.csv"
# Get the location of the script for the output files. Output files
# will be located where script is being run.
$path = Get-Location
$IOCFilePath = "$path\" + "$IOCOutputFileName"
$CSVFilePath = "$path\" + "$CSVOutputFileName"
#This location is where the threats will be uploaded.
$Url = "https://us.api.insight.rapid7.com/idr/v1/customthreats/key/" + $ThreatKey + "/indicators/replace?format=csv"
#Configure the download to use TLS 1.2
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = 'Tls12'
#delete text download file if it exists already
if (Test-Path tempindicators.txt) {
Write-Host "Deleting existing indicator file: tempindicators.txt"
Remove-Item tempindicators.txt
}
#delete text download file if it exists already
if (Test-Path $IOCFilePath) {
Write-Host "Deleting existing indicator file: $IOCFilePath"
Remove-Item $IOCFilePath
}
#delete csv file of downloaded indicators if it exists already
if (Test-Path $CSVFilePath) {
Write-Host "Deleting existing CSV file: $CSVFilePath"
Remove-Item $CSVFilePath
}
#Download the indicators from the specified URLs.
$IOCblocklist = New-Object Net.WebClient
Write-Host "Downloading indicators from website: " $IOCURL1
$IOCblocklist.DownloadString($IOCURL1) > tempindicators.txt
#Clean up the temp file of downloaded indicators.
#This script pulls out an indicator from the first field in the list of output. You may need to select a different field.
#Change the Select Field1 line to match whatever field has the indicators in it.
#The rest of this block cleans up the download and adds commas to end of each line (so it will be a CSV file).
Write-Host "Reformat the downloaded list of indicators into a comma-delimited text file"
$IOCblocklist = Import-CSV tempindicators.txt -Header "Field1", "Field2", "Field3", "Field4", "Field5", "Field6" `
| Select Field3 `
| ConvertTo-CSV -NoTypeInformation `
| % {$_ -replace ` '\G(?<start>^|,)(("(?<output>[^,"]*?)"(?=,|$))|(?<output>".*?(?<!")("")*?"(?=,|$)))' ` ,'${start}${output}'} `
| %{$_ -replace '$',','}`
| Out-File $IOCFilePath -fo -en ascii ;
#You can uncomment the following line to delete blank lines from the output, if there are any.
#(Get-Content $IOCFilePath) | ? {$_.trim() -ne "" } | set-content $CSVFilePath
Write-Host "Clean up the file by removing the header"
#Skip reading the first two lines of the file, which are headers describing the fields in the file.
#Delete all of the lines that start with a #, which are also part of the header.
Get-Content $IOCFilePath | Select-Object -Skip 2 | Where { $_ -notmatch '^\#' } | Set-Content $CSVFilePath
#checks for indicator csv file. If it does not exist, end script.
if (-not (Test-Path $CSVFilePath)) {
Write-Host "Empty Indicators List, Ending Script without uploading any content"
Break
}
#Command to emulate curl with powershell.
Write-Host "Starting command to connect to API"
$ContentType = 'text/csv'
$Response = Invoke-WebRequest -Uri $url -Headers $headers -InFile $CSVFilePath -Method Post -ContentType $ContentType -UseBasicParsing
Write-Host "Script has finished running. Check your results."
