Last updated at Mon, 11 Mar 2024 17:41:12 GMT

Have you ever noticed how things that sound really difficult can sometimes be surprisingly simple to do? I am an avid knitter, but I didn’t try knitting a cabled sweater for years because cables look hard. A few months ago, when attempting the feat, it was a pleasant surprise to find that knitting cables looks a lot harder than it actually is!

The same applies to using the InsightIDR REST API. In this blog, I am going to explain how to automate updating threat feeds in InsightIDR by using the REST API. As you will see, it is probably a lot easier than you expected it to be.

In addition to all of the built-in detections in InsightIDR, you can add in your own list of Indicators of Compromise (IoC). You can get more information about this feature here, including the many reasons why you might want to add in your own threat feeds.

Adding your own list of Indicators of Compromise (IoC) to your threat feeds in InsightIDR.

It is easy to add in bad IPs, URLs, domains, and hashes into InsightIDR a few at a time, but who wants to update a threat list manually, right? As IoCs often change frequently, you need a way to automate uploading IoCs into your cloud SIEM.

You don’t need any programming experience to use the API

While in later blogs we’ll go over how to write your own scripts for the Rapid7 REST API, this blog focuses on getting started by running some existing scripts that will allow you to upload to your private threats in InsightIDR.

Check out the video below and follow along:

Step 1: Add a Threat in InsightIDR

Let’s get our threat key, which we will need for the API. Every threat has its own threat key, which is used to upload the indicators into that threat.

Start by creating a private threat in InsightIDR, which you will find under Settings -> Alert Settings -> Community Threats. Select Add Threat, and fill out the fields for your threat feed. You must have at least one indicator in order to save the threat, so I have entered in a harmless IP address in as threat. This IP address will be removed from the threat later on automatically with our script, so it is not important what you use for this first indicator.

Adding a custom threat in InsightIDR

Step 2: Get your Threat API Key

After saving the threat, select View for the same threat so that you can get the threat key. Navigate to the Threat Key dropdown and fill out the form displayed with these selections: CSV, Replace existing indicators, and File Upload.

This threat key will be used to uniquely identify the custom threats you care about.

Step 3: Get your Platform API Key

Now that we have the Threat API Key, we need to get a Platform API Key. You need both to upload indicators. Note that you need to be an Insight Platform admin to do this.

Read about this key and the REST API [here](https://help.rapid7.com/insightidr/en-us/api/v1/docs.html#]. To get your platform API Key, follow the instructions by logging in to https://insight.rapid7.com, clicking the gear icon in the top right, then selecting API Keys. Once here, click the button to generate an organization key to use.

Step 4: Update an Existing Script

While there are loads of threat feeds floating around, the three scripts used in this blog focus on downloading indicators from the popular open-source website, https://abuse.ch. Later blogs will discuss editing the scripts to work with other sites.

Before continuing, you may wish to take a few minutes to browse to https://abuse.ch and read about the Feodo Tracker, Ransomware Tracker, and URL Haus projects.

Pick one of the three scripts below, copy it, and paste it into a text editor. Add in your Threat and Platform keys and save the file with a “ps1” (ps1 for PowerShell) extension. Finally, run it as a PowerShell script from any system that supports PowerShell. I recommend running the script to verify it works before proceeding to the next step.

Step 5: Schedule the script(s) to run

Now that you have a working script, you will likely want to schedule it to run on a regular cadence. The frequency for updating the indicators depends on what they are, but you would typically update them at least once a day.

If your InsightIDR collector is running a Windows operating system, you can use it as the host for these scripts. However, you can use any Windows-based workstation or server that you want. To automatically update a threat in InsightIDR, use a scheduling tool like Task Scheduler to run the script on a schedule.

Maybe you’re not yet ready to knit up a cabled sweater, but I do hope that you agree with me that uploading IOCs into InsightIDR isn’t difficult to do with our REST API. If you'd like to learn more, please check out part two of this series, Unlocking the Power of the InsightIDR Threat API, Part 2.

Start a free trial of InsightIDR today to leverage these capabilities

Get Started

Script 1: Feodo Tracker

############################################################################
# Copyright (c) Rapid7, LLC 2019  https://www.rapid7.com/
# All rights reserved. This material contains unpublished, copyrighted
# work including confidential and proprietary information of Rapid7.
############################################################################
#
# abusech_feodo_indicators.ps1
#
# Script version: 2.1.1
# PowerShell Version: 4.0.1.1
# Source: consultant-public
#
# THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY 
# KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
# PARTICULAR PURPOSE.
#
# Tags: INSIGHTIDR
# 
# Description:
# This script will download indicators from the location specified, place
# them into a CSV file, and then upload them to the private threat feed
# specified.  This script is intended to be used with the InsightIDR 
# Threat Community threats and uses the InsightiDR REST API v1.
#

#***** VARIABLES TO BE UPDATED *****

#Change the value below to the threat list that you wish to import.
#$IOCURL = " [ paste in the Target Threat List to import ] "
$IOCURL = "https://feodotracker.abuse.ch/downloads/ipblocklist.txt"

#Enter the Threat Key for the threat that is being modified.  
#Get the threat key by opening your community threat and selecting Threat Key.
$ThreatKey = " [ paste in your Threat Key from IDR Threats ] "

$headers = @{}
#Enter in your platform API key.  This can be generated from the Rapid7 Platform home.
#Log into https://insight.rapid7.com and use the API Management section to generate a key.
$headers["X-Api-Key"] = " [ paste in your Platform API Key ]  "

#***** END OF VARIABLES TO BE UPDATED *****

#These files are used when downloading the indicators and converting them to CSV format.
#They are left in situ on purpose so that you can verify that the script works.  If this bothers you,
#use the sections below to delete these temp files after the indicators are uploaded.
#The first file contains a list of indicators scraped from the $IOCURL website.  It is not cleaned up.
$IOCOutputFileName = "indicators.txt"
#The CSV file is clean and ready to be uploaded.
$CSVOutputFileName = "indicators.csv"

# Get the location of the script for the output files.  Output files 
# will be located where script is being run.
$path = Get-Location
$IOCFilePath = "$path\" + "$IOCOutputFileName"
$CSVFilePath = "$path\" + "$CSVOutputFileName"

#This location is where the threats will be uploaded.
$Url = "https://us.api.insight.rapid7.com/idr/v1/customthreats/key/" + $ThreatKey + "/indicators/replace?format=csv"

#Configure the download to use TLS 1.2
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = 'Tls12'

#delete text download file if it exists already
if (Test-Path $IOCFilePath) {
    Write-Host "Deleting existing indicator file: $IOCFilePath"
    Remove-Item $IOCFilePath
    }

#delete csv file of downloaded indicators if it exists already
if (Test-Path $CSVFilePath) {
    Write-Host "Deleting existing CSV file: $CSVFilePath"
    Remove-Item $CSVFilePath
    }
#Download the indicators from the specified URL.
Write-Host "Downloading indicators from website"
$IOCblocklist = New-Object Net.WebClient
$IOCblocklist.DownloadString($IOCURL) > tempindicators.txt

#checks for blank text file and exits the program if the file is blank
Get-Content tempindicators.txt | Measure-Object -word
if ($word -eq 0){
    Write-Host "Empty Indicators List, Ending Script"
    Break
    }

#Clean up the temp file of downloaded indicators.
#This script pulls out an indicator from the first field in the list of output.  You may need to select a different field.
#Change the Select Field1 line to match whatever field has the indicators in it.
#The rest of this block cleans up the download and adds commas to end of each line (so it will be a CSV file).
	
$IOCblocklist = Import-CSV tempindicators.txt -Header "Field1", "Field2", "Field3", "Field4", "Field5", "Field6" `
	| Select Field1 `
	| ConvertTo-CSV -NoTypeInformation `
	| % {$_ -replace  ` '\G(?<start>^|,)(("(?<output>[^,"]*?)"(?=,|$))|(?<output>".*?(?<!")("")*?"(?=,|$)))' ` ,'${start}${output}'} `
	| %{$_ -replace '$',','}`
	| Out-File $IOCFilePath -fo -en ascii ; 


#You can uncomment the following line to delete blank lines from the output, if there are any.
#(Get-Content $IOCFilePath) | ? {$_.trim() -ne "" } | set-content $CSVFilePath
Write-Host "Clean up the file by removing the header"
#Skip reading the first line of the file, which is a header.
#Delete all of the lines that start with a #, which are also part of the header.
Get-Content $IOCFilePath | Select-Object -Skip 1 | Where { $_ -notmatch '^\#' } | Set-Content $CSVFilePath

#Command to emulate curl with PowerShell.
Write-Host "Starting command to connect to API"
$ContentType = 'text/csv'
$Response = Invoke-WebRequest -Uri $url -Headers $headers -InFile $CSVFilePath -Method Post -ContentType $ContentType -UseBasicParsing

Write-Host "Script has finished running. Check your results."

Final script for ransomware tracker

############################################################################
# Copyright (c) Rapid7, LLC 2019  https://www.rapid7.com/
# All rights reserved. This material contains unpublished, copyrighted
# work including confidential and proprietary information of Rapid7.
############################################################################
#
# abusech_ransomwaretrackers_indicators.ps1
#
# Script version: 2.1.0
# PowerShell Version: 4.0.1.1
# Source: consultant-public
#
# THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY 
# KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
# PARTICULAR PURPOSE.
#
# Tags: INSIGHTIDR
# 
# Description:
# This script will download indicators from the location specified, place
# them into a CSV file, and then upload them to the private threat feed
# specified.  This script is intended to be used with the InsightIDR 
# Threat Community threats and uses the InsightiDR REST API v1.
#

#***** VARIABLES TO BE UPDATED *****

#Change the value below to the threat list that you wish to import.
$IOCURL1 = "https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt"
$IOCURL2 = "https://ransomwaretracker.abuse.ch/downloads/RW_URLBL.txt"
$IOCURL3 = "https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt"

#Change this value to the Threat Key for the threat that is being modified.  
#Get the threat key by opening your community threat and selecting Threat Key.
$ThreatKey = ""

$headers = @{}
#Enter in your platform API key.  This can be generated from the Rapid7 Platform home.
#Log into https://insight.rapid7.com and use the API Management section to generate a key.
$headers["X-Api-Key"] = ""

#***** END OF VARIABLES TO BE UPDATED *****

#These files are used when downloading the indicators and converting them to CSV format.
#They are left insitu on purpose so that you can verify that the script works.  If this bothers you,
#use the sections below to delete these temp files after the indicators are uploaded.
#The first file contains a list of indicators scraped from the $IOCURL website.  It is not cleaned up.
$IOCOutputFileName = "indicators.txt"
#The CSV file is clean and ready to be uploaded.
$CSVOutputFileName = "indicators.csv"

# Get the location of the script for the output files.  Output files 
# will be located where script is being run.
$path = Get-Location
$IOCFilePath = "$path\" + "$IOCOutputFileName"
$CSVFilePath = "$path\" + "$CSVOutputFileName"

#This location is where the threats will be uploaded.
$Url = "https://us.api.insight.rapid7.com/idr/v1/customthreats/key/" + $ThreatKey + "/indicators/replace?format=csv"

#Configure the download to use TLS 1.2
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = 'Tls12'

#delete text download file if it exists already
if (Test-Path tempindicators.txt) {
    Write-Host "Deleting existing indicator file: tempindicators.txt"
    Remove-Item tempindicators.txt
    }

#delete text download file if it exists already
if (Test-Path $IOCFilePath) {
    Write-Host "Deleting existing indicator file: $IOCFilePath"
    Remove-Item $IOCFilePath
    }

#delete csv file of downloaded indicators if it exists already
if (Test-Path $CSVFilePath) {
    Write-Host "Deleting existing CSV file: $CSVFilePath"
    Remove-Item $CSVFilePath
    }
#Download the indicators from the specified URLs.

$IOCblocklist = New-Object Net.WebClient
Write-Host "Downloading indicators from website: " $IOCURL1
$IOCblocklist.DownloadString($IOCURL1) > tempindicators.txt
Write-Host "Downloading indicators from website: " $IOCURL2
$IOCblocklist.DownloadString($IOCURL2) | Out-File tempindicators.txt -Append 
Write-Host "Downloading indicators from website: " $IOCURL3
$IOCblocklist.DownloadString($IOCURL3) | Out-File tempindicators.txt -Append 

#Clean up the temp file of downloaded indicators.
#This script pulls out an indicator from the first field in the list of output.  You may need to select a different field.
#Change the Select Field1 line to match whatever field has the indicators in it.
#The rest of this block cleans up the download and adds commas to end of each line (so it will be a CSV file).
	
Write-Host "Reformat the downloaded list of indicators into a comma-delimited text file"	
$IOCblocklist = Import-CSV tempindicators.txt -Header "Field1", "Field2", "Field3", "Field4", "Field5", "Field6" `
	| Select Field1 `
	| ConvertTo-CSV -NoTypeInformation `
	| % {$_ -replace  ` '\G(?<start>^|,)(("(?<output>[^,"]*?)"(?=,|$))|(?<output>".*?(?<!")("")*?"(?=,|$)))' ` ,'${start}${output}'} `
	| %{$_ -replace '$',','}`
	| Out-File $IOCFilePath -fo -en ascii ; 


#You can uncomment the following line to delete blank lines from the output, if there are any.
#(Get-Content $IOCFilePath) | ? {$_.trim() -ne "" } | set-content $CSVFilePath
Write-Host "Clean up the file by removing the header"
#Skip reading the first line of the file, which is a header.
#Delete all of the lines that start with a #, which are also part of the header.
Get-Content $IOCFilePath | Select-Object -Skip 1 | Where { $_ -notmatch '^\#' } | Set-Content $CSVFilePath

#checks for indicator csv file.  If it does not exist, end script.	
if (-not (Test-Path $CSVFilePath)) {
	Write-Host "Empty Indicators List, Ending Script without uploading any content"
	Break
	}

#Command to emulate curl with powershell.
Write-Host "Starting command to connect to API"
$ContentType = 'text/csv'
$Response = Invoke-WebRequest -Uri $url -Headers $headers -InFile $CSVFilePath -Method Post -ContentType $ContentType -UseBasicParsing

Write-Host "Script has finished running.  Check your results."

Final script for URL Haus

############################################################################
# Copyright (c) Rapid7, LLC 2019  https://www.rapid7.com/
# All rights reserved. This material contains unpublished, copyrighted
# work including confidential and proprietary information of Rapid7.
############################################################################
#
# abusech_urlhaus_indicators.ps1
#
# Script version: 2.1.0
# PowerShell Version: 4.0.1.1
# Source: consultant-public
#
# THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY 
# KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
# PARTICULAR PURPOSE.
#
# Tags: INSIGHTIDR
# 
# Description:
# This script will download indicators from the location specified, place
# them into a CSV file, and then upload them to the private threat feed
# specified.  This script is intended to be used with the InsightIDR 
# Threat Community threats and uses the InsightiDR REST API v1.
#

#***** VARIABLES TO BE UPDATED *****

#Change the value below to the threat list that you wish to import.
$IOCURL1 = "https://urlhaus.abuse.ch/downloads/csv/"

#Change this value to the Threat Key for the threat that is being modified.  
#Get the threat key by opening your community threat and selecting Threat Key.
$ThreatKey = ""

$headers = @{}
#Enter in your platform API key.  This can be generated from the Rapid7 Platform home.
#Log into https://insight.rapid7.com and use the API Management section to generate a key.
$headers["X-Api-Key"] = ""

#***** END OF VARIABLES TO BE UPDATED *****

#These files are used when downloading the indicators and converting them to CSV format.
#They are left insitu on purpose so that you can verify that the script works.  If this bothers you,
#use the sections below to delete these temp files after the indicators are uploaded.
#The first file contains a list of indicators scraped from the $IOCURL website.  It is not cleaned up.
$IOCOutputFileName = "indicators.txt"
#The CSV file is clean and ready to be uploaded.
$CSVOutputFileName = "indicators.csv"

# Get the location of the script for the output files.  Output files 
# will be located where script is being run.
$path = Get-Location
$IOCFilePath = "$path\" + "$IOCOutputFileName"
$CSVFilePath = "$path\" + "$CSVOutputFileName"

#This location is where the threats will be uploaded.
$Url = "https://us.api.insight.rapid7.com/idr/v1/customthreats/key/" + $ThreatKey + "/indicators/replace?format=csv"

#Configure the download to use TLS 1.2
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = 'Tls12'

#delete text download file if it exists already
if (Test-Path tempindicators.txt) {
    Write-Host "Deleting existing indicator file: tempindicators.txt"
    Remove-Item tempindicators.txt
    }

#delete text download file if it exists already
if (Test-Path $IOCFilePath) {
    Write-Host "Deleting existing indicator file: $IOCFilePath"
    Remove-Item $IOCFilePath
    }

#delete csv file of downloaded indicators if it exists already
if (Test-Path $CSVFilePath) {
    Write-Host "Deleting existing CSV file: $CSVFilePath"
    Remove-Item $CSVFilePath
    }
#Download the indicators from the specified URLs.
$IOCblocklist = New-Object Net.WebClient
Write-Host "Downloading indicators from website: " $IOCURL1
$IOCblocklist.DownloadString($IOCURL1) > tempindicators.txt

#Clean up the temp file of downloaded indicators.
#This script pulls out an indicator from the first field in the list of output.  You may need to select a different field.
#Change the Select Field1 line to match whatever field has the indicators in it.
#The rest of this block cleans up the download and adds commas to end of each line (so it will be a CSV file).
	
Write-Host "Reformat the downloaded list of indicators into a comma-delimited text file"	
$IOCblocklist = Import-CSV tempindicators.txt -Header "Field1", "Field2", "Field3", "Field4", "Field5", "Field6" `
	| Select Field3 `
	| ConvertTo-CSV -NoTypeInformation `
	| % {$_ -replace  ` '\G(?<start>^|,)(("(?<output>[^,"]*?)"(?=,|$))|(?<output>".*?(?<!")("")*?"(?=,|$)))' ` ,'${start}${output}'} `
	| %{$_ -replace '$',','}`
	| Out-File $IOCFilePath -fo -en ascii ; 


#You can uncomment the following line to delete blank lines from the output, if there are any.
#(Get-Content $IOCFilePath) | ? {$_.trim() -ne "" } | set-content $CSVFilePath
Write-Host "Clean up the file by removing the header"
#Skip reading the first two lines of the file, which are headers describing the fields in the file.
#Delete all of the lines that start with a #, which are also part of the header.
Get-Content $IOCFilePath | Select-Object -Skip 2 | Where { $_ -notmatch '^\#' } | Set-Content $CSVFilePath

#checks for indicator csv file.  If it does not exist, end script.	
if (-not (Test-Path $CSVFilePath)) {
	Write-Host "Empty Indicators List, Ending Script without uploading any content"
	Break
	}

#Command to emulate curl with powershell.
Write-Host "Starting command to connect to API"
$ContentType = 'text/csv'
$Response = Invoke-WebRequest -Uri $url -Headers $headers -InFile $CSVFilePath -Method Post -ContentType $ContentType -UseBasicParsing

Write-Host "Script has finished running.  Check your results."