Last updated at Thu, 11 Jan 2018 15:19:32 GMT
Many security teams use endpoint threat detection solutions to detect and respond to threats like malware, credential theft, and more. In a common architecture using a SIEM or Log Management solution, alerts from endpoint detection products can be managed and correlated with telemetry from other solutions or logs, and validated:
Generally, a human being has to get involved anywhere from the third step forward. Can we do better?
Using a typical architecture with a real endpoint threat detection solution (Sysdig Falco) along with Splunk acting as our SIEM, we’ll show you how you can use Komand to automate detection, enrichment, notification, and response.
The Setup: Our Threat Detection Architecture
Below, we have a typical architecture for an AWS environment. Our bastion host represents the centralized chokepoint for all login activity; any user must log in here before they can access any of the servers in our virtual private cloud. All login activity is protected by Duo Security 2FA authentication.
On each EC2 server in our VPC, we’ve configured an open source endpoint threat detection and response tool, Sysdig Falco, to perform security monitoring. Falco is a simple product that allows you to configure rules to trigger on suspicious behaviors on Linux hosts. Alerts from Falco are fed right into Splunk, which trigger meta alerts we’ll be able to instrument via Komand.
For example, we've configured a Splunk alert that triggers on unauthorized Privilege Escalations on our servers, as reported by Sysdig Falco. Here's what the raw event looks like:
20:27:05.363221899: Warning Sudo/su privilege escalation detected. (rule=privilege_escalation user=jandre command='sudo bash')
As part of their workflow to validate and respond to this potential threat, a security team responding to this alert will need to perform additional steps in various stages of that workflow. Sample activities include:
| STAGE | ABOUT | EXAMPLES | FREQUENCY | ERROR SENSITIVITY |
|---|---|---|---|---|
| Enrichment, or context gathering | Before a security analyst can scope a threat and determine if it is a false positive or a real violation, context from additional systems are needed. | Querying Splunk for related login activity logs on the bastion host Enrichment the IP addresses in the login activity logs with GeoIP information |
Very frequent | Low |
| Notification, or escalation | Once a potential threat is suspicious, additional team members may be notified for more investigation, or a ticket created to track this incident with other suspected artifacts. | Creating a ticket and assigning it to a senior team member for investigation Asking the employee to verify if they can remember conducting the suspicious behavior |
Medium | Low to Medium (could lead to frustration or inefficiencies too many false positive notifications happen) |
| Response | If a threat is confirmed, steps should be taken to contain the threat. If a breach is suspected, additional forensic activity may be required | Disabling a user account | Infrequent | High (could cause business disruption of an error occurs) |
A process like this may take hours to perform manually. Using Komand, we can automate activities across all stages of investigation -> response, reducing that time to 30 minutes or even less. As part of our response to the endpoint threat, we’ll demonstrate automating the notification portions using Slack, and the containment of user accounts in Duo Security by disabling the user accounts.
Part 1
Reacting to Splunk Alerts: Configuring the Komand Splunk Trigger
Using Komand’s Splunk Alert trigger, we can hook our Privilege Escalation alert for automation. It’s as easy as supplying the name of the Splunk alert we want to instrument:
That’s it! Now we're ready to execute automation against our Splunk alerts!
Part 2
Enriching the Original Alert
The alert we’ll get from Splunk isn’t very useful. Remember, this what the alert looks like coming from Sysdig falco:
20:27:05.363221899: Warning Sudo/su privilege escalation detected. (rule=privilege_escalation user=jandre command='sudo bash')
We’ll also get some information from Splunk (the source hostname).
As an analyst, one thing I may want to correlate this activity with is the login activity of the user account: just a quick check against a history to see if the user is logged in from a new or unusual location. Remember, the logs are on the bastion host. Using another Splunk query, I can pull some of this information from the bastion host:
The IP addresses shown as login_ip can then be evaluated against a GeoIP lookup database to get me a full picture of who logged in, where.
Great news is we can automate these follow-on queries using Komand, and then use our GeoIP plugin to lookup host information!
Using the Splunk search action, let’s perform a query on bastion host for the user account referenced in the logs:
We can also add use the GeoIP plugin to correlate the login_ip information with location lookups:
This is just one example of enrichment tasks you can perform against your Splunk alert. From here, you can add additional enrichment steps: querying the endpoint for active command activity, looking up the employee information, adding more Splunk queries... The sky’s the limit!
Notifying the Security Team
Armed with this additional information, let’s create a notification for the security team via Slack with the additional context attached:
You may choose at this point to create a ticket in a solution like JIRA, ServiceNow, or another case management platform. It’s really up to you and your process.
Komand makes it easy to modify your processes, and using our Test feature right in the workflow builder, you can validate your changes. Switching the notification mechanism in Komand is only a few clicks away.
Performing Human-Validated Response
If it turns out credentials have been compromised, it may be appropriate to disable them as part of our incident response process. We can easily add a human decision point here which allows us to validate whether or not credentials should be disabled in Duo, or if the alert should be dismissed.
Using the Duo modify user action via the Duo Admin API, we can set an account to disabled, effectively preventing a user from logging into the bastion host until more thorough investigation actions can be taken.
Wrapping It Up
That’s how easy it is to build a detect <-> investigate <-> notify <-> response workflow in Komand. By instrumenting existing tools with Komand’s simple workflow automation layer, you can take process execution time from hours to minutes; accelerating your team’s productivity and reliability.
If you'd like to see the power of Komand for yourself, you can view our recent demo webinar recording.
