OSSEC Series: Configuration Pitfalls

Last updated at Thu, 14 Dec 2023 19:51:05 GMT

Synopsis:

OSSEC is a popular host intrusion and log analysis system. It’s a great tool, and when configured and customized properly it can be a very powerful and holistic addition to your environment.

In this article I will list a number of problems I’ve encountered while using OSSEC over the years. Many of these are the result of incomplete documentation.

The purpose of this article is to save you time if you’re having trouble getting things working while doing similar tasks.

Pitfalls:

A growing list of things you may, or may not know, proceeds.

Active Response Regular Expression Bug

There’s a race condition and a regular expression bug in the included Active Response (AR) scripts. I noticed it when there were many alerts in a short time. The following snippet is from the OSSEC documentation and included AR scripts.

 # Getting full alert 
 ALERTTIME=`echo "$ALERTID" | cut -d "." -f 1`
 
 # Getting end of alert 
 ALERTLAST=`echo "$ALERTID" | cut -d "." -f 2`
 
 grep -A 10 "$ALERTTIME" ${PWD}/../logs/alerts/alerts.log | grep -v ".$ALERTLAST: " -A 10 | mail $MAILADDRESS -s "OSSEC Alert"

$ALERTID contains a value of the format timestamp.unique that looks like this 1470863865.139339 where the unique number seems to resemble a delta time between alerts.

One issue is that the period in the following example should be escaped, it matches any preceding character not the literal dot. We should also match the delimiter because the numbers could be unique but contain a repeating pattern across the log. For example, take 1470863865.1393343 and 1470863865.1393393 where grep would match both if they’re within 10 lines and the pattern was 1470863865.1393, as well as if the alerts.log log file was written to faster than the AR script could read the first alert from it – very likely with shell scripts and lots of alerts.

The first grep command below is commonly used in AR scripts and listed in the example code above.

 grep -v ".$ALERTLAST: " -A 10 # OK: Matches any preceding character and delimiter 
 grep -v "\.$ALERTLAST: " -A 10 # Better: Match literal dot/period and delimiter

Let’s demonstrate the problem with the examples above by modifying the timestamp and unique fields of 2 alerts. The first alert is the real alert, the second and third have been modified to demonstrate an issue with the regular expression.

 ALERTTIME=472674786 
 ALERTLAST=788 
 $ grep -A 10 "$ALERTTIME" /var/ossec/logs/test.log | grep -v ".$ALERTLAST: " -A 10 
 2016 Aug 31 20:19:46 ossec->/var/log/secure 
 Rule: 5402 (level 3) -> 'Successful sudo to ROOT executed' 
 User: centos 
 Aug 31 20:19:46 ossec sudo: centos : TTY=pts/1 ; PWD=/home/centos ; USER=root ; COMMAND=/bin/bash
 
 ** Alert 14726747863788: - syslog,sudo 
 2016 Aug 31 20:19:46 ossec->/var/log/secure 
 User: Example 1 
 Aug 31 20:19:46 ossec No period in timestamp, the period is matching any single character and thus the 3. We shouldn't be shown.
 
 ** Alert 1472674786.7888: - syslog,sudo 
 2016 Aug 31 20:19:46 ossec->/var/log/secure 
 User: Example 2 
 Aug 31 20:19:46 ossec We have one extra character in the unique field, we shouldn't match either

A better solution is to match the full $ALERTTIME, $ALERTLAST, the literal ., and the delimiter value of : across a single alert.

Alerts in alerts.log are separated by 2 new lines followed by a short header.

** Alert 1472605611.270: - syslog,sudo

Below, we have an alert snippet, note that the last two characters are new lines.

 head -n 6 /var/ossec/logs/alerts/alerts.log | hexdump -c 
 0000000 * * A l e r t 1 4 7 2 6 0 3 
 ... 
 0000100 N D = / b i n / b a s h \n \n

We can use awk to match the alert line and only grab the alert which is a multi-line block of text. The syntax is a little tricky but awk needs multiple escapes for it to internally match a literal .. Let’s run our new expression on the file and see what happens.

 awk -v ts=''"${ALERTTIME}"\\\\."${ALERTLAST}"':' 'BEGIN { RS="\n\n"; ORS="\n" } $0 ~ ts { print }' /var/ossec/logs/test.log 
 ** Alert 1472674786.788: - syslog,sudo 
 2016 Aug 31 20:19:46 ossec->/var/log/secure 
 Rule: 5402 (level 3) -> 'Successful sudo to ROOT executed' 
 User: centos 
 Aug 31 20:19:46 ossec sudo: centos : TTY=pts/1 ; PWD=/home/centos ; USER=root ; COMMAND=/bin/bash

Notice that we actually get the correct alert with our very explicit pattern match.

Another example where we manually fill in the ts variable so you can see what it looks like.

 $ echo $PWD 
 /var/ossec/active-response 
 $ awk -v ts='472674786\\.788:' 'BEGIN { RS="\n\n"; ORS="\n" } $0 ~ ts { print }' /var/ossec/logs/test.log 
 ** Alert 1472674786.788: - syslog,sudo 
 2016 Aug 31 20:19:46 ossec->/var/log/secure 
 Rule: 5402 (level 3) -> 'Successful sudo to ROOT executed' 
 User: centos 
 Aug 31 20:19:46 ossec sudo: centos : TTY=pts/1 ; PWD=/home/centos ; USER=root ; COMMAND=/bin/bash

Using this more accurate expression will improve the accuracy of your AR scripts.

Active-Response Execution Chains

A powerful thing is to have multiple AR (Active Response) scripts triggered on useful alerts. A few examples, where a single script handles each task.

• Send alert data to multiple outputs

• Chat (Slack, HipChat, IRC, etc.)

• Ticket system (JIRA, RT, etc.)

• Perform actions on alert data

• Employee database lookups

• DNS, network management, and whois lookups

• Intelligence lookups (VirusTotal, CIF, etc.)

In OSSEC, you would expect a configuration that runs multiple AR scripts to look something like the following.

 <active-response> 
     <command>ip_lookup</command> 
     <location>server</location> 
 </active-response>
 
 <active-response> 
     <command>dns_lookup</command> 
     <location>server</location> 
 </active-response>
 
 <active-response> 
     <command>whois_lookup</command> 
     <location>server</location>
 </active-response>
 
 <active-response> 
     <command>syscheck_lookup</command> 
     <rules_group>syscheck</rules_group> 
     <level>5</level> 
     <location>server</location>
 </active-response>

However, OSSEC doesn’t seem to follow the expected order of trying the first stanza, then the second, third, and so on.

This can be really frustrating and I haven’t been able to observe a repeatable pattern of behavior. Instead, I came up with a work around for active-response scripts that match a specific alert category which I’ll call catch-all scripts.

These scripts are executed for all events in a specified category and with limited criteria. Upon execution, the catch-all scripts will determine the specific task scripts to run the alert data through next. This way one can get the execution order just right if they’re having trouble.

The new active response scripts are still placed in the same location $OSSEC/active-response/bin/. An example of a catch-all script for all regular alerts (not rootcheck or syscheck alerts) is presented below and available on Github.

 $ cat /var/ossec/active-response/bin/rule_all.sh 
 #!/usr/bin/env bash 
 # Author: Jon Schipp 
 CHAT=/usr/local/bin/ircsay 
 PROG=OSSEC 
 SCRIPT=$0 
 CHANNEL="#ossec-alerts" 
 MAIL=user@org 
 ACTION=$1 
 USER=$2 
 IP=$3 
 ALERTID=$4 
 RULEID=$5
 
 # Exit in case we receive a syscheck alert 
 [[ "$*" =~ syscheck ]] && exit
 
 # This scripts calls others because only one can be executed by OSSEC
 
 die(){ 
 if [ -f ${COWSAY:-none} ]; then 
 $COWSAY -d "$*" 
 else 
 printf "$*\n" 
 fi 
 exit 0 
 }
 
 is_ip(){
 [[ $IP ]] || return 1 
 [[ $IP == '-' ]] && return 1 
 return 0 
 }
 
 # Set paths of launchable active-response scripts 
 LOCAL=$(dirname $0); 
 cd $LOCAL 
 cd ../ 
 PWD=$(pwd) 
 CHAT="$PWD/bin/alert2chat.sh" 
 CIF="$PWD/bin/cif.sh" 
 BHR="$PWD/bin/bhr.sh" 
 CDB="$PWD/bin/add_to_cdb.sh" 
 CMDS="$PWD/bin/command_search.sh" 
 TS="$PWD/bin/time_lookup.sh" 
 LDAP="$PWD/bin/ldap_lookup.sh"
 
 printf "$(date) $0 $ACTION $USER $IP $ALERTID $RULEID $6 $7 $8\n" >> ${PWD}/../logs/active-responses.log
 
 # Chat 
 [[ -x $CHAT ]] && $CHAT $ACTION $USER $IP $ALERTID $RULEID
 
 # Collect system user accounts from 'new user' events, only work on rule 5902 
 [[ -x $CDB ]] && [[ $RULEID -eq 5902 ]] && $CDB $ACTION $USER $IP $ALERTID $RULEID
 
 # Search for suspicious commands 
 [[ -x $CMDS ]] && $CMDS $ACTION $USER $IP $ALERTID $RULEID
 
 # Check if system's clock is off 
 [[ -x $TS ]] && $TS $ACTION $USER $IP $ALERTID $RULEID
 
 # Lookup user's in LDAP 
 [[ -x $LDAP ]] && $LDAP $ACTION $USER $IP $ALERTID $RULEID
 
 ## Alerts that contain IP addresses should have these AR scripts run
 # CIF Feed 
 is_ip && [[ -x $CIF ]] && $CIF $ACTION $USER $IP $ALERTID $RULEID
 
 # BHR Block 
 is_ip && [[ -x $BHR ]] && $BHR $ACTION $USER $IP $ALERTID $RULEID

We launch the rule-all script with the following active-response configuration stanzas in ossec.conf. It will handle executing the appropriate scripts after each matching alert.

 <command> 
     <name>rule_all</name> 
     <executable>rule-all.sh</executable> 
     <expect></expect> 
 </command>
 
 <active-response> 
     <command>rule_all</command> 
     <level>4</level> 
     <location>server</location> 
 </active-response>

We can go further now and create chains of other alert categories such as those created by the syscheck daemon.

 <command> 
     <name>syscheck_all</name> 
     <executable>syscheck-all.sh</executable> 
     <expect>filename</expect> 
 </command>
 
 <command> 
     <name>rule_all</name> 
     <executable>rule-all.sh</executable> 
     <expect></expect> 
 </command>
 
 <active-response> 
     <command>syscheck_all</command> 
     <rules_group>syscheck</rules_group> 
     <level>5</level> 
     <location>server</location> 
 </active-response>
 
 <active-response> 
     <command>rule_all</command> 
     <level>4</level> 
     <location>server</location> 
 </active-response>

Listing the more specific AR configs in ossec.conf first is crucial to reducing the amount of total execution for the scripts as well as the time it takes to act on the alert. The more specific scripts are those that contain AR options to limit execution unless the criteria is met. The level, location, and rules_group tags are among the available AR options for limiting execution. For a more advanced example see the README file at ossec-tools

When we don’t understand OSSEC’s parsing and AR execution implementation we can guarantee the order of execution for multiple events using this method. In addition, we can write more complex conditions to meet before running an AR script, giving us more flexibility over the more limited AR config language. One unfortunate side effect of this is that we have to execute at least 2 scripts for each alert so their will be more fork(2)ing and exec(2)ing.

Rule and Decoder Order

When including a new rule or decoder file, parent rules or decoders must be loaded first if the new ones have dependencies.

 <!-- Custom decoders to load without dependencies --> 
 <decoder_dir>decoders</decoder_dir>
 
 <!-- Load default decoders --> 
 <decoder>etc/decoder.xml</decoder>
 
 <!-- Load improved decoders, which build upon the default decoders, dependencies must be met --> 
 <decoder>etc/improved_decoders.xml</decoder>
 
 <!-- Load default rules --> 
 ... 
 <include>attack_rules.xml</include>
 <include>local_rules.xml</include>
 
 <!-- Custom rules to load, the depend on the default ruleset --> 
 <include>ncsa_rules.xml</include> 
 <include>ncsa_ar_rules.xml</include> 
 <include>ncsa_syscheck_rules.xml</include> 
 <include>overwrite_rules.xml</include>

Action & Status Rule Options

The action and status options to the rule language are not documented but used in ossec_rules.xml.

 <action>run-this-script.sh</action> 
 <status>blah</status>

Action can be used to execute an AR script for a particular rule and status can be used to pass an argument to the AR script.

 <rule id="601" level="3"> 
     <if_sid>600</if_sid> 
     <action>firewall-drop.sh</action> 
     <status>add</status> 
     <description>Host Blocked by firewall-drop.sh Active Response</description> 
 <group>active_response,</group> 
 </rule>
 
 <rule id="602" level="3"> 
     <if_sid>600</if_sid> 
     <action>firewall-drop.sh</action> 
     <status>delete</status> 
     <description>Host Unblocked by firewall-drop.sh Active Response</description> 
 <group>active_response,</group>
 </rule>

The included firewall-drop.sh accepts status as the first argument to the script. The value of status gets assigned to the variable ACTION where it’s used to add or delete a firewall rule.

 ACTION=$1 
 USER=$2 
 IP=$3 
 ...
 
 LOG_FILE="${PWD}/../logs/active-responses.log" 
 echo "`date` $0 $1 $2 $3 $4 $5" >> ${LOG_FILE}
 
 # Checking for an IP 
 if [ "x${IP}" = "x" ]; then
     echo "$0: <action> <username> <ip>" 
     exit 1; 
 fi
 
 # Blocking IP 
 if [ "x${ACTION}" != "xadd" -a "x${ACTION}" != "xdelete" ]; then 
     echo "$0: invalid action: ${ACTION}" 
     exit 1; 
 fi
 
 # We should run on linux 
 if [ "X${UNAME}" = "XLinux" ]; then
     if [ "x${ACTION}" = "xadd" ]; then 
         ARG1="-I INPUT -s ${IP} -j DROP" 
         ARG2="-I FORWARD -s ${IP} -j DROP" 
     else 
         ARG1="-D INPUT -s ${IP} -j DROP" 
         ARG2="-D FORWARD -s ${IP} -j DROP" 
     fi
 ...

Passing Filenames to Active Response Scripts

To pass a filename to an AR script the command stanza must contain

<expect>filename</expect>

Otherwise the filename is not passed. A full example is shown below.

 <command> 
     <name>syscheck_all</name> 
     <executable>syscheck_all.sh</executable> 
     <expect>filename</expect> 
 </command>
 
 <active-response> 
     <command>syscheck_all</command> 
     <location>server</location> 
     <level>5</level> 
     <rules_group>syscheck</rules_group> 
 </active-response>

Note that filename is really passed as the 8th argument when ran for an agent, not the 7th suggested by the OSSEC documentation. I counted the arguments in a log entry from active-response.log to illustrate this.

 0 1 2 3 4 5 6 7 8 
 /var/ossec/active-response/bin/syscheck-all.sh add - - 1453832931.21559200 550 (host1.company.com) 10.1.1.5->syscheck /usr/bin/javap

What are the arguments are passed to the script?

1. action (delete or add)
2. user name (or – if not set)
3. src ip (or – if not set)
4. Alert id (uniq for every alert)
5. Rule id
6. Agent name/host
7. Agent->OSSEC service or location
8. Filename

I preface my shell scripts to assign all the available variables

 SCRIPT=$0 
 ACTION=$1 
 USER=$2 
 IP=$3
 ALERTID=$4 
 RULEID=$5 
 AGENT=$6 
 SERVICE=$7 
 FILENAME=$8

File Changes

It would be really nice to have a rule that can detect a file name, grab the new hash, and look it up in a list of malware hashes. Unfortunately, what might seem intuitive doesn’t work. I wrote a decoder for the rule below that worked in the ossec-logtest tool but not in practice. It turns out that the syscheck code doesn’t pass the hash to the rule.

 # Nice try but this doesn't work 
 <rule id="110000" level="13"> 
     <if_matched_group>syscheck</if_matched_group> 
     <decoded_as>integrity_new_hash</decoded_as> 
     <list field="id" lookup="match_key">lists/mal.list</list> 
     <description>Hash found in malware database!</description> 
 </rule>

It’s a little disappointing since you can do the same task except to look up an IP address rather than a hash. Just remember that syscheck alerts go through a different code path, so don’t get stuck on this for hours like I did.

 # Doing a similar thing but for IP addresses does work! 
 <rule id="100026" level="12"> 
     <if_group>authentication_success</if_group> 
     <list field="srcip" lookup="address_match_key" >lists/ip.list</list> 
     <description>Successful authentication from bad IP</description> 
 </rule>

However, you can alert on changes of an important file such as the one below.

 <rule id="110001" level="12"> 
     <if_sid>550</if_sid> 
     <match>/etc/krb5.keytab</match> 
     <description>Kerberos keytab file modified</description> 
 </rule>

We do have a solution though, to perform a malware hash look we can use AR scripts because the hashes are available in the alert. We can parse them out and do comparisons. I wrote two scripts, one to query Virus Total and the other for Team Cymru Malware Hash Registry.

Overriding Rules and Decoders

You can override a rule without have to use if_sid to attach to it and handle the expression. Often, we don’t need many of the included rules and sometimes we just want a few rules to act slightly different from the defaults. The overwrite rule option is how we can do that.

All you need to do is get the original rule and add it to a local rule file with the ovewrite="yes" option, then modify it so the overwrite rule takes the original rules place.

 <group name="overwrite"> 
     <rule id="502" level="3" overwrite="yes"> 
         <if_sid>500</if_sid> 
         <match>Ossec started</match> 
         <description>Ossec server started.</description> 
     </rule>
 
     <rule id="554" level="10" overwrite="yes"> 
         <category>ossec</category> 
         <decoded_as>syscheck_new_entry</decoded_as> 
         <description>File added to the system.</description> 
         <group>syscheck,</group> 
     </rule> 
 </group>

Because it’s a task that is completed often, I recommend creating a new rule file called overwrite_rules.xml and include it in the main configuration.

 grep overwrite_rules /var/ossec/etc/ossec.conf 
     <include>overwrite_rules.xml</include>

Unfortunately, there’s no way to do this for decoders and thus the pitfall. You have to manually edit or not include the main decoders file etc/decoders.xml.

Limitations in Sudo Decoder

The default sudo decoder doesn’t grab fields other than the user, so I wrote an improvement that will allow you match on other variables.

Update the decoder.xml file with the one below and you will be able to match on the original user, file path, elevated user, and command ran.

 <decoder name="sudo"> 
     <program_name>^sudo</program_name> 
     <regex>^\s+(\S+)\s:\sTTY=\S+\s;\sPWD=(\S+)\s;\sUSER=(\S+)\s;\sCOMMAND=(\.+)$</regex> 
     <order>id,url,dstuser,status</order> 
     <fts>name,user,location</fts> 
     <ftscomment>First time user executed the sudo command</ftscomment> 
 </decoder>

The newly decoded fields are displayed below, they will prove useful for doing things like matching commands and tracking users that can sudo.

 $ /var/ossec/bin/ossec-logtest
 
 **Phase 1: Completed pre-decoding. 
     full event: 'Aug 31 22:48:07 ossec sudo: jschipp : TTY=pts/1 ; PWD=/home/jschipp ; USER=root ; COMMAND=/bin/bash -c echo stuff' 
     hostname: 'ossec' 
     program_name: 'sudo' 
     log: ' jschipp : TTY=pts/1 ; PWD=/home/jschipp ; USER=root ; COMMAND=/bin/bash -c echo stuff'
 
 **Phase 2: Completed 
     decoding. 
     decoder: 'sudo' 
     id: 'jschipp' 
     url: '/home/jschipp' 
     dstuser: 'root' 
     status: '/bin/bash -c echo stuff'

Poorly Documented Options

The following options are in the documentation but not available in OSSEC 2.8.1 and maybe later.

 <rootcheck> 
     <check_policy>yes</check_policy> 
     <skip_nfs>no</skip_nfs> 
 </rootcheck>
 
 <syslog_output> 
     <use_fqdn>yes</use_fqdn> 
     <location>/var/log/secure</location> 
 </syslog_output>