People often ask us how we like to setup OSSEC or how we use it internally on my own servers. We always do a set of customizations to make sure we use it the best way possible.
In this article we will show step by step those steps and hopefully it can be helpful to other OSSEC users out there.
Note that we will be focusing on a standalone install, but the principles apply to the agent/manager setup as well.
1- Download and Install
Installing OSSEC is pretty simple, so we won’t spend too much time on it. We always like to use the https://bitbucket.org/dcid/ossec-hids/get/tip.tar.gz package from Bitbucket:
$ wget https://bitbucket.org/dcid/ossec-hids/get/tip.tar.gz $ tar -zxvf tip.tar.gz $ cd dcid-ossec-hids-* # ./install.sh
Tip is the latest source on Bitbucket and we always keep it stable, so the risk of it breaking something is very low.
If you run into issues, check if make, gcc and libc are installed:
# yum install gcc make libc-dev (for Redhat/fedora/centos) # apt-get install gcc make libc-dev
2- Finding the logs
Once OSSEC is installed, we can start tunning it to the server. The first thing we do is find what is running and what type of logs are available.
we generally use lsof to list what is there:
# lsof | grep log httpd 30764 apache 19w REG 202,0 5856044 2088893 /var/log/httpd/site4.access.log httpd 30764 apache 20w REG 202,0 413775 2088897 /var/log/httpd/site3.access.log httpd 32200 apache 2w REG 202,0 2894 2088895 /var/log/httpd/error_log httpd 32200 apache 7w REG 202,0 2894 2088895 /var/log/httpd/error_log httpd 32200 apache 8w REG 202,0 11293853 2088903 /var/log/httpd/site2.error.log syslogd 2220 root 2w REG 202,0 317 121533 /var/log/messages syslogd 2220 root 3w REG 202,0 3474 121774 /var/log/secure syslogd 2220 root 4w REG 202,0 147252 121855 /var/log/maillog syslogd 2220 root 5w REG 202,0 46954 121875 /var/log/cron syslogd 2220 root 6w REG 202,0 0 121873 /var/log/spooler syslogd 2220 root 7w REG 202,0 0 121874 /var/log/boot.log mysqld 2359 mysql 1w REG 202,0 20162 121188 /var/log/mysqld.log mysqld 2359 mysql 2w REG 202,0 20162 121188 /var/log/mysqld.log .. and many more...
And we compare with what OSSEC added automatically
# cat /var/ossec/etc/ossec.conf |grep "<location>/" <location>/var/log/auth.log</location> <location>/var/log/syslog</location> <location>/var/log/dpkg.log</location>
The logs that are missing we add manually using the util.sh tool that comes with OSSEC:
# /var/ossec/bin/util.sh addfile /var/log/httpd/site4.access.log /var/ossec/bin/util.sh: File /var/log/httpd/site4.access.log added. # /var/ossec/bin/util.sh addfile /var/log/mysqld.log /var/ossec/bin/util.sh: File /var/log/mysqld.log added. .. for all others ..
We also check using netstat -tanep to see what is running on the server to see if we might have missed something:
# netstat -tanep |grep LISTEN tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 27 4659 2359/mysqld tcp 0 0 :::22 0.0.0.0:* LISTEN 0 4561 2278/sshd tcp 0 0 :::80 :::* LISTEN 0 4793 2411/httpd
3- Testing the logs
Before even putting into production, we like to test if OSSEC is able to parse all the logs properly.
For that, we use the tool ossec-logtest with the -a option to analyse old events and compare with a manual audit of the logs.
On my desktop, if we run it against the /var/log/syslog I get:
# cat /var/log/syslog | /var/ossec/bin/ossec-logtest -a 2012/03/29 17:49:48 ossec-testrule: INFO: Reading local decoder file. 2012/03/29 17:49:48 ossec-testrule: INFO: Started (pid: 7440). ** Alert 1333054188.1: mail - syslog,errors, 2012 Mar 29 17:49:48 goiabada->stdin Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.' Mar 29 10:11:04 goiabada kernel: [604548.729517] chrome[11541]: segfault at 60 ip b3d27f27 sp bf940a50 error 4 in libgtk-x11-2.0.so.0.2400.6[b3c34000+461000] ** Alert 1333054188.2: mail - syslog,errors, 2012 Mar 29 17:49:48 goiabada->stdin Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.' Mar 29 10:29:06 goiabada kernel: [605630.394325] chrome[20438]: segfault at cc ip b37a6d87 sp bfa6e024 error 4 in libpthread-2.13.so[b379e000+17000]
We can also run without the -a to see if it is properly decoding the logs:
4- Monitoring commands
We use the command monitoring on OSSEC to be able to track state changes and to get a full picture of all my agents.
Those are the commands we like to monitor:
<localfile> <log_format>full_command</log_format> <command>/sbin/iptables -nL</command> </localfile> <localfile> <log_format>full_command</log_format> <command>netstat -tan |grep LISTEN |grep -v 127.0.0.1 | sort</command> </localfile> <localfile> <log_format>full_command</log_format> <command>last -n 5</command> </localfile>
We specially like the last, netstat, df, iptables and similar commands since they allow me to view the current state of the system.
We can go to the queue directory and see the state from all the systems, like who logged in on all of them:
# cat /var/ossec/queue/diff/*/535/last-entry ossec: output: 'last -n 5': root pts/0 1.2.3.4 Mon Jun 4 21:00 - 03:04 (06:04) root pts/0 2.3.4.5 Mon May 7 20:59 - 20:46 (4+23:46) daniel pts/1 2.3.3.4 Fri May 4 22:34 - 22:34 (00:00)
5- Adding Decoders/Rules
It is not often that we need to write custom rules/decoders, but it is always good to be ready for them when needed.
to be added later
6- Integrity checking
Integrity checking is one of the most misused options on OSSEC. It can range from very annoying to completely useless if you don’t do it properly. The default options from OSSEC don’t help much with that as well.
The real goal of integrity checking is to verify that the integrity of the files are still intact (they haven’t changed). However, if a file changes many times per week, it becomes very hard to distinguish between a normal and a malicious change.
How we use integrity checking:
1- On real time 2- Only on a few set of files 3- Storing the changes so I can compare if needed
The real time option is specially important, because I can tie it with my policy rules to alert when they happen on times it shouldn’t.
So, on my web servers, we add the following in there:
<directories realtime="yes" report_changes="yes" restrict=".php|.js">/var/www</directories>
Which restricts the integrity checking to the files we care (generally .php, .js, .htm, etc) and monitors those in real time. So when a file changes I get the alert showing what was modified as well.
7- Conclusion
This is not a finished document, we will keep updating it as time permits. But if you have questions or concerns, please don’t hesitate to ask.