bookmark_borderMultiple 577 entries in the eventlog (from Windows)

I was monitoring the Windows logs from a client network and I noticed that a few boxes were constantly generating audit failure 577 events:

WinEvtLog: Security: AUDIT_FAILURE(577): Security: xxx: XX-HQ: YY-HQ: Privileged Service Called: Server: Security Primary User Name: abc Primary Domain: XX-HQ Privileges: SeIncreaseBasePriorityPrivilege

To make it worse, ossec was alerting me by e-mail every time of “Rule: 18151 fired (level 10) -> “Multiple failed attempts to perform a privileged operation by the same user.””. My initial thought was that these machines were infected by spyware or something similar, so I went investigating them. I didn’t want to disable this rule on ossec, because it is fairly important for “normal” systems.

After some research, I found out that their problem was having Windows XP installed 🙂 (without the latest Service pack). Basically there is a bug on Windows that causes this behavior.. If you ever experience this problem, check the following link: http://support.microsoft.com/kb/831905 and make sure to install the latest updates.

.

bookmark_borderOSSEC Logo/Mascot contest

Can you guess what is missing in the ossec project? If you guessed a mascot (or a logo), you are right. Every open source project has one, but ourselves.

How to fill this gap? If you are you a good designer with a creative mind, you can help us out. We just opened a contest to select a new logo and/or mascot for the ossec project. This logo (or mascot) will be the official symbol of the ossec project and our new face.

There are some prizes included, but the main reward will be the satisfaction of helping out the open source community..

If you are interested, check the following link: http://www.ossec.net/wiki/index.php/CContest

bookmark_borderEight daily steps to a more secure network

Michael Mullins wrote an interesting article with eight daily steps to secure your network. What I really liked is that at least 3 of these 8 steps mentioned involves looking at logs. He mentioned looking at antivirus, security and IDS/firewall logs… great suggestions! However, monitoring these logs MANUALLY every day, as he suggested, can be very hard and time consuming… In addition to that, just by browsing the logs you will miss a lot of good information and correlation that just an automated tool can find. What about using a tool designed for this purpose? OSSEC can analyze every mentioned log and much more…

bookmark_borderOSSEC Performance

A friend of mine recently asked me what is the maximum number of logs per second that ossec could handle, but I didn’t have an answer for him. I heard of a few reports of ossec handling more than 508 logs per second in a setup with more than 400 agents. I also installed it on an ISP that on average receives between 200/250 logs per second. However, there is no definitive answer to this question because it depends a lot on the hardware being used and the format of the logs… Anyway, I decided to run some quick tests to see how good (or bad) it would perform here.

My performance test was very simple: send as many logs as possible to analysisd (main ossec process) and check the /var/ossec/stats/totals directory to see how many logs it was able to process per second.

Test setup:
I created 5 ossec configurations for logcolletor and initiated one separated daemon for each config. Each process was monitoring one log file. The operating system was OpenBSD 3.9 on an old PIII 700 with 512M of RAM.
# /var/ossec/bin/ossec-logcollector -c /var/ossec/etc/log1.conf
# /var/ossec/bin/ossec-logcollector -c /var/ossec/etc/log2.conf
# /var/ossec/bin/ossec-logcollector -c /var/ossec/etc/log3.conf
# /var/ossec/bin/ossec-logcollector -c /var/ossec/etc/log4.conf
# /var/ossec/bin/ossec-logcollector -c /var/ossec/etc/log5.conf

# tail /var/ossec/logs/ossec.log
2007/01/20 15:14:49 ossec-logcollector(1950): Analyzing file: ‘/data/test-logs/log1′.
2007/01/20 15:14:50 ossec-logcollector(1950): Analyzing file: ‘/data/test-logs/log2′.
..
2007/01/20 15:14:56 ossec-logcollector(1950): Analyzing file: ‘/data/test-logs/log5′.
2007/01/20 15:14:56 ossec-logcollector: Started (pid: 15448).

To be fair, I chose 5 different log formats and wrote a simple script to keep filling the logs as faster as possible.

# cd /data/test-logs
# while [ 1 ]; do ./fill-logs.sh; done
# cat fill-logs.sh
cat PIX-sample >> log1 &
cat accesslog-sample >> log2 &
cat authlog-sample >> log3 &
cat messages-sample >> log4
cat squid-sample >> log5

Test results:
I let this setup running for a while (6 to 7 hours) and also wrote a small script to monitor the process/memory utilization. On average, the CPU utilization was around 10% with peaks of 18%. Memory usage was constant of about 2100K.

The average number of events that I got during this 6 hour test was around 1,238,989 or 344 logs per second.

What does it proves? Nothing. Just that OSSEC v1.0 on an old PIII with 512 of RAM can support more than 340 logs per second without using more than 10% of CPU..
Do you run ossec on a large environment? How many logs per second is it monitoring on your environment? Would be nice to have a few examples for future comparison.

bookmark_borderOSSEC version 1.0 is available

OSSEC version 1.0 is now publicly available. This version comes with numerous new features, including support for:Registry monitoring on WindowsDynamic/nat’ed IP addresses in the server/agent communicationASL (Apple system log)Lotus dominoSymantec AVWindows RAR

A full list with all the functionatilies and bug fixes are available at the Changelog.

Make sure to check, as well, our new web interface (in beta) for monitoring your logs and ossec alerts.

Download the new version (and the web interface) here.

bookmark_borderSecurity Monitoring

Richard Bejtlich posted an excellent entry in his blog (taosecurity) about the difference between alert centric tools and Network Security Monitoring (NSM). He says:

Network Security Monitoring (NSM) is different. Generating statistical, session, full content, and alert data gives analysts some place to go when they get an alert — or when they want to do manual analysis… With NSM, an alert is the beginning of the investigation, not the end..

He also points that with alert centric tools (including most SIMs), the investigation ends with the alert.

I agree with everything he says, but I would like to add an important point to his NSM approach: Log data. Even with full session/content/alert data, you can not see everything that is happening. One of the easiest ways to exemplify that is with encrypted protocols. How do you know what is happening in a specific sshd session? What about SSL traffic? In addition to that, what happens at the host level? Kernel or processes errors are not going to be seeing at all. I will give some examples bellow:

1- SSHD traffic
You see an SSH connection to your system, how do you know what happened? There is no difference at the network level between a failed login attempt (three password attempts) and a successful quick sshd session. However, if you look at your logs, you have:

Jan 10 11:23:02 enigma1 sshd[10732]: Failed password for invalid user tst from 192.168.2.56 port 10395 ssh2

You know that the authentication failed. However, failure events is not the only think to watch. For example, at OSSEC we use FTS (First time seen) to alerts on first time events. Whenever a user logs to a system that he/she has never logged in before, you get the following:

Received From: enigma2->/var/log/auth.log
Rule: 10100 fired (level 4) -> “First time user logged in.”
Portion of the log(s):
Jan 10 11:29:36 enigma2 sshd[30291]: Accepted publickey for vuser from 192.168.2.67 port 52636 ssh2

It can also alert on multiple failed login attempts that you would never know for sure from the network level:

Location: enigma->/var/log/authlog
Src IP: 85.25.147.156
SSHD brute force trying to get access to the system.
Jan 7 23:33:17 enigma sshd[392]: Failed password for invalid user ftp from 85.25.147.156 port 49786 ssh2
Jan 7 23:33:17 enigma sshd[392]: Invalid user ftp from 85.25.147.156
Jan 7 23:33:16 enigma sshd[23470]: Failed password for invalid user admin from 85.25.147.156 port 49717 ssh2
Jan 7 23:33:16 enigma sshd[23470]: Invalid user admin from 85.25.147.156
Jan 7 23:33:13 enigma sshd[29767]: Failed password for invalid user mysql from 85.25.147.156 port 49583 ssh2

And more importantly, how do you know what happens AFTER a valid user got access to the system? Did he tried to increase his privileges using su/sudo? Did he succeeded? I don’t think there is any way to do know besides monitoring your logs. Using FTS for su/sudo, we know whenever a user succeeded (for the first time) or failed using these commands.

2007 Jan 10 11:49:03
Rule: 5403 (level 4) -> ‘First time user executed sudo.’
Jan 10 11:47:26 localhost sudo: dcid-user : TTY=pts/2 ; PWD=/home/dcid ; USER=root ; COMMAND=/bin/dir

2- SSL traffic
Most NIDS (Network-based Intrusion detection systems) can not look at SSL encrypted traffic, but we use SSL everywhere now. How to monitor this traffic? Using log analysis you can see exactly what requests the client made and if they were successful or not. I always run Snort (with sguil) on my network and it misses some attacks that ossec using log analysis alerts me. The following one is a good example:

Received From: (wserver) 192.168.2.0->/var/log/apache-ssl/ssl.error_log
Rule: 30117 fired (level 10) -> “Invalid URI, file name too long.”
Portion of the log(s):
[Tue Jan 9 22:16:07 2007] [error] [client 142.162.1.79] request failed: URI too long

And I have numerous others cases where logging was the only way to know what happened (e.g sql injection or other web application attacks against SSL servers).

A few more examples are available in a paper that I wrote a while back ago: log analysis for intrusion detection.

To conclude, Richard also pointed in the past: “I guarantee I could determine if the system was compromised, and by how many parties, faster using NSM techniques than manual log analysis”. I completely agree with him. Manual log analysis is very hard to do and provide little value for the time spent… This is why I wrote OSSEC: to provide an automatized log analysis tool that is easy to use and extend, providing the alerts in a useful way that adds a lot to a security team. Together with NSM, log analysis can detect a compromised system much faster than without it…

bookmark_border2006 OSSEC download numbers

As a late christmas gift to all curious OSSEC users out there, here is some information about the number of downloads in 2006 (note that I only included major released versions). I am very pleased to see that we went from 596 downloads on version 0.5 to almost 5,000 on 0.9-3 (great work to all contributors and to the dev team)…

It is also interesting to note that less than 10% of our users check their downloads using our pgp key and only 15% check the md5/sha1 sum of the packages. Actually, the average of downloads of the pgp keys is only 6% and only 13% for the checksum. I would expect these numbers to be much higher..

How I got these values? Just a simple combination of cat/grep/cut/sort/uniq made the deal:

cat $LOGS | cut -d ” -f 2 | cut -d ” ” -f 2 | grep -E “.tar.gz$|.exe$” | grep -v checksum | grep -o -E “([^/]+)$” | cut -d ? -f 1 | sort | uniq -c

Basically, I cated my log file, extracted the arguments from it (cut and grep), sorted (sort) and generated the total count using uniq… There is probably other ways to do that, but this is what I came up with. If you want a simple way to see the total number for all the files in your logs, the following should work:

cat access_log | cut -d ” -f 2 | cut -d ” ” -f 2 | cut -d ? -f 1 | sort | uniq -c

*This will only work on Apache access logs.

bookmark_borderOSWUI (web ui) screenshots

As requested, I am attaching a few screenshots of our web ui. I will also post a link to a demo site later…

Below is an explanation of each picture (click on them to expand).

  1. Main page. Show the agent status, latest modified files and latest events.
  2. Search results (looking for multiple authentication failures).
  3. Search options.
  4. Stats page.
Main page
Search results (multiple authentication failures)
Search options
OSSEC Stats

bookmark_borderWindows registry monitoring (syscheckd)

I just completed adding support for monitoring the Windows registry on ossec. It seems to be fairly stable right now and hopefully a beta version will be available soon (lots of tests will be required).

The configuration will have the following options available: (inside the syscheck area):

<windows_registry>HKEY_LOCAL_MACHINE</windows_registry> <registry_ignore>HKEY_LOCAL_MACHINESoftwareMicrosoft<registry_ignore>

Where the first option is a list (comma separated) of registry entries to monitor and the second is a list of entries to ignore.

A question now for the Windows users out there:Which registry entries should we monitor by default?

I was thinking on everything at HKEY_LOCAL_MACHINESYSTEM, HKEY_LOCAL_MACHINESECURITY and HKEY_LOCAL_MACHINESAM. Is there anything else worth checking too?
Comments are more than welcome..

bookmark_borderOSSEC Web UI beta 2 available

The second beta version of oswui (ossec web ui) is available. It has numerous bug fixes (for all issues reported so far) and new features, including support for php5 and major design improvements made by Rafael Capovilla.Download it from here.

Installation instructions below:1.0 – Prerequisites

  • -Apache with PHP (4.x or 5.x) installed.
  • -OSSEC (version >0.9-3) already installed.

1.1 – Download the package (link)

1.2 – Uncompress/untar it and move the files to somewhere acessible by your web server

# tar -zxvf ossec-wui-0.1-BETA2.tar.gz
# mv ossec-wui* /var/www/htdocs/ossec-wui

1.3 – Run the setup script

# cd /var/www/htdocs/ossec-wui
# ./setup.sh

1.4 – Add your web server user to the ossec group

# vi /etc/group
..
From:
ossec:x:1002:
To (if your web server user is www):
ossec:x:1002:www

1.5 – Fix the permissions for the tmp directory and restart Apache (for the new permissions to work)

# chmod 770 tmp/
# chgrp www tmp/
# apachectl restart

As the ossec-hids project, it is released under the GPL and donations are welcome.