The guys from matasano posted in their blog an entry about the current state of open source rootkit detection. While I agree that we are way behind the latest rootkit technologies (specially for windows), if you look at the public known unix-based rootkits, we are not that bad. Most of them only use basic system call redirections and can be detected by ossec/rootcheck. It looks like very little has been done focusing on unix-based systems lately…
Below is my reply in their blog:
I think the tool you mentioned that does the connect/bind+kill stuff is rootcheck (now part of ossec). It basically does four things to detect anomalies in the system (that may indicate the presence of a rootkit):
1-Attempts to bind to every TCP and UDP port. If it can’t bind the port (port is used), we check if netstat is reporting it.
2-Attempt to kill(0), getsid and getpgid every process (from 1 to maxpid). We compare the output of these three system calls with ps and proc (where available).
3-Compare the output of stat st_nlink with the count from readdir.
4-Attempt to read every file in the system and compares the size read with the one from stat.
I know these techniques can be evaded, but they are sucessfull against most of the public known unix-based rootkits (99% still based on system call redirection). Rootcheck/ossec also has the rootkits signatures stuff…
In addition to that, OSSEC also does file integrity checking and log analysis to complete its HIDS tasks..
In my opinion, the best way to protect against rootkits is by having an updated and “as secure as possible” system. However, as soon as an attacker finds a way in and gets root (kernel) access, the battle becomes much harder… Early warning systems to detect the attack (not the rootkit) may be the only thing left (anything from log analysis to integrity checking and NIDS).