I am writing this up for several reasons. One of course is to help others learn how to better secure their server, the other is that I am trying to organize my own notes and clean up my own methods of how I would go about investigating a server. Any additions or suggestions are welcome.

There are a few things in this article I want to clarify first before any things else. This is a how to regarding the preliminary investigation of a server compromise. This is not how you would perform computer forensics as some of the methods actually contradict forensic method. Investigating a compromise means that you are interacting with it, preferable in its native environment so that you can try to see the compromise in action, where as in forensics the very first thing you do is isolate the server, cut the power with out shutting down so its state does not change and then remove the hard drive. Also this is not a how to on fixing a compromise. In my own opinion and many others, once compromised it is to be considered as a compromised until it is rebuilt. This is not to say that you have to rebuild just because some one got through with a minor compromise. You patch those areas and then monitor it and if it does not mess things up too badly you let it be until that day arrives when you really do need to upgrade. That is unless you are a bank or running a site that accepts credit cards, in which case most will rebuild after even the slightest of any incident.

Why should you do a hands on investigation when you have a firewall and anti virus software? I like to think of this in layman terms such as putting a security system in your home. You start off by getting a dog, and when any one comes around the dog barks and hopefully they go away. If some one gets past the dog then you might try getting an alarm system. If some one breaks into your home then the guards are sent out to try to catch the person that broke into your home and hopefully they catch them before they get away. When you go into a server and keep an eye on things and watch the processes and log files, that is like sitting at home with a baseball bat. You get to catch them in the act and smack them right on the head! The thing is though we all have a life and have things to do and we cant spend our entire life sitting at home with a baseball bat waiting on a burglar to come along, so you get the barking dog and alarm system as well.


Entering the Server

I am an intuitive type of person where as some others that might write a similar article might write it a different way or even disagree with my methods. There are advantages and disadvantages to both. When you very first enter a server and get to the root user you want to do it subtly, the last thing you want to do is reboot. Remember you are hoping to sneak up on some one or some virus or worm that might just be looking for you as well. They want to get away and escape before you detect them. You dont want to go charging in and running a virus scan and other utilities just yet. The first three things you want to look at are who is on the server, what is the network activity, and what processes are running. The catch is that the way you find these things out is through programs that may have been altered, so you may not be able to trust the information you get. In this you want to use multiple ways of looking at the same information. As you investigate keep a notepad open and cut and past your work into notepad so you can look back and any possible changes as you investigate. Here I am logging into the server remotely through ssh. Dont get distracted with what you find, focus on gathering on as much information as possible before disturbing the environment.


Who is on the Server:

root:~#w
root:~#netstat -nalp |grep ":22"

Or:

root:~#w && netstat -nalp |grep ":22"


Who Has Been on the Server

root:~#last

root:~#cat /var/log/secure* |grep ssh |grep Accept

root:~#cat /var/log/secure* |grep ftp |grep Accept


What is the Current Network Activity:

root:~#netstat -nalp
root:~#nmap localhost

Or:

root:~#netstat -nalp && nmap localhost


What Processes are Running:

root:~#ps -elf
root:~#ls /proc/*/exe -la


Or a nifty little script some co-workers wrote that compairs the two:

for i in $(ps -ax | awk '{print $1"-"$5}' );do PSNAME=$(echo $i | cut -f2- -d-) ; PID=$(echo $i | cut -f1 -d-); echo "PID=$PID" PS SAYS $PSNAME PROC SAYS: $(ls -l /proc/$PID/exe | awk '{print $11}');done

The next things you want to look at are the most common points of attack. The days of the super genius hacker that is trying to outsmart you are pretty much gone. They are still out there but more often than not it is not a hacker at all but a worm that is spreading through the internet, or some kid that found a script on some message board of how to break into servers. Many attacks will either need access to the hard drive to store the binary or will leave behind other temporary files. Many times they will create subfolders that are difficult to read so we will use the –lab option of ls to assist in seeing them. The one thing to remember is that /dev/shm is a virtual mount point, any thing in this directory will be erased after a system reboot.


What Files are in the Common Attack Points:

root:~#ls /tmp -lab
root:~#ls /var/tmp -lab
root:~#ls /dev/shm -lab

Many times you will find they will try to hide subdirectories in ways that make it difficult to read and difficult to find the right combination to enter the folder. Hitting the tab key for the auto complete often helps. The following are a few examples of what to look for.


root:~#ls -la
total 2
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:32
drwxr-xr-x 5 nobody nobody 120 2005-11-25 18:32 .
drwxr-xr-x 33 nobody nobody 2320 2005-11-25 18:31 ..
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:32 ..
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:31 ...


There are three folders here that dont belong in an other wise empty directory. Using the -lab option makes it easier to find.


root:~#ls -lab
total 2
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:32
drwxr-xr-x 5 nobody nobody 120 2005-11-25 18:32 .
drwxr-xr-x 33 nobody nobody 2320 2005-11-25 18:31 ..
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:32 ..
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:31 ...


With this option a space character is replaces with a backslash, the third is that the directory "..." should not exist. Some times the following will work to display the folder.

root:~#ls " " -la
root:~#ls ".. " -la

Or

root:~#ls / -la
root:~#ls .. / -la

As in my own work most of the time I am doing this kind of investigation on a server that I may not have ever seen before that is run by some one else. They may be exceptionally skilled at system administration and some thing managed to sneak past them, or it may be that they are not very skilled at all and the server has been neglected. So the next thing I want to know is what am I really working on. Dont delete any thing or make changes just yet, just catalog every thing. Do not access a file with cat or strings, calalog the files and save that for later. Once you start deleting things you can no longer further investigate as to how deep they have penetrated. Dont be fooled into seeing a common Apache compromise and think it ended there. Many times that was just the broken window they used to get in the first time, meanwhile they are tunneling deeper trying to get into root access.


What version of Linux is running

root:~#cat /etc/redhat-release

or for non Red-Hat Linux

root:~#cat /etc/issue


Comapre this to the kernel

root:~#uname -a

and

root:~#cat /proc/version

Obviously if some one else is on the server using ssh you want to catalog the IP address and use or account they are using. With netstat you are looking for any connections that are out of the ordinary and what the program is named and the process ID, the same goes for any program that you do not recognize that has ports listening. Compare what ports are listening to the nmap and see if there are any differences. From ps –elf are there any process running that you do not recognize. Also eith the –elf option it shows the process ID of what program spawned the program, are there any programs running that look different than all its counter parts in that is was spawned by a different process than all the rest.

Finally any of those strange files you might have found in the common attack points, what is the file ownership, what are the permissions, what is the time stamp. Some additional options you might want to look at before analyzing the file itself.


Who is the author of the file:

root:~#ls –la --author


When was the last time the file has been accessed and by who:

root:~#ls -l --time=access

Before you run off and use the cat command it is good to first check the file type with the file command. Many a time I myself have been fooled seeing a file marked as something.html and finding it was really a binary file.


What kind of file is it:

root:~#file filename

or

root:~#file /path/to/directory/*

Now you know if it is save to use cat or if you need to use the strings command to look deeper into the file. At this point google.com is your best friend for finding more information about what you are looking for, it is time to do your research. Search for the filename by itself, search for the filename with the term exploit. Perhaps it is a root kit of some kind, or perhaps it is just some not so well known application you are not familiar with, or even perhaps it is an application that had become outdated and needs upgrading. If you definitely suspect the server is compromised and have not found any thing just yet, the last thing I like to look at is the network activity in real time.


Real Time Network Activity Examples:

root:~#watch -d "netstat -nalp |grep -v DGRAM |grep -v STREAM |grep -v LISTEN"
root:~#watch "netstat -nalp"|grep ":TCP PORT Number"
root:~#watch "netstat -nalp"|grep ":22"

You have been trying to be sneaky and not have any obvious virus scan running in the process list so as to not be detected, but that is tedious work and slow. Once you get a look at what is going on you can bring out the heavy duty security power tools to make your job easier. If the server load is not too high the next thing I like to do is update the locate database in the background. Many will use complicated find commands which is fine, but I want to know if the locate function is working properly or if it is being fooled into reporting false information. Let this run in the background as you do some other searches.


Update the Locate Database:

root:~#updatedb &

If this is a web server then the next thing to hunt for is signs of Apache exploits and SQL injection scripts. This nice little script was handed down to me from a co-worker and does a nice job of hunting through the log files rather than the long tedious work of searching manually.


Search for Apache Exploit

for i in `locate access_log` ; do echo $i ; egrep -i '(chr(|system()|(curl|wget|chmod|gcc|perl)%20' $i ; done

or

egrep -i '(chr(|system()|(curl|wget|chmod|gcc|perl)%20' /path/to/log/files/*


cPanel
egrep -i '(chr(|system()|(curl|wget|chmod|gcc|perl)%20' /usr/local/apache/logs/*


Ensim
egrep -i '(chr(|system()|(curl|wget|chmod|gcc|perl)%20'/home/virtual/site*/fst/var/log/httpd/*


Plesk
egrep -i '(chr(|system()|(curl|wget|chmod|gcc|perl)%20' /home/httpd/vhosts/*/statistics/logs/*
egrep -i '(chr(|system()|(curl|wget|chmod|gcc|perl)%20' /var/log/httpd/*

Of course not every thing that it locates is an exploit, it takes some interpreting. You are looking for obvious things such as a call to wget to download a file, or a call to perl that looks strangely out of place. However one thing this script does not search for is a call to shell code. Once you locate the access logs run a second search for long strings containing /x90/. I have yet to find a good interpreter of what these shell codes do but again google.com is your friend. Most often you can do a search for the code itself and come up with a technical explanation of what the code does.


Search for Shell Code:
cat /path/to/access/logs/* |grep "/x90/"

Finally now you can run your favorite virus scan and root kit hunter. Some favorites include rkhunter and chkrootkit.


ChkRootkit
http://www.chkrootkit.org


RKHunter
http://www.rootkit.nl

Hopefully by now the updatedb has finished, you can check the status of that by usingthe jobs command. Look for things that are out of place and dont belong, are there .bash_history or access_log files that seem to be in the wrong area. This is where you try to think like a hacker and try to find the strange things that just don’t seem right.


Think Like a Hacker in Your Search

root:~#locate .bash_history
root:~#locate access_log
root:~#locate "..."
root:~#locate ".. "
root:~#locate " .."
root:~#locate ". "
root:~#locate " ."

Again when you find some thing take the same care in cataloging every thing you can. If you have some strange TCP port listening try using telnet to access it and see what comes up. Only after you have cataloged as much as possible then you can start deleting every thing and start locking down the server.

By now you might have some user accounts and possibly some IP addresses and an idea of how they got into the server, what is the best way of handling this is a legal suitable manner. You could try contacting the owner of the server that the attack originated from, but that might open a whole new can of leagal worms. The best way is to instead contact the data center that operates the IP address and pass along the log files and other data you have gathered about the attack. You most likely wont hear back from them as they have their own policies but more often than not they are handling it. Untill they do some thing about it then firewall that IP address so they cannot access your server.


Who Ownes the IP Address

root:~#whois 12.34.56.78

Gather all Data About the IP Address

root:~#for i in `ls /var/log -la |grep -v drwx |grep -v lrwx | awk '{print $9}'`; do echo /var/log/$i; cat /var/log/$i |grep "12.34.56.78 ; done

Looks like a good guide from what I looked over.

Some constructive criticism/ideas:
lsof -p PID is also a great tool when investigating a running process as it lets you track down all the files that a process uses

I would also suggest look in /var/spool/samba and /usr/local/apache/proxy as they tend to be pretty common places.

The find command will let you search by owner. I find it very useful to search a server for all files owned by nobody (or the apache user) if I cannot find anything.

As I look this over and proof read it I am finding more things that "I forgot to mention this" or "how to look at that". Actually some other input as Eth00 has offered is definately what I am looking for, more ideas and tricks of tracking this down. Might have to update the original post if there are enough good ideads in here.

-=- Like watching .bash_history look at .mysql_history

-=- as Ethoo suggested lsof -p PID you can also
cat /proc/ID Number/cmdline
cat /proc/ID Number/environment
cat /proc/ID Number/maps

Any other ideas are more than welcome

A nice list of things to use/watch for when you think system is compromised.
Thank for getting the thread going....

Thanks a lot, this just helped me track down a client's stuck script!

Oh yeah I also made http://www.eth0.us/hacked it is targeted more at finding perl/apache non-root exploits but it may help out some more.

Some additional things to check for more advanced topics:

cat /etc/exports
cat /etc/sysconfig/network |grep NIS
cat /etc/sudoers
cat /etc/xinetd.d/telnet
who /var/log/wtmp

I will have to get around to writing up more detailed information on these.

P.S.

I just have to say kudo's to Cyborg and eth00, I have learned more here by reading their posts than much else on this board.

Additional Script for cPanel

egrep -i '(chr(|system()|(curl|wget|chmod|gcc|perl)%20' /home/*/statistics/logs/*

It is getting to be more and more popular now for the script kiddies to have the scripts delete them self after running making it more and more fun to try and find. Here is a quick check for this:

cat /proc/*/maps |grep deleted

nice work... must have taken forever to type lol.

sometimes netstat -an | grep :22 is not sufficient, either they arent logged in anymore, or they trojan'd the netstat binaries. But most of the time they leave something running, sshd-scan or something like that...

use 'ps aux' or throw them a curve ball and use 'pstree'(they dont usually trojan this... heck who even knows about it!?) and get the pid number. Then 'cd /proc/$pid/ ; cat environ'.

The reason this is great is because the environment variables have one called SSH_CLIENT. Which stores the ip of the person who executed that command.

Hope this helps someone one day,
James

Actually there are a few tricks up my sleave yet that I am not willing to publish just because it seems that when I do that the script kiddies seem to be reading this too. I cant give away all my secrets!

Rule #1
the kiddies and hackers are already ahead of you

Rule #2) Keep your friends close but your enemies closer. only then will you learn those "secrets" that you thought you knew. hint hint......

Hi,

Very useful post

I have just tried to organize this set of command in one post which will useful for checking compromised system:

http://forum.ev1servers.net/showthread.php?t=63029

thx

After typing this:
root@silent [~]# cat /proc/*/maps |grep deleted

i got allot of this:
b5cfb000-b5d56000 rw-s 00000000 00:06 0 /SYSV00000000 (deleted)
b5d56000-b7d56000 rw-s 00000000 00:06 9722 /dev/zero (deleted)
b5cfb000-b5d56000 rw-s 00000000 00:06 0 /SYSV00000000 (deleted)
b5d56000-b7d56000 rw-s 00000000 00:06 9722 /dev/zero (deleted)
..............

is this concerning?

How do I run
egrep -i '(chr\(|system\()|(curl|wget|chmod|gcc|perl)%20' /usr/local/apache/logs/* When I enter it in as root it just brings me back to the server prompt.

I have a cpanel server that was compromised. But it looks clean now. They used a script that replaced all the "index" and "default" pages throughout the entire server.

Found out the linux kernel was way out of date and vulnerable, so I got that updated. Problem is the datacenter refuses to support it now unless they do an OS Reload. I would if they got WHM and cpanel working with all the sites, but apparently they'll just reload it and watch me squirm....

This is one of those commands that you need to get a feel for what the baseline should be, so for example the /SYSV and /dev/zero folders really are not something important. Now if you saw some files that ended in .lib or the like, then you have reason to really worry.

Seems with VMWare becoming more popular that there are some new interesting root kits out there that take advantage of being able to encapsulate the operating system. So theoretically the operating system is not rooted, but rather has become a virtual machine. I am no expert in VMWare but from testing on some compromised server with this configuration, since there is only one virtual machine there is no IP address translation. Both the real server and the virtual server are on the same IP address (Or at least as far as I can tell. I no longer have the resources I once has for investigating these issues.) The footprint of this is that the proc directory is going to look very strange if you really dig into it. On a normal server the kernel process should return normal:

lsattr /proc/1/exe
------------- /proc/1/exe


If you see all sorts of strange sticky bits then you know some thing is very wrong.